Multi-agent generative adversarial imitation learning (MAGAIL) is a recent approach that extends single-agent GAIL to problems in multi-agent imitation learning. While MAGAIL shows promising results on cooperative and competitive tasks, it requires agent-environment interactions during training, which may reduce sample efficiency in practice. Moreover, MAGAIL was validated empirically on only a handful of agents, and its scalability to larger numbers of agents remains a question. We propose a multi-agent imitation learning algorithm that addresses these issues. Specifically, we apply multi-agent actor-critic (MAAC) and multi-agent attention-actor-critic (MAA2C)–off-policy multi-agent reinforcement learning (MARL) approaches–in the MARL imitation learning inner loop, as opposed to MACK–the onpolicy MARL method used in MAGAIL. We then model centralized and decentralized discriminators to evaluate whether a given behavior results from agent or expert actions, defining reward functions for the MARL inner loop. We demonstrate that our method scales more effectively, and more sampleefficient, than MAGAIL. We also demonstrate that imitation learning with decentralized discriminators is robust, performing surprisingly well for a large number of agents compared to its centralized counterpart.