As a key enabler for 6th generation (6G) communications, millimeter-wave (mmWave) ultra-dense network (UDN) has been examined. However, due to the dense deployment of SBSs, an excessive number of data links and frequent handover incur highly inefficient energy consumption in user association. Despite many recent works on power-saving user association in mmWave UDN, energy-efficiently associating users for a long time is left as a challenging problem. In this paper, we propose a multi-agent actor-critic (MA-AC)-based user association scheme to minimize the energy consumption mmWave UDN. By applying actor-critic, a kind of deep reinforcement learning (DRL), the proposed scheme learns to optimally associate users to minimize long-term energy consumption. In order to overcome the extreme signaling overhead in mmWave UDN, local agents in SBSs distributively associate users based on local information. From the simulations, we demonstrate that the proposed user association scheme reduces mmWave UDN energy consumption substantially.