In this paper we investigate the use of hierarchical reinforcement learning to speed up the
acquisition of cooperative multi-agent tasks. We extend the MAXQ framework to the multi-
agent case. Each agent uses the same MAXQ hierarchy to decompose a task into sub-tasks.
Learning is decentralized, with each agent learning three interrelated skills: how to perform
subtasks, which order to do them in, and how to coordinate with other agents. Coordination
skills among agents are learned by using joint actions at the highest level (s) of the …

In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed
up the acquisition of cooperative multi-agent tasks. We introduce a hierarchical multi-agent
reinforcement learning (RL) framework, and propose a hierarchical multi-agent RL algorithm
called Cooperative HRL. In this framework, agents are cooperative and homogeneous (use
the same task decomposition). Learning is decentralized, with each agent learning three
interrelated skills: how to perform each individual subtask, the order in which to carry them …