With the ever-increasing traffic congestion issues and fast development of autonomous and connected vehicles, autonomous intersection management (AIM) has been recently proposed as a promising concept to effectively and safely enhance the traffic efficiency at intersections. However, most current literature focus on discrete space modeling, and there is few investigation on continuous space modeling in AIM due to high computational complexity, though continuous modeling can lead to better accuracy and performance. In this paper, we propose a novel multi-agent reinforcement learning-based AIM protocol with continuous intersection modeling and action space. To address the challenge of high data dimensionality and computational complexity in continuous modeling, we further provide an attention mechanism in our learning-based AIM protocol, termed as self-attention proximal policy optimization (SA-PPO) algorithm. The proposed SA-PPO algorithm can make a vehicle to control the speed more precisely and extract relevant information from neighboring vehicles to reduce data complexity. Results demonstrate that our proposed protocol can improve the traffic performance by up to 30% compared with commonly used DQN algorithm in complex and dense traffic environments.