This paper proposes and presents the details of an agent-based architecture that has the capacity to model, simulate the movement of, and guide crowds with panic behavior in highly-dense areas in the events of manmade and natural threats. The most populated human gathering in the world, the Muslim pilgrimage event (also known as Hajj) which attracts more than two million people in six days is chosen to guide the identification of the requirements and main concepts of the agent simulation. The inferred requirements are used to guide the selection of an appropriate simulation tool following an extensive survey of potential simulation tools. The architecture uses software agents to represent groups of people along with their real time characteristics such as density, movements and panic levels, collected from various data sources. Support vector machine is combined with the fuzzy logic theory to predict pilgrims' behaviors and movements in case of panic situations. Initial simulation tests and results of the SVM model demonstrated that it is feasible to minimise the problem of modelling large crowds by selecting the best representative crowds that best reflect pilgrims’ behaviors. Furthermore, the implications of this architecture and a future plan are presented.