An agent-based transport simulation model is used to examine the impacts of Autonomous Vehicles (AVs) on the mobility of certain groups of people. In the state of the art, it has been found that the researchers primarily have simulation studies focusing on the impacts of AVs on people regardless of certain groups. However, this study focuses on assessing the impacts of AVs on different groups of users, where each group is affected variously by the introduction of different penetration levels of AVs into the market. The Multi-Agent Transport Simulation (MATSim) software, which applies the co-evolutionary algorithm and provides a framework to carry out large-scale agent-based transport simulations, is used as a tool for conducting the simulations. In addition to the simulation of all travellers, 3 groups of users are selected as potential users of AVs, as follow:(1) long commuters with high-income,(2) elderly people who are retired, and (3) part-time workers. Budapest (Hungary) is examined in a case study, where the daily activity plans of the households are provided. Initially, the existing daily activity plans (ie, the existing condition) of each group are simulated and assessed before the introduction of AVs into the market. After that, the AVs are inserted into the road network, where different fleet sizes of AVs are applied based on the demand of each group. The marginal utility of the travel time spent in case of a transport mode, the AV fleet size, and the cost of the travel are the key variables that determine the use of a transport mode. The key variables are set based on the characteristics of the case study (ie, demand) and the AVs. The results of the simulations suggest that the AVs have different degrees of influences on certain groups as demonstrated in the occurred changes on the modal share. The value of changes depends on the Value of Travel Time (VOT) of people and the used fleet size of AVs. Moreover, the influence of the traveller’s characteristics on the AVs is manifested, such as different values of fleet utilization. Furthermore, the study demonstrates that an increase in the fleet size of AVs beyond 10% of the demand does not significantly raise the modal share of AVs. The outcome of this paper might be used by decision-makers to define the shape of the AVs’ use and those groups who are interested in using AVs.