Autonomous systems are already available for public and private transport. The necessary hardware and software products have been created, and novel designs for (semi-) autonomous vehicles are launched every year, but their use is limited, and the penetration is not increasing rapidly. While this might be owing to their high price, their perception is also not universally positive. Many are afraid of not only using, but being around them. After introducing the relevant literature on trust in autonomous vehicles and the factors affecting it, the current article presents the data of an international quantitative research of 666 people. It highlights the biggest perceived threats and their prevalence, and also tries to uncover why more than half of the respondents are afraid of autonomous vehicles. In line with the data presented in the article, the topic is gendered–male respondents were more open towards autonomous vehicles. Furthermore, those who are not ready for autonomous vehicles have a generally higher level of fear of potential negative consequences, such as hacker attacks, system malfunctions, or lack of control. On the other hand, those in favour of automated vehicles believe that they have a positive effect on the occurrence of accidents, owing to their heightened reaction speed provided by the sensory system and the computing capacity which is far superior to that of humans, as well as on the society, on carbon emission, and, as a result, on our natural environment. Consequently, autonomous vehicles could form an important element of the transport systems of future smart cities.