Autonomous vehicles (AVs) have emerged as a promising technology for enhancing road safety and mobility. However, designing AVs involves various critical aspects, such as software and system requirements, that must be carefully addressed. This paper investigates safety-aware approaches for AVs, focusing on the software and system requirements aspect. It reviews the existing methods based on software and system design and analyzes them according to their algorithms, parameters, evaluation criteria, and challenges. This paper also examines the state-of-the-art artificial intelligence-based techniques for AVs, as AI has been a crucial element in advancing this technology. This paper reveals that 63% of the reviewed studies use various AI methods, with deep learning being the most prevalent (34%). The article also identifies the current gaps and future directions for AV safety research. This paper can be a valuable reference for researchers and practitioners on AV safety.