The multifarious video contents generated through various applications is growing exponentially resulting in huge volumes. Analyzing these contents manually is a cumbersome task particularly in terms of extracting key information by satisfying a criterion. The concept of smart city is popular across the world to develop technology-driven cities where facilities are offered intelligently. In smart cities, the video analysis may play a crucial role in several real-life applications such as, smart security surveillance, traffic monitoring, video forensics, sports, entertainment, medical, etc. Thus, video analysis plays a vital role where larger real-time videos can be intelligently analyzed to detect key interesting patterns to yield an application-centric shorter summary. Moreover, machine or deep paradigms can be applied on video data generated in various smart applications in the smart cities to create a real-time model. In this study, we explore the usage of video analysis in various real-time applications in smart cities. Hence, the work aims to expound a detailed investigation of computer vision-based video analysis approaches in various aspects of smart cities. Besides, a generic video analysis layered architecture is also presented which highlights the deployment of video analysis-centric approaches for real-life smart cities facilities. Our analysis of the existing approaches clearly demonstrates the pertinency of video analysis in several smart city’s mundane infrastructure. However, the study also reveals numerous scopes where video analysis yet to be explored and that offers a clear insight to the researchers. In addition to opportunities, our study identifies some open research challenges to the active research community. Moreover, the survey can serve as a reference to the investigators as well as to the planning and development authorities of smart cities.