Pedestrian detection, tracking, and suspicious activity recognition have grown increasingly significant in computer vision applications in recent years as security threats have increased. Continuous monitoring of private and public areas in high-density areas is very difficult, so active video surveillance that can track pedestrian behavior in real time is required. We present an innovative and robust deep learning system as well as a unique pedestrian dataset that includes student behavior like as test cheating, laboratory equipment theft, student disputes, and danger situations in institutions. It is the first of its kind to provide pedestrians with a unified and stable ID annotation. Again, we also presented a comparative analysis of result achieved by the recent deep learning approach of pedestrian detection, tracking, and suspicious activity recognition methods on a recent benchmark dataset. Our investigation will provide new research directions in vision-based surveillance for practitioners and research scholars.