Image-based sensing is a powerful source of information for a wide range of applications such as surveillance, monitoring, and automation. The volume of data collected by surveillance devices is enormous; therefore, images are processed to extract application-driven data, moreover, centralized processing is unfeasible when several cameras are distributed across broad geographical regions. In such scenario, with processing limitations and bandwidth availability being important restrictions, an automated process is required for extracting meaningful information from the collected data during surveillance process and reducing operator fatigue. The proposed design, which continuously processes the image stream to meet real-time requirements, provides integrated video analytics with enhancement, panoramic views, and automated motion detection. This system enhances pictorial information for human interpretation and analyzes scene data using machine-vision modules. These features aid in the use of surveillance system for continuous monitoring of border areas without the need for human intervention under adverse-weather conditions. The proposed system architecture is based on distributed video-analytics using visual and infrared-band sensors. The system has a visual detection range of 5 km with 80% detection probability for a 2.3 m X 2.3 m target under standard clear visibility conditions. The preliminary results indicate that the system is capable of processing up to 25 video frames per second. Additionally, the implemented system records data for post-analysis and uses video analytics to reduce the bandwidth requirement of communication link. Finally, this system can be distinguished because it is flexible, compact, affordable, and scalable, and therefore, can be considered for monitoring and surveillance.