We propose an edge segment based statistical backgroundmodeling algorithm and a moving edge detectionframework for the detection of moving objects. We analyzethe performance of the proposed segment based statisticalbackground model with traditional pixel based, edge pixelbased and edge segment based approaches. Existing edgebased moving object detection algorithms fetches difficultydue to the change in background motion, object shape, illuminationvariation and noise. The proposed algorithmmakes efficient use of statistical background model usingthe edge-segment structure. Experiments with natural imagesequences show that our method can detect moving objectsefficiently under the above mentioned environments.