Local image features or interest points provide compact and abstract representations of patterns in an image. In this paper, we address the problem of detecting and tracking multiple moving people based on color interest points. The proposed method uses the statistical Gaussian Mixture Model (GMM) for the segmentation, extraction of moving people and background area. After that, from the detected foreground we determine the rules that define skin regions for good people detection. Color Interest Points are identified in the detected regions of skin using Harris algorithm. The use of an interest points set allows us to track people by matching these ones from image to image based on ZNCC correlation approach (Zero mean Normalized Cross Correlation). Finally, by calculating Euclidean distance between the best matches and other interest points detected on each consecutive images of video sequence, we can observe the motion of people tracked in the scene. Proposed results are obtained from two different types of videos, namely sport video and class video. The simulations and the experimental results show the robustness of our method to achieve the track with a good precision. The results are very encouraging, as well as, our proposed method fits well with noise conditions and contrast changes.