Mango is an agricultural produce with high export value as it is being consumed internationally. To ensure its production yield, the manual handling and classification tasks should be performed with precision and care by local farmers. Image processing and machine learning has improved the way classification, defect detection, and yield approximation are handled. Detection is considered as an initial step prior to performing these tasks. This paper presents an automatic mango detector by combining a Support Vector Machine (SVM) classifier trained with Histogram of Oriented Gradients (HOG) features and image segmentation. The image segmentation performed on both HSV and RGB color spaces using image processing techniques achieved a mean IoU of 0.7938. A HOG-SVM based classifier was trained and achieved an F-score of 89.38%. Results show that combining segmentation with HOG-SVM can detect and localize healthy and defective mango images with different background color and illumination.