Fabric defect detection from images is critical for the textile industry. This paper presents a proposed framework for classifying fabric images as defect-free or defective. A saliency map is used in the proposed framework to determine where a specific region differs from its neighbors in terms of image features. The saliency map is then used to extract global features using keypoint detection techniques, such as Speeded Up Robust Features (SURF) and Feature Accelerated Segment Test (FAST). The dimensions of the resulting features are then reduced using fast Principal Component Analysis (PCA) as a feature reduction technique. For the defect detection process, the obtained features are submitted to a Support Vector Machine (SVM) or a K-Nearest Neighbor (K-NN) classifier. Performance of this classifier is determined in terms of accuracy. According to the simulation results, the proposed framework has the best accuracy of 99.68% with SVM classifier. It has been observed that the proposed framework is efficient and can assist experts in determining whether a fabric image is defective or defect-free.