For CMOS image sensor (CIS) manufacturing, automatic optical inspection (AOI) is the critical equipment for defect reduction and yield enhancement. The capability of automated defect classification embedded in AOI equipment is the key factor to empower manufacturing intelligence. In manufacturing practice, catching rate (CR) and false alarm rate (FAR) are two indices to evaluate the performance of classification results. Rather than enhancing catching rate, reducing false alarm rate of the classification result is more concerned for the domain engineers since false alarms may cause poor decision-making and waste resource. Hence, domain users are easy to lose their confidence for the classification model even though the model had good performance on catching rate. Focusing on the realistic needs, this study aims to develop a manufacturing intelligence framework integrating defect inspection, feature extraction, support vector machine classifier, and similarity matching approach to reduce false alarm of defect classification, while the catching rate is enhanced. An empirical study was conducted in a leading CIS manufacturing company in Taiwan to estimate the validity and the results also demonstrated the practical value of the proposed approach.