A new weak fault diagnosis method based on feature reduction with Supervised Orthogonal Local Fisher Discriminant Analysis (SOLFDA) is proposed. In this method, the Shannon mutual information (SMI) between all samples and training samples is combined into SMI feature sets to represent the mutual dependence of samples as incipient fault features. Then, SOLFDA is proposed to compress the high-dimensional SMI fault feature sets of testing and training samples into low-dimensional eigenvectors with clearer clustering. Finally, Optimized Evidence-Theoretic k-Nearest Neighbor Classifier (OET-KNNC) is introduced to implement weak failure recognition for low-dimensional eigenvectors. Under the supervision of class labels, SOLFDA achieves good discrimination property by maximizing the between-manifold divergence and minimizing the within-manifold divergence. Meanwhile, an orthogonality constraint on SOLFDA can make the output sparse features statistically uncorrelated. Therefore, SMI feature set combining SOLFDA is able to extract the essential but weak fault features of rotating machinery effectively, compared with popular signal processing techniques and unsupervised dimension reduction methods. The weak fault diagnosis example on deep groove ball bearings demonstrates the advantage of the weak fault diagnosis method proposed in this paper.