Rotating machinery is widely applied in industrial applications. Fault diagnosis of rotating machinery is vital in manufacturing system, which can prevent catastrophic failure and reduce financial losses. Recently, Deep Learning (DL)-based fault diagnosis method becomes a hot topic. Convolutional Neural Network (CNN) is an effective DL method to extract the features of raw data automatically. This paper develops a fault diagnosis method using CNN for InfRared Thermal (IRT) image. First, IRT technique is utilized to capture the IRT images of rotating machinery. Second, the CNN is applied to extract fault features from the IRT images. In the end, the obtained features are fed into the Softmax Regression (SR) classifier for fault pattern identification. The effectiveness of the proposed method is validated using two different experimental data. Results show that the proposed method has a superior performance in identification various faults on rotor and bearings comparing with other deep learning models and traditional vibration-based method.