Electrocardiograms (ECGs) have been extensively utilized for diagnosing cardiovascular abnormalities. However, due to the mixed noise and the subtle differences between ECGs, it is generally arduous to spot the ECG abnormalities with satisfactory efficiency with the naked eye. To address these issues, we proposed a novel automatic system for diagnosing arrhythmia. In this paper, several independent component analysis and principal component analysis networks (ICA-PCANets) were developed as the ECG feature extraction methods. To verify their effectiveness, linear support vector machine (SVM), K-nearest neighbors (KNN) and random forest (RF) were adopted as the classifier models in this work. Among them, the combination of ICA-PCANet and linear SVM achieved the highest accuracies of 98.01%, 98.63%, and 91.77% by classifying 2 classes, 5 classes (AAMI standard), and 14 detailed categories, respectively, on the MIT-BIH database. Based on the above comprehensive performances, the proposed system can be applied to clinical monitoring of heart conditions.