A novel power-transformer protection algorithm based on the shape of the magnetization hysteresis curve is proposed. The magnetization hysteresis loop is studied and described by a voltage-differential current (V-DCurr) curve; furthermore, the characteristics of the V-DCurr curve are studied at different operating conditions. Three basic characteristics that can distinguish a fault from the other operating conditions are fed as the input to a support vector machine (SVM); the faults can be detected by the classifier of the SVM, based on the definite characteristics of the V-DCurr curve. Additionally, an optimization method for determining a suitable classifier, based on the grid search technique, is introduced to improve the reliability and efficiency of the proposed method. Digital simulation and dynamic-model experiments are conducted to verify the proposed method. Thus, the method is shown to have good application prospects in power systems.