To test the use of well‐studied and widely used classification methods alongside newly developed data‐filtering techniques specifically designed for imbalanced‐data classification in order to demonstrate proof of principle for an automated radiation therapy (RT) quality assurance process on prostate cancer treatment.

A series of acceptable (majority class, n = 61) and erroneous (minority class, n = 12) RT plans as well as a disjoint set of acceptable plans used to develop features (n = 273) were used to develop a dataset for testing. A series of five widely used imbalanced‐data classification algorithms were tested with a modularized guided undersampling procedure that includes ensemble‐outlier filtering and normalized‐cut sampling.

Hybrid methods including either ensemble‐outlier filtering or both filtering and normalized‐cut sampling yielded the strongest performance in identifying unacceptable treatment plans. Specifically, five methods demonstrated superior performance in both area under the receiver operating characteristics curve and false positive rate when the true positive rate is equal to one. Furthermore, ensemble‐outlier filtering significantly improved results in all but one hybrid method (p < 0.01). Finally, ensemble‐outlier filtering methods identified four minority instances that were considered outliers in over 96% of cross‐validation iterations. Such instances may be considered distinct planning errors and merit additional inspection, providing potential areas of improvement for the planning process.

Traditional imbalanced‐data classification methods combined with ensemble‐outlier filtering and normalized‐cut sampling provide a powerful framework for identifying erroneous RT treatment plans. The proposed methodology yielded strong classification performance and identified problematic instances with high accuracy.