Deoxynivalenol and zearalenone contamination in maize poses a threat to food safety. Thus, the development of a cost-effective and non-destructive method to classify deoxynivalenol and zearalenone contaminated maize is an important research issue. Natural deoxynivalenol and zearalenone contaminated maize were randomly selected for this experiment. Visible light imaging under ultraviolet light excitation combined with polarized light imaging was used to extract texture and color features from the endosperm and germ sides. And six classification models were established for the single-kernel classification of 1035 contaminated maize. The results indicated that the optimal detection surface was the endosperm side, and the optimal model was SVM with a classification accuracy of 88.1%. Then, local feature analysis of images and mycotoxin concentrations verification experiment were used to reduce secondary contamination and further evaluate the performance of the model, respectively. This method can be verified to be effective in identifying three deoxynivalenol and zearalenone contamination levels of maize, and can also reduce the costs of test stands.