Ground penetrating radar (GPR) has been widely applied in the assessment of railway ballast conditions (fouling, moisture) by using the spectrum of the tested GPR signal. However, the drawbacks of the low time‐frequency resolution and mode mixing prevent the traditional spectrum methods from a wide application. This paper uses the advanced time‐frequency analysis of GPR signal based on optimized variational mode decomposition to extract the features of ballast. The new approach overperforms the conventional frequency spectrum methods of GPR signal processing by giving a clear and quantitative assessment of ballast signals. Experimental results of GPR with dry and wet fouled ballasts demonstrate that, by comparison with the feature extraction method of conventional spectrum methods such as spectrogram and wavelet, the feature extraction method based on the optimized VMD has much better separability and quantitative identification capability.