The acquisition and evaluation of road cracks are essential to ensure the availability of roads and necessary maintenance. However, the road cracks images have been obsessed with the problem of imbalance in the category and the number of categories. Among them, the category imbalance makes the network focus on the background and the detection result will be complete black. The imbalanced number of categories leads to the missed detection of thin cracks. In addition, a large number of images generated in real time put forward higher requirements on memory and calculations. The RAO‐UNet is built which is an efficient and effective network for crack detection in road images using encoder–decoder and residual attention module‐based image frequency relationship. Compared with otheTr methods, RAO‐UNet could learn multiple‐spatial‐frequency features, thus can enhance the differentiation of high‐frequency features while saving the computational cost. Regarding the space optimisation, a novel balance loss function is proposed, which not only solves the balance problem, but also ensures the stability and consistency in the optimisation process. We evaluated RAO‐UNet on public data sets. Compared with state‐of‐the‐art methods, it achieves better performance on processing speed and detection accuracy. Specifically, RAO‐UNet achieves 98.32% / 97.86% Precision, 97.84% / 95.89% Recall, 97.61% / 97.04% F1 score on CFD and AigleRN data sets, respectively.