As video surveillance exponentially increases, a method that automatically detects abnormal events in video surveillance is essential. Several anomaly detection methods have been proposed to detect abnormal events in video surveillance. Much research has recently used deep learning to obtain high anomaly detection accuracy. Most of the research considered only anomaly detection accuracy, but they do not consider the anomaly detection speed that is essential in video surveillance. In this paper, we propose a Cross U-Net framework that considers anomaly detection accuracy and speed. The Cross U-Net framework uses a newly proposed deep learning model that uses two subnetworks based on U-Net. It makes that every third layer’s output in the contracting path combines with the corresponding layer’s output in the other subnetwork for use as the next layer’s input. This framework also uses a cascade sliding window method, a newly proposed method estimating the anomaly score of a frame. We evaluated the Cross U-Net framework’s anomaly detection accuracy and speed using Ped2, Avenue, and ShanghaiTech datasets. We achieved competitive anomaly detection accuracy and real-time anomaly detection in the three datasets.