Roads make a huge contribution to the economy of a territory. As a platform for transportation, roads are widely used by every countries in the world. Potholes in road are one of the major concerns in the transportation infrastructure. A lot of research works have been proposed using Computer Vision to detect potholes that include wide range of image processing and object detection algorithms. There is a need for potholes detection with adequate accuracy and speed, and that can be implemented with ease and low setup cost. In this paper, we have developed efficient deep learning Convolution Neural Networks (CNNs) to detect potholes in real-time with adequate accuracy. This paper compares the performance of YOLOv5 Large (Y), YOLOv5 Medium (Y) and YOLOv5 Small (Y) with ResNet101 backbone and Faster R-CNN with ResNet50 (FPN), VGG16 and MobileNetV2 backbone. The experiments results show that YOLOv5s is more applicable for real-time potholes detection because of its speed. It is able to detect potholes in high as well as low resolution images in 0.009s. With the use of Inverse Perspective Mapping with YOLOv5 to estimate area, we concluded that the detection can still be done in real time with area estimation.