Regular inspection and maintenance work is required to ensure the structural integrity of historic structures, especially the masonry structures which are deteriorating due to ageing and man-made activities. The structures are typically examined by visual inspection, which is a costly and laborious procedure, and often, the inspection results are subjective. In this study, an automatic image-based crack detection system using Convolutional Neural Network (CNN) for masonry structures is proposed to aid the inspection procedure. Previous crack detection systems generally involve handcrafted features, which are then classified by classification algorithms. This approach relies heavily on feature extraction stage, which may not offer accurate results as some hidden features may not be extracted. In this paper, the feature extraction process is done by CNN from RGB images, and then the softmax layer is replaced by other classifiers to improve classification accuracy. Three classifiers are studied, namely the CNN itself, Support Vector Machines (SVM) and Random Forest (RF). A dataset containing images of cracks from masonry structures was created using a digital camera and an unmanned aerial vehicle. The images were used in training and validating the proposed system. The collected images were also used to build a 3D model using the technique based on Structure from Motion (SFM), which allowed images containing cracks to be located in 3D world coordinates. It was found that the combined CNN and SVM model performs the best among other methods with the detection accuracy of approximately 86% in the validation stages and 74% in the testing stage. As shown in this paper, the integration of CNN and other classifiers can improve detection accuracy. In addition, it was shown that the system can be used to detect cracks automatically for the images of masonry structures, which is useful for the inspection of heritage structures.