The deep learning techniques have been playing an important role in the identification and classification problems such as diseases in medical science, marketing in the industry, manufacturing in engineering, and identification in plant taxonomy science. Fruit identification and its family classification is among one of the areas that needs more emphasis for the sake of automation. With this inspiration, fruit images for 52 species belonging to four different families (Apiaceae, Brassicaceae, Asteraceae, and Apocynaceae) have been used in this study to build a deep learning analysis dataset. Further, the dataset has been augmented to 3800 images, divided to 2660 images for training and 1440 for testing, and different 14 fruit images belonging to the same families have been used for prediction of the testing module. A novel Convolution Neural Network (CNN) model architecture has been proposed to extract the fruit features, classify each image with its family, and use the trained model to predict that the new fruits belong to the same four families. The maximum accuracy obtained for the training and testing module was 99.82%. The prediction for this module succeeded by 93% since all fruits’ success predicted was attained except one from the family number 2 (Brassicaceae). The same dataset was applied to two different models to evaluate our proposed model, the Deep learning model, aka Residual Neural Network, 20 layers (ResNet-20), and Support Vector Machine (SVM). The proposed CNN model achieved higher accuracy and efficiency than the ResNet-20 and SVM.