In this paper, an effective framework is proposed to classify four types of citrus leaf diseases out of healthy leaves through leaf features inspection. Four considered citrus leaf diseases are canker, sooty mold, greening and leaf-miner. Our framework consists of three main stages of pre-processing, feature extraction and classification. Since image analysis of citrus leaf disease is based on texture, color and shape features through leaf region, the region of main leaf is first segmented from the complex background in the pre-processing stage. Then leaf features are extracted in different color spaces and prominent ones are chosen based on feature distribution analysis. The selected features are fed to the SVM classification stage to detect and classify the diseases. In our study, we propose a hierarchical SVM classification model based on leaf features analysis to improve the accuracy rate. The numerical results prove the effectiveness of our proposed hierarchical SVM model over the multi-class SVM model. Our proposed hierarchical SVM model gives the infected leaf detection rate of 92.5%, and a high accuracy rate of 91.76%.