Most state-of-the-art local interpretation methods explain the behavior of deep learning classification models by assigning importance scores to image pixels based on how influential each pixel was towards the final decision. These interpretations are unable to provide further details to aid understanding of a complex concept in a domain such as medicine. We propose a novel Hierarchical Visual Concept (HVC) interpretation framework for CNN-based image classification models. As an explanation of the classification decision of a given image, HVC presents a concept hierarchy of most relevant visual concepts at multiple semantic levels. These concepts are automatically learned during training such that the lower-level concepts in the hierarchy support the corresponding higher-level concepts. Our quantitative and qualitative evaluation of the interpretation of VGG16 and ResNet50 classifiers on public and private colonoscopy image datasets shows very promising results.