Deep learning-based predictive models, for identifying malign tumor data in various cancer types, emerged as a hot research topic in the Internet of Medical Things (IoT). Such models can be deployed onto biomedical machines acting as IoMT devices to provide highly accurate breast cancer screening. While there has been a significant advancement in deep learning models for classifying cancer imaging data, a key shortcoming exists in terms of their use as blackbox algorithms rendering them unexplainable or non-interpretable. Caregivers, such as oncologists and radiologists, however, need to understand the nature of the model outcome. We address this in this paper by providing an end-to-end explainable AI framework for analyzing breast cancer prediction models based on a publicly available mammography dataset. In addition, we demonstrate how the various methods in such an end-to-end system can be effectively evaluated with appropriate performance measures.