Deep learning refers to a set of computer models that have recently been used to make unprecedented progress in the way computers extract information from images. These algorithms have been applied to tasks in numerous medical specialties, most extensively radiology and pathology, and in some cases have attained performance comparable to human experts. Furthermore, it is possible that deep learning could be used to extract data from medical images that would not be apparent by human analysis and could be used to inform on molecular status, prognosis, or treatment sensitivity. In this review, we outline the current developments and state-of-the-art in applying deep learning for cancer diagnosis, and discuss the challenges in adapting the technology for widespread clinical deployment.