Ocular diseases can lead to irreversible vision impairment if not treated timely. Various imaging techniques have been developed to aid in the detection of ocular diseases, including the widely employed color fundus photography. Nevertheless, early-stage ocular diseases are difficult to be accurately diagnosed because of the few visible symptoms, and automatic ocular disease classification based only on imaging is extremely challenging. In this paper, we propose a knowledge distillation-based method to improve the performance of imaging-based automatic ocular disease classification models. Specifically, two deep neural networks are optimized sequentially. A teacher network is trained that can exploit the information from inputs of both color fundus photographs and radiologists provided clinical features. Then, through distilling the knowledge of the teacher model, a student network is learned that can self-speculate the clinical features-relevant information from the sole inputs of images. Extensive experiments validate that our student model can largely recover the performance of the teacher model and thus, the proposed method can significantly enhance the imaging-based ocular disease diagnosis without the reliance on clinical features.