Cataract is the leading cause for visual impairment and blindness. Cataract screening can effectively improve the recovery rate of cataract, and the left and right eye classification is a significant step in cataract screening. Anterior segment optical coherence tomography (AS-OCT) is a non-contact, high-resolution ophthalmic imaging technique, which can quickly obtain pathological information of cataract and left and right eye position information through three-dimensional (3D) imaging. In order to improve the efficiency of cataract screening, we propose a multi-task three-dimensional convolutional neural network (MT-CNN) for automatic cataract detection and left and right eye classification simultaneously based on the 3D AS-OCT images. The MT-CNN is designed based on the hard sharing mechanism, achieving better performance with fewer parameters than single-task learning. The results on an AS-OCT image dataset show that the 3D CNN model obtains better classification performance than the 2D CNN model. Compared with the single-task 3D CNN model, MT-CNN achieves higher accuracy under the premise of greatly parameters reduction and computational complexity reduction.