Coarse aggregate plays the role of skeleton and accounts for the highest proportion in asphalt pavement, and its particle size and gradation distribution have an important impact on pavement performance. Traditional gradation detection methods by square sieves are inefficient and labor-intensive. This paper is a study of automated gradation detection methods based on 3D point clouds and deep learning. A line structured light based acquisition system was built to collect aggregate point cloud data, and a multi-threshold segmentation method is proposed to segment each aggregate from the background. To solve the problem of high aggregate complexity and low discrimination between different particle sizes, we propose an aggregate particle size classification model ISRNet (Improved Structure Relation Network) with pointnet. Based on SA (Set Abstraction), we add adaptive sampling (AS) after Farthest Point Sampling (FPS) to shift the sampling points adaptively. In addition, for the characteristic that the aggregate particle size is affected by different local structures, we propose to add ISRN after SA to learn the relation features. For the quality prediction problem, a GA-LightGBM model was developed with nine 3D features as input and parameter optimization inspired by genetic algorithms. In addition, the effects of different acquisition parameters on the model are discussed. Compared with other classification algorithms, the accuracy of the proposed classification method achieved 99.12% in the range of 0–10 mm and 98.89% in the range of 10–20 mm. Moreover, the maximum error of aggregate gradation evaluation is 1.63%, which can meet the industrial requirement. It proves the advantages of 3D point cloud in determining the aggregate particle size and gradation, improving the automatic detection efficiency and saving labor costs.