In the present study, damage initiation and growth in a polycrystalline aggregate are investigated. In this regard, an anisotropic continuum damage mechanics coupled with rate-dependent crystal plasticity theory is employed. Using a thermodynamically consistent procedure, a finite deformation formulation is derived. For this purpose, the damage tensor is incorporated in the crystal plasticity formulation for a cubic single crystal. The damage evolution is considered to be dependent on the history of damage, equivalent plastic strain, stress triaxiality, and Lode parameters. This material model is implemented in the commercial finite-element code Abaqus/Standard by developing a user material subroutine (UMAT). Using the available experimental tests of 316L single crystal in the literature, the crystal plasticity hardening and damage parameters are calibrated considering the stress–strain curve before and after necking, respectively. The damage sites in a single-phase polycrystalline aggregate are also considered using a polycrystalline model consisting of grains with random sizes and orientations. The results show that the damage arises at the grain boundaries and triple junctions. Moreover, growth of the damage mostly occurs in the grains with higher Schmid factor compared to the neighboring grains. The presented model manifests capacity for determination of damage initiation sites and damage evolution in polycrystalline models.