Decades of white-tailed deer (Odocoileus virginianus (Zimmermann, 1780)) overpopulation have dramatically homogenized forests across much of the eastern United States, creating depauperate forest understory communities. The rate at which these communities recover once deer browsing has been reduced remains an open question. We evaluate overbrowsing legacy effects by examining how forest herbaceous layers respond in terms of biodiversity, density, and community composition over 11 years using exclosures and control plots within a mature beech–maple forest. Although little recovery occurred in the first 5 years, total density and preferred browse density rebounded substantially during the final years of the study. Although community composition began to diverge between exclosure and control plots after 5 years, diversity failed to recover even after 11 years of excluding browsers. Our findings show that vulnerable species can increase after excluding browsers but only if those species were initially present. Biodiversity recovery may be extremely slow because preferred browse species have been nearly extirpated from many forests and thus are unable to recruit into refugia. We empirically demonstrate the extent of the ghost of herbivory past or legacy effect of browsing, i.e., the substantial time delay between herbivore abatement and community response after decades of high deer densities.