Which isoforms of apolipoprotein E (apoE) we inherit determine our risk of developing late-onset Alzheimer’s Disease (AD), but the mechanism underlying this link is poorly understood. In particular, the relevance of direct interactions between apoE and amyloid-β (Aβ) remains controversial. Here, single-molecule imaging shows that all isoforms of apoE associate with Aβ in the early stages of aggregation and then fall away as fibrillation happens. ApoE-Aβ co-aggregates account for ~50% of the mass of diffusible Aβ aggregates detected in the frontal cortices of homozygotes with the higher-risk APOE4 gene. We show how dynamic interactions between apoE and Aβ tune disease-related functions of Aβ aggregates throughout the course of aggregation. Our results connect inherited APOE genotype with the risk of developing AD by demonstrating how, in an isoform- and lipidation-specific way, apoE modulates the aggregation, clearance and toxicity of Aβ. Selectively removing non-lipidated apoE4-Aβ co-aggregates enhances clearance of toxic Aβ by glial cells, and reduces secretion of inflammatory markers and membrane damage, demonstrating a clear path to AD therapeutics.