This study aimed to determine the relationship between the apolipoprotein E (APOE) ε4 allele and cerebrospinal fluid (CSF) and neuroimaging biomarkers of Alzheimer’s disease, and cognition in cognitively unimpaired (CU) middle-aged adults (n = 82; M_age = 58.2), and in Aβ- CU older adults (n = 71; M_age = 71.8). Aβ- CU middle-aged ε4 carriers showed lower CSF Aβ₄₂ levels, higher levels of CSF total tau (t-tau) and neurofilament light (NfL), and poorer cognitive performance compared to noncarriers (Cohen’s d: 0.30–0.56). In Aβ- CU older adults, ε4 carriers also had lower CSF Aβ₄₂ levels and higher levels of CSF t-tau and p-tau181, compared to noncarriers (Cohen’s d: 0.65–0.74). In both Aβ- middle-aged and older adults, hippocampal and total brain volume were equivalent between ε4 carriers and noncarriers. In Aβ- CU middle-aged adults, APOE ε4 is associated with decreased levels of Aβ, increased tau and NfL, and poorer cognition. Similar relationships were observed in Aβ- CU older adults. These results have implications for understanding clinicopathological relationships between APOE ε4 and the emergence of cognitive and biomarker abnormalities in Aβ- adults.