Psychological stress is a potential modifiable risk factor for cognitive decline. However, the extent to which self-reported psychological stress is differentially associated with decline in specific cognitive domains remains unclear. Differences may be due to heterogeneity in the aspects of psychological stress investigated, for example, neuroticism (which is linked to vulnerability to stress), perceived stress, or exposure to stressful life events. This review aims to establish the associations between these aspects of self-reported psychological stress and cognitive decline. PsychINFO, Embase and MEDLINE were searched from database inception to September 2021. Studies were included if they were observational, prospective, and if they investigated the association between self-reported psychological stress and cognitive decline in adults with a minimum mean age of 40 years at baseline. Thirty studies satisfied the inclusion criteria, with most examining neuroticism (n = 17) as a predictor of cognitive decline. Fewer examined perceived stress (n = 7) or stressful life events (n = 6). There was evidence of an association between neuroticism and cognitive decline, particularly in the domain of memory. Similarly, across studies, perceived stress was also associated with memory decline. Research investigating the relationship between stressful life events and cognitive decline had fewer outcomes to interpret. Overall, the findings highlight that memory may be particularly susceptible to high levels of neuroticism and perceived stress. We identified a lack of research into some cognitive domains, such as executive function, which should be addressed by future studies.