The impact of the initial severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infecting strain on downstream immunity to heterologous variants of concern (VOCs) is unknown. Studying a longitudinal healthcare worker cohort, we found that after three antigen exposures (infection plus two vaccine doses), S1 antibody, memory B cells, and heterologous neutralization of B.1.351, P.1, and B.1.617.2 plateaued, whereas B.1.1.7 neutralization and spike T cell responses increased. Serology using the Wuhan Hu-1 spike receptor binding domain poorly predicted neutralizing immunity against VOCs. Neutralization potency against VOCs changed with heterologous virus encounter and number of antigen exposures. Neutralization potency fell differentially depending on targeted VOCs over the 5 months from the second vaccine dose. Heterologous combinations of spike encountered during infection and vaccination shape subsequent cross-protection against VOC, with implications for future-proof next-generation vaccines.