Absolute pitch (AP) refers to the ability of identifying the pitch of a given tone without reliance on any reference pitch. The downside of possessing AP may be the experience of disturbance when exposed to out-of-tune tones. Here, we investigated this so-far unexplored phenomenon in AP, which we refer to as auditory aversion. Electroencephalography (EEG) was recorded in a sample of AP possessors and matched control musicians without AP while letting them perform a task underlying a so-called affective priming paradigm: Participants judged valenced pictures preceded by musical primes as quickly and accurately as possible. The primes were bimodal, presented as tones in combination with visual notations that either matched or mismatched the actually presented tone. Both samples performed better in judging unpleasant pictures over pleasant ones. In comparison with the control musicians, the AP possessors revealed a more profound discrepancy between the two valence conditions, and their EEG revealed later peaks at around 200 ms (P200) after prime onset. Their performance dropped when responding to pleasant pictures preceded by incongruent primes, especially when mistuned by one semitone. This interference was also reflected in an EEG deflection at around 400 ms (N400) after picture onset, preceding the behavior responses. These findings suggest that AP possessors process mistuned musical stimuli and pleasant pictures as affectively unrelated with each other, supporting an aversion towards out-of-tune tones in AP possessors. The longer prime-related P200 latencies exhibited by AP possessors suggest a delay in integrating musical stimuli, underlying an altered affinity towards pitch–label associations.