The human sensitivity to interaural temporal differences in the acoustic waveforms to the two ears shows remarkable acuity but is also very sluggish. Fast changes in binaural parameters are not detectable, and this inability contrasts sharply with the excellent temporal resolution of the monaural auditory system. We studied the response of binaural neurons in the inferior colliculus of the cat to sinusoidal changes in the interaural correlation of broadband noise. Responses to the same waveforms were also obtained from auditory nerve fibers and further analyzed with a coincidence analysis. Overall, the auditory nerve and inferior colliculus showed a similar ability to code changes in interaural correlation. This ability extended to modulation frequencies an order of magnitude higher than the highest frequencies detected binaurally in humans. We conclude that binaural sluggishness is not caused by a lack of temporal encoding of fast binaural changes at the level of the midbrain. We hypothesize that there is no neural substrate at the level of the midbrain or higher to read out this temporal code and that this constitutes a low-pass “sluggishness” filter.