During the inspiral and merger of a binary black hole, gravitational radiation is emitted anisotropically due to asymmetries in the merger configuration. This anisotropic radiation leads to a gravitational wave kick, or recoil velocity, as large as∼ 4000 km s–1. We investigate the effect gravitational recoil has on the retention of intermediate-mass black holes (IMBHs) within the population of Galactic globular clusters by simulating the response of IMBHs to black hole mergers. Assuming that our current understanding of IMBH formation is correct and yields an IMBH seed in every globular cluster, we find a significant problem in retaining low-mass IMBHs (≲ 1000 M⊙) in the typical merger-rich globular cluster environment. Given a uniform black hole spin distribution and orientation and a stellar-mass black hole mass function generated in a low-metallicity system, we find that only three of the Milky Way globular clusters can retain an IMBH with an initial mass of 200 M⊙. Even if IMBHs have an initial mass of 1000 M⊙, only 60 would remain within Milky Way globular clusters, and each would reside only in the most massive clusters. Our calculations show that if there are black holes of mass M> 50 M⊙ in a cluster, repeated IMBH-black hole encounters will eventually eject a M= 1000 M⊙ IMBH with greater than 30% probability. As a consequence, a large population of rogue black holes may exist in our Milky Way halo. We briefly discuss the dynamical implications of this process and its possible connection to ultraluminous X-ray sources (ULXs).