During development, cortical plasticity is associated with the rearrangement of excitatory connections. While these connections become more stable with age, plasticity can still be induced in the adult cortex. Here we provide evidence that structural plasticity of inhibitory synapses onto pyramidal neurons is a major component of plasticity in the adult neocortex. In vivo two-photon imaging was used to monitor the formation and elimination of fluorescently labeled inhibitory structures on pyramidal neurons. We find that ocular dominance plasticity in the adult visual cortex is associated with rapid inhibitory synapse loss, especially of those present on dendritic spines. This occurs not only with monocular deprivation but also with subsequent restoration of binocular vision. We propose that in the adult visual cortex the experience-induced loss of inhibition may effectively strengthen specific visual inputs with limited need for rearranging the excitatory circuitry.