To uncover the circuit-level alterations that underlie atypical sensory processing associated with autism, we adopted a symptom-to-circuit approach in the Fmr1-knockout (Fmr1^–/–) mouse model of Fragile X syndrome. Using a go/no-go task and in vivo two-photon calcium imaging, we find that impaired visual discrimination in Fmr1^–/– mice correlates with marked deficits in orientation tuning of principal neurons and with a decrease in the activity of parvalbumin interneurons in primary visual cortex. Restoring visually evoked activity in parvalbumin cells in Fmr1^–/– mice with a chemogenetic strategy using designer receptors exclusively activated by designer drugs was sufficient to rescue their behavioral performance. Strikingly, human subjects with Fragile X syndrome exhibit impairments in visual discrimination similar to those in Fmr1^–/– mice. These results suggest that manipulating inhibition may help sensory processing in Fragile X syndrome.