Visual perception relies on low-level encoding of local orientation. Recent studies show an age-dependent impairment in orientation discrimination of stimuli embedded in external noise, suggesting that encoding of orientation is inefficient in older adults. In the present study we ask whether aging also reduces decoding, i.e., selecting the neural representations of target orientation while discarding those conflicting with it. We compared younger and older participants capability (mean age 24 and 68 years respectively) in discriminating whether the orientation of a Gabor target was left or right from the vertical. We measured (d′), an index of discrimination sensitivity, for orientation offset ranging from 1° to 12°. In the isolated target condition, d′ was reduced by aging and, in the older group, did not increase with orientation offset, thus resulting in a larger group difference at large than small orientation offsets from the vertical. Moreover, oriented elements in the background impaired more discrimination in the older group. However, distractors reduced more d′ when target-background orientation offset was large than when target and flanker had similar orientation, indicating that the effect of the background was not local, i.e., due to target inhibition by similarly oriented flankers. Altogether, these results indicate that aging reduces the efficiency in discarding the response to orientations differing from the target. Our results suggest that neural decision-making mechanisms, involving not only signal enhancement but also non-signal inhibition, become inefficient with age. This suggestion is consistent with the neurophysiological evidence of inefficient visual cortical inhibition in aging.