Motion of an object is thought to be perceived independently of the object’s surface properties. However, theoretical, neuropsychological and psychophysical observations have suggested that motion of textures, called ‘second-order motion’, may be processed by a separate system from luminance-based, or ‘first-order’, motion. Functional magnetic resonance imaging (fMRI) responses during passive viewing, attentional modulation and post-adaptation motion after-effects (MAE) of these stimuli were measured in seven retinotopic visual areas (labeled V1, V2, V3, VP, V4v, V3A and LO) and the motion-sensitive area MT/MST (V5). In all visual areas, responses were strikingly similar to motion of first- and second-order stimuli. These results differ from a prior investigation, because here the motion-specific responses were isolated. Directing attention towards and away from the motion elicited equivalent response modulation for the two types. Dramatic post-adaptation (MAE) differences in perception of the two stimuli were observed and fMRI activation mimicked perceptual changes, but did not reveal the processing differences. In fact, no visual area was found to respond selectively to the motion of second-order stimuli, suggesting that motion perception arises from a unified motion detection system.