The ultrafast optical nonlinearity of an optically characterized single gold nanorod is investigated around its surface plasmon resonance, by combining a far-field spatial modulation technique with a high sensitivity pump-probe setup. The spectrally and temporally dependent response is quantitatively interpreted in terms of the bulklike optical nonlinearity enhanced by the plasmonic effect. The plasmon resonance dynamics is shown to be mostly governed by nonequilibrium electron and phonon processes. Their contributions to the nonlinear optical response of a single metal nano-object are elucidated, and the latter is connected to the nonlinearities of ensembles.