Nonlinear optical responses of materials play a vital role for the development of active nanophotonic and plasmonic devices. Optical nonlinearity induced by intense optical excitation of mobile electrons in metallic nanostructures can provide large-amplitude, dynamic tuning of their electromagnetic response, which is potentially useful for all-optical processing of information and dynamic beam control. Here we report on the sub-picosecond optical nonlinearity of indium tin oxide nanorod arrays (ITO-NRAs) following intraband, on-plasmon-resonance optical pumping, which enables modulation of the full-visible spectrum with large absolute change of transmission, favourable spectral tunability and beam-steering capability. Furthermore, we observe a transient response in the microsecond regime associated with slow lattice cooling, which arises from the large aspect-ratio and low thermal conductivity of ITO-NRAs. Our results demonstrate that all-optical control of light can be achieved by using heavily doped wide-bandgap semiconductors in their transparent regime with speed faster than that of noble metals.