We report on the broadband transient optical response of anisotropic, amorphous silicon nanobricks that exhibit Mie-type resonances. A quantitative model is developed to identify and disentangle the three physical processes that govern the ultrafast changes of the nanobrick optical properties, namely, two-photon absorption, free-carrier relaxation, and lattice heating. We reveal a set of operating windows where ultrafast all-optical modulation of transmission is achieved with full return to zero in 20 ps. This is made possible because of the distinct dispersive features exhibited by the competing nonlinear processes in transmission and despite the slow (nanosecond) internal lattice dynamics. The observed ultrafast switching behavior can be independently engineered for both orthogonal polarizations using the large anisotropy of nanobricks, thus allowing ultrafast anisotropy control. Our results categorically ascertain the potential of all-dielectric resonant nanophotonics as a platform for ultrafast optical devices and reveal the possibility for ultrafast polarization-multiplexed displays and polarization rotators.