Ultrafast nanophotonics is an emerging research field aimed at the development of nanodevices capable of light modulation with unprecedented speed^{, , –}. A promising approach exploits the optical nonlinearity of nanostructured materials (either metallic or dielectric) to modulate their effective permittivity via interaction with intense ultrashort laser pulses. Although the ultrafast temporal dynamics of such nanostructures following photoexcitation has been studied in depth, sub-picosecond transient spatial inhomogeneities taking place at the nanoscale have been overlooked so far. Here, we demonstrate that the inhomogeneous spacetime distribution of photogenerated hot carriers induces a transient symmetry breaking in a highly symmetric plasmonic metasurface. The process is fully reversible and results in a broadband transient dichroism with a recovery of the initial isotropic state in less than 1 ps, overcoming the speed bottleneck caused by slower (electron–phonon and phonon–phonon) relaxation processes. Our results pave the way to ultrafast dichroic devices for high-speed modulation of light polarization.