We show that the form of the recently proposed subleading soft graviton and gluon theorems in any dimension are severely constrained by elementary arguments based on Poincaré and gauge invariance as well as a self-consistency condition arising from the distributional nature of scattering amplitudes. Combined with the assumption of a local form as it would arise from a Ward identity the orbital part of the subleading operators is completely fixed by the leading universal Weinberg soft pole behavior. The polarization part of the differential subleading soft operators in turn is determined up to a single numerical factor for each hard leg at every order in the soft momentum expansion. In four dimensions, factorization of the Lorentz group allows us to fix the subleading operators completely.