In this paper, we consider numerical approximations of diffusion terms for finite volume as well as discontinuous Galerkin schemes. Both classes of numerical schemes are quite successful for advection equations capturing strong gradients or even discontinuities, because they allow their approximate solutions to be discontinuous at the grid cell interfaces. But, this property may lead to inconsistencies with a proper definition of a diffusion flux. Starting with the finite volume formulation, we propose a numerical diffusion flux which is based on the exact solution of the diffusion equation with piecewise polynomial initial data. This flux may also be used by discontinuous Galerkin schemes and gives a physical motivation for the Symmetric Interior Penalty discontinuous Galerkin scheme. The flux proposed leads to a one-step finite volume or discontinuous Galerkin scheme for diffusion, which is arbitrary order accurate simultaneously in space and time. This strategy is extended to define suitable numerical fluxes for nonlinear diffusion problems.