In this paper, we propose a modeling framework for large eddy simulations of particle-laden turbulent flows that captures the interaction between the particle and fluid phase on both the resolved and subgrid scales. Unlike the vast majority of existing subgrid-scale models, the proposed framework not only accounts for the influence of the subgrid-scale velocity on the particle acceleration but also considers the effect of the particles on the turbulent fluid flow. This includes the turbulence modulation of the subgrid scales by the particles, which is taken into account by the modeled subgrid-scale stress tensor and the effect of the unresolved particle motion on the resolved flow scales. Our modeling framework combines a recently proposed model for enriching the resolved fluid velocity with a subgrid-scale component, with the solution of a transport equation for the subgrid-scale kinetic energy. We observe very good agreement of the particle pair separation and particle clustering compared to the corresponding direct numerical simulation. Furthermore, we show that the change of subgrid-scale kinetic energy induced by the particles can be captured by the proposed modeling framework.