We combine far-infrared photometry from Herschel (PEP/HerMES) with deep mid-infrared spectroscopy from Spitzer to investigate the nature and the mass assembly history of a sample of 31 luminous and ultraluminous infrared galaxies ((U) LIRGs) at z∼ 1 and 2 selected in GOODS-S with 24 μm fluxes between 0.2 and 0.5 mJy. We model the data with a self-consistent physical model (GRASIL) which includes a state-of-the-art treatment of dust extinction and reprocessing. We find that all of our galaxies appear to require massive populations of old (> 1 Gyr) stars and, at the same time, to host a moderate ongoing activity of star formation (SFR⩽ 100 M☉ yr− 1). The bulk of the stars appear to have been formed a few Gyr before the observation in essentially all cases. Only five galaxies of the sample require a recent starburst superimposed on a quiescent star formation history. We also find discrepancies between our results and those based on optical-only spectral energy distribution (SED) fitting for the same objects; by fitting their observed SEDs with our physical model we find higher extinctions (by ΔA V∼ 0.81 and 1.14) and higher stellar masses (by Δlog (M⋆)∼ 0.16 and 0.36 dex) for z∼ 1 and z∼ 2 (U) LIRGs, respectively. The stellar mass difference is larger for the most dust-obscured objects. We also find lower SFRs than those computed from L IR using the Kennicutt relation due to the significant contribution to the dust heating by intermediate-age stellar populations through" cirrus" emission (∼ 73% and∼ 66% of the total L IR for z∼ 1 and z∼ 2 (U) LIRGs, respectively).