Near-infrared long-slit spectroscopy has been used to study the stellar population (SP) of the low luminosity active galactic nuclei (AGNs) and matched analogues (LLAMA) sample. To perform the SP fits we have employed the X-shooter simple stellar population models together with the starlight code. Our main conclusions are: The star formation history of the AGNs is very complex, presenting many episodes of star formation during their lifetimes. In general, AGN hosts have higher fractions of intermediate-age SP (light-weighted mean ages, <t > _L ≲ 4.5 Gyr) when compared with their analogues (<t > _L ≲ 8.0 Gyr). AGNs are more affected by reddening and require significant fractions of featureless continuum and hot dust components. The ratio between the AGN radiated energy and the gravitational potential energy of the molecular gas (E_Rad/E_PG) for the AGN is compared with the <t > _L and a possible anticorrelation is observed. This suggests that the AGN is affecting the star formation in these galaxies, in the sense that more energetic AGN [log(E_Rad/E_PG) ≳ 3] tend to host nuclear younger SP (<t > _L ≲4 Gyr). We found that the recent (t <2 Gyr) returned (recycled) stellar mass is higher in AGN than in the controls. We also provide evidence that the mass-loss of stars would be enough to feed the AGN, thus providing observational constraints for models that predict that AGN feeding is partially due to the recycled gas from dying stars.