Solving the Bethe-Salpeter equation (BSE) for the optical polarization functions is a first principles means to model optical properties of materials including excitonic effects. One almost ubiquitously used approximation neglects the frequency dependence of the screened electron-hole interaction. This is commonly justified by the large difference in the magnitude of electronic plasma frequency and exciton binding energy. We incorporated dynamical effects into the screening of the electron-hole interaction in the BSE using two different approximations as well as exact diagonalization of the exciton Hamiltonian. We compare these approaches for a naphthalene organic crystal, for which the difference between exciton binding energy and plasma frequency is only about a factor of ten. Our results show that in this case, corrections due to dynamical screening are about 15% of the exciton binding energy. We analyze the effect of screening dynamics on optical absorption across the visible spectral range and use our data to establish an effective screening model as a computationally efficient approach to approximate dynamical effects in complex materials in the future.