A solid with larger sound speeds usually exhibits higher lattice thermal conductivity. Here, we report an exception that CuP₂ has a quite large mean sound speed of 4155 m s⁻¹, comparable to GaAs, but single crystals show very low lattice thermal conductivity of about 4 W m⁻¹ K⁻¹ at room temperature, one order of magnitude smaller than GaAs. To understand such a puzzling thermal transport behavior, we have thoroughly investigated the atomic structures and lattice dynamics by combining neutron scattering techniques with first-principles simulations. This compound crystallizes in a layered structure where Cu atoms forming dimers are sandwiched in between P atomic networks. In this work, we reveal that Cu atomic dimers vibrate as a rattling mode with frequency around 11 meV, which is manifested to be remarkably anharmonic and strongly scatters acoustic phonons to achieve the low lattice thermal conductivity.