A new approach is proposed for measuring structural dynamics in materials from multi-speckle scattering patterns obtained with partially coherent X-rays. Coherent X-ray scattering is already widely used at high-brightness synchrotron lightsources to measure dynamics using X-ray photon correlation spectroscopy, but in many situations this experimental approach based on recording long series of images (i.e. movies) is either not adequate or not practical. Following the development of visible-light speckle visibility spectroscopy, the dynamic information is obtained instead by analyzing the photon statistics and calculating the speckle contrast in single scattering patterns. This quantity, also referred to as the speckle visibility, is determined by the properties of the partially coherent beam and other experimental parameters, as well as the internal motions in the sample (dynamics). As a case study, Brownian dynamics in a low-density colloidal suspension is measured and an excellent agreement is found between correlation functions measured by X-ray photon correlation spectroscopy and the decay in speckle visibility with integration time obtained from the analysis presented here.