For future quantum technologies, the combination of a long quantum state lifetime and an efficient interface with an external optical excitation is required. In solids, the former is, for example, achieved by individual spins, while the latter is found in semiconducting artificial atoms combined with modern photonic structures. One possible combination of the two aspects is reached by doping a single quantum dot, providing a strong excitonic dipole with a magnetic ion that incorporates a characteristic spin texture. Here, we perform four-wave mixing spectroscopy to study the system’s quantum coherence properties. We characterize the optical properties of the undoped CdTe quantum dot and find a strong photon echo formation that demonstrates a significant inhomogeneous spectral broadening. Incorporating the Mn²⁺ ion introduces its spin-5/2 texture to the optical spectra via the exchange interaction, manifesting as six individual spectral lines in the coherent response. The random flips of the Mn-spin result in a special type of spectral wandering between the six transition energies, which is fundamentally different from the quasi-continuous spectral wandering that results in the Gaussian inhomogeneous broadening. Here, the discrete spin-ensemble manifests in additional dephasing and oscillation dynamics.