In this work, we propose an inversion scheme to tailor the chromatic response of an all-dielectric structure. To this end, we couple, through a previously defined objective functional involving the concept of color difference, a forward solver with an optimization algorithm. The former is based on the differential method, whereas the latter is based on particle swarm optimization. The optimal geometrical parameters of the structure that generates a specific color are obtained through the solution of an approximation problem. We illustrate the performance of our inversion scheme through examples and discuss its limitations and potential applications.