There is intense interest in uncovering design rules that govern the formation of various structural phases as a function of chemical composition in multi-principal element alloys (MPEAs). In this paper, we develop a machine learning (ML) approach built on the foundations of ensemble learning, post hoc model interpretability of black-box models, and clustering analysis to establish a quantitative relationship between the chemical composition and experimentally observed phases of MPEAs. The originality of our work stems from performing instance-level (or local) variable attribution analysis of ML predictions based on the breakdown method, and then identifying similar instances based on k-means clustering analysis of the breakdown results. We also complement the breakdown analysis with Ceteris Paribus profiles that showcase how the model response changes as a function of a single variable, when the values of all other variables are fixed. Results from local model interpretability analysis uncover key insights into variables that govern the formation of each phase. Our developed approach is generic, model-agnostic, and valuable to explain the insights learned by the black-box models. An interactive web application is developed to facilitate model sharing and accelerate the design of MPEAs with targeted properties.