In this study, a simplified mechanical model is proposed based on the failure characteristics of a gypsum pillar–roof system obtained through in situ investigation. Then, a cusp catastrophe model for the gypsum pillar–roof system is established based on the given stress–strain relations of the gypsum rock. Using this model, the instability mechanism for the gypsum pillar–roof system is investigated. The results of the analysis indicate that the stress–strain relation of gypsum rock derived from damage mechanics theory can accurately describe its strain-softening behavior after the peak stress and that increases in the relative humidity around a pillar or both a pillar and the roof bed can increase the stiffness ratio of the support system and consequently result in a more stable support system. However, if the relative humidity around the roof bed increases, the stiffness ratio of the system decreases, increasing the possibility of catastrophic failure of the support system. The case study revealed that the influence of the relative humidity on the support system is significant, higher relative humidity in the mine’s atmosphere increases the probability of instability in the support system, and the influence of the relative humidity change around the pillar on system stability is greater than that around the roof bed.