Phenotypic plasticity, the ability of an organism to display different phenotypes across environments, is widespread in nature. Plasticity aids survival in novel environments. Herein, we review studies from yeast that allow us to start uncovering the genetic architecture of phenotypic plasticity. Genetic variants and their interactions impact the phenotype in different environments, and distinct environments modulate the impact of genetic variants and their interactions on the phenotype. Because of this, certain hidden genetic variation is expressed in specific genetic and environmental backgrounds. A better understanding of the genetic mechanisms of phenotypic plasticity will help to determine short- and long-term responses to selection and how wide variation in disease manifestation occurs in human populations.