Sudden changes of flame shape are an undesired, yet poorly understood feature of swirl combustors used in gas turbines. The present work studies flame shape transition mechanisms of a bistable turbulent swirl flame in a gas turbine model combustor, which alternates intermittently between an attached V-form and a lifted M-form. Time-resolved velocity fields and two-dimensional flame structures were measured simultaneously using high-speed stereo-particle image velocimetry (PIV) and planar laser-induced fluorescence of OH (OH-PLIF) at 10 kHz. The data analysis is performed using two novel methods that are well adapted to the study of transient flame shape transitions: First, the linear stability analysis (LSA) of a time-varying mean flow and second, the recently proposed spectral proper orthogonal decomposition (SPOD). The results show that the transitions are governed by two types of instability, namely a hydrodynamic instability in the form of a precessing vortex core (PVC) and a thermoacoustic (TA) instability. The LSA shows that the V-M transition implies the transient formation of a PVC as the result of a self-amplification process. The V-M transition, on the other hand, is induced by the appearance of a TA instability that suppresses the PVC and thereby modifies the flowfield such that the flame re-attaches at the nozzle. In summary, these results provide novel insights into the complex interactions of TA and hydrodynamic instabilities that govern the shape of turbulent swirl-stabilized flames.