This paper presents real-time monitoring data and analysis results of the non-stationary vibrations of an operational wind turbine. The advanced time-frequency spectrum analysis reveals varied non-stationary vibrations with timevarying frequencies, which are correlated with certain system natural modes characterized by finite element analysis. Under the effects of strong wind load, the wind turbine system exhibits certain resonances due to blade passing excitations. The system also exhibits certain instabilities due to the coupling of the tower bending modes and blade flapwise mode with blade passing excitations under the variation of wind speed. An analytical model is used to elaborate the non-stationary and instability phenomena observed in experimental results. The properties of the nonlinear instabilities are evaluated by using Lyapunov exponent estimation.