Body gesture recognition can be applied not only to social security but also to rescue operations. In reality, body gesture can produce unique micro-Doppler signatures (MDSs), which can be used for identification. In this paper, we first acquired the echo signals of four body gestures via a Ka-band dual polarization radar system under different angles and distances. The four gestures are respectively swinging arm up and down, swinging arm left and right, nodding, and shaking head. Then, time-frequency spectrograms were obtained by short-time Fourier transform, from which we can see that different body gestures have different polarimetric MDSs. Finally, we propose to classify four body gestures using the deep convolutional neural network (DCNN) method. It is shown that by combining HH and HV polarizations, about 92.7% recognition rate is achieved while only about 77.5% and 89.3% rates are obtained by using single HH polarization and single HV polarization, respectively.