Fall is a challenging task that poses a great danger to the elderly person’s health as they carry out their daily routines and activities and could lead to serious injuries, long hospitalization, or even death. One of the key solutions to this important problem is the prompt and automatic detection of their fall motion, among other activities, so that immediate help can be rendered to avoid further complications. Cameras and other wearable sensors are the conventional means employed to monitor the daily activities performed by elderly people. However, recently radar sensing technology has been favored largely due to its short-range and velocity estimation. And most importantly, their ability to remotely capture the human activity motion electromagnetic wave signal without infringing on the user’s privacy. To this end, we present an elderly people fall detection system that integrates the radar signal micro-Doppler features extracted with the convolutional neural networks (CNNs). The features were extracted using Alex-Net, VGG-16-Net, and VGG-19-Net pre-trained models. We adapted the canonical correlation analysis (CCA) algorithm by proposing a channel attention network to fuse the extracted features effectively. The channel attention module employed a series of convolutional filters to determine the most discriminative features to focus on and discard the redundant ones. The fused features are classified using an SVM classifier. Our proposed method achieved the best performance against the state-of-the-art approaches. Specifically, our approach attained 99.77% test accuracy, which is about a 2% to 4 % increase compared to the recent state-of-the-art approaches.