Studies show lots of advanced research on various data types such as image, speech, and text using deep learning techniques, but nowadays, research on video processing is also an emerging field of computer vision. Several surveys are present on video processing using computer vision deep learning techniques, targeting specific functionality such as anomaly detection, crowd analysis, activity monitoring, etc. However, a combined study is still unexplored. This paper aims to present a Systematic Literature Review (SLR) on video processing using deep learning to investigate the applications, functionalities, techniques, datasets, issues, and challenges by formulating the relevant research questions (RQs). This systematic mapping includes 93 research articles from reputed databases published between 2011 and 2020. We categorize the deep learning technique for video processing as CNN, DNN, and RNN based. We observe significant advancements in video processing between 2017 and 2020, primarily due to the advent of AlexNet, ResNet, and LSTM based deep learning techniques. The prominent fields of video processing research are observed as human action recognition, crowd anomaly detection, and behavior analysis. This SLR is a helpful guide for the researchers to explore the recent literature, available datasets, and existing deep learning techniques for video processing.