This paper proposes the summarization technique for the multimedia data present in the form of video, over the internet to provide a quick overview of the content present in it. This is very challenging task because finding the significant and useful portion of the video, needs to understand the content present in it. Moreover, the categories of the videos over the wide web are very diverse, like home videos, documentaries and sports videos etc. So, it makes video summarization more tough because of the unavailability of the prior knowledge. Currently, traditional hand crafted features have been utilized for video summarization, which fails to capture the information and content from all the scenes. To tackle this problem, we first find the scene boundaries using motion features. Then we pass the data to our proposed CNN architecture that gives us the frame level importance against each frame present in specific scene. The redundancy of the frames has been removed using the bidirectional LSTM. Experiments have been performed using the publically available dataset TVSUM50. Obtained results show that our proposed methodology outperforms the traditional feature based approaches in terms of relative F measure score.