The widespread propagation of numerous rumors and fake news have seriously threatened the credibility of microblogs. Previous works often focused on maintaining the previous state without considering the subsequent context information. Furthermore, most of the early works have used classical feature representation schemes followed by a classifier. We investigate the rumor detection problem by exploring different Deep Learning models with emphasis on considering the contextual information in both directions: forward and backward, in a given text. The proposed system is based on Bidirectional Long Short-Term Memory with Convolutional Neural Network, effectively classifying the tweet into rumors and non-rumors. Experimental results show that the proposed method outperformed the baseline methods with 86.12% accuracy. Furthermore, the statistical analysis also shows the effectiveness of the proposed model than the comparing methods.