Neural networks have enabled many new and interesting applications in a wide variety of domains. However, deep learning models are very computationally expensive and outsourcing the computation to the Cloud creates privacy concerns. Homomorphic encryption (HE) allows computation on encrypted data, thus preserving its privacy. Common neural networks architectures, such as CNNs and fully connected networks, have been adapted for HE. However, there is very little work on recurrent neural networks (RNNs). Existing solutions for RNNs over encrypted data require interaction between the data owner and the Cloud. In this paper, we present parallel RNN blocks, an RNN architecture that can be run on encrypted data without client-server interaction. We describe our proposed approach and evaluate it on a real-world dataset of online product reviews and IMDb movie reviews. Our results are promising; we can achieve 88.8% F1 score on the product reviews. The model generalizes well to the IMDb data set with a 74.36% F1 score using our proposed architecture. The performance is within 3 percentage points of our baseline.