Deep learning has been recently applied to many problems in wireless communications including modulation classification and symbol decoding. Many of the existing end-to-end learning approaches demonstrated robustness to signal distortions like frequency and timing errors, and outperformed classical signal processing techniques with sufficient training. However, deep learning approaches typically require hundreds of thousands of floating points operations for inference, which is orders of magnitude higher than classical signal processing approaches and thus do not scale well for long sequences. Additionally, they typically operate as a black box and without insight on how their final output was obtained, they can't be integrated with existing approaches. In this paper, we propose a novel neural network architecture that combines deep learning with linear signal processing typically done at the receiver to realize joint modulation classification and symbol recovery. The proposed method estimates signal parameters by learning and corrects signal distortions like carrier frequency offset and multipath fading by linear processing. Using this hybrid approach, we leverage the power of deep learning while retaining the efficiency of conventional receiver processing techniques for long sequences. The proposed hybrid approach provides good accuracy in signal distortion estimation leading to promising results in terms of symbol error rate. For modulation classification accuracy, it outperforms many state of the art deep learning networks.