We demonstrate a novel method for the automatic modulation classification based on a deep learning autoencoder network, trained by a nonnegativity constraint algorithm. The learning algorithm aims to constrain the negative weights, learns features that amount to a part-based representation of data, and disentangles a more meaningful hidden structure. The performance of this algorithm is tested on the fourth-order cumulants of the modulated signals. The results indicate that the autoencoder with nonnegativity constraint (ANC) improves the sparsity and minimizes the reconstruction error in comparison with the conventional sparse autoencoder. The classification accuracy of an ANC based deep network shows improved accuracy under limited signal length and fading channel.