Time series classification (TSC) problem has been a significantly attractive research problem for decades. A large number of models with various types of features have been proposed. However, with the rapid development of new applications, like IoT and intelligent manufacturing, the time series data from different industries and applications are constantly emerging. To classify these data accurately, data scientists face the challenges of 1) how to select the optimal features and classification models and 2) how to interpret the results. To tackle these two challenges, in this paper, we propose a heterogeneous feature ensemble network, named FEnet. Multiple features, including time-domain, frequency-domain, and so on, are combined to build the model so that it can deal with the diversity of the data characteristics. Furthermore, to improve the interpretability, we propose a two-level attention mechanism. Finally, we propose two model optimization strategies to enhance classification accuracy and efficiency. Extensive experiments are conducted on real datasets and the results verify the accuracy, operation efficiency, and interpretability of FEnet.