Anomalous trajectory detection which plays an important role in taxi fraud detection and trajectory data preprocessing is a crucial task in trajectory mining fields. Traditional anomalous trajectory detection methods which utilize density and isolation approaches mainly focus on the differences of a new trajectory and the historical trajectory dataset. Although these methods can capture the particular characteristics of trajectories, they still suffer from the following two disadvantages. (1) These methods cannot capture the sequential information of the trajectory well. (2) These methods only concentrate on the given source and destination which may lead to data sparsity issues. To overcome above shortcomings, we propose a novel method called Anomalous Trajectory Detection using Recurrent Neural Network (ATD-RNN) which characterizes the trajectory by learning the trajectory embedding. The trajectory embedding can capture the sequential information of the trajectory and depict the internal characteristics between anomalous and normal trajectory. To address the potential data sparsity problem, we enlarge the dataset between a source and a destination by taking the relevant trajectories into consideration. Extend experiments on real-world datasets validate the effectiveness of our method.