In this paper we present a learning-based trajectory prediction method for different road users, including vehicles, pedestrians, and cyclists. The model uses history position information of traffic agents, and predicts future positions of subjects within a finite horizon. Instead of developing different model architectures for different agent types, a generic model architecture is used to learn trajectory patterns. This common architecture is then trained using agent-specific datasets, providing individualized models for different agent types. We evaluate the model on the Lyft dataset-a public dataset collected by a set of autonomous vehicles-and compare its performance against extended Kalman filter (EKF) as a benchmark. Results indicate that the learning-based method outperforms the benchmark method and provides high accuracy predictions in 5-second prediction horizons across all agent types. We also show that the prediction accuracy on rarely-seen agents can be greatly improved using transfer learning,