In this paper we propose a deep learning model, which we call step attention, for pedestrian trajectory prediction. The proposed model has a special architecture which consists of recurrent neural networks, convolutional neural networks, and an augmented attention mechanism. Rather than developing architectures to model factors that may affect the walking behavior, the proposed model learns trajectory patterns directly from input sequences. We evaluate the performance of the step attention model using TrajNet–a publicly available benchmark dataset collected from diverse real-world crowded scenarios. We compare the performance of step attention with three existing state-of-the-art algorithms, including social LSTM, social GAN, and occupancy LSTM on the TrajNet benchmark dataset. Our experiments show that the average displacement error (ADE) of step attention for a 4.8-seconds-long prediction horizon is about 0.53 m. The final displacement error (FDE) is 1.72 m. Both average and final displacement errors are favorable compared to the benchmark methods. We conduct a second set of experiments using data collected from a four-way intersection through roadside camera sensor platforms to study the effectiveness of the proposed model in uncrowded environments. On this dataset, the proposed model has an ADE of 0.74 m and a FDE of about 1.40 m for a 6-seconds-long prediction horizon. A complementary set of experiments is conducted to further investigate model performance in a real-world intersection. In these experiments, the model gains an ADE/FDE of 0.76/1.70 m. The proposed model also produces accurate prediction results on different scenarios composed of different walking patterns (e.g., straight and curvy) and different environments (e.g., sidewalk and street). The average displacement errors on all investigated datasets are within the length of a single step of an adult. The experiments also indicate that the displacement error grows almost linearly with the prediction horizon.