Uncovering human travel behavior is crucial for not only travel demand analysis but also ride-sharing opportunities. To group similar travelers, this paper develops a deep-learning-based approach to classify travelers’ behaviors given their trip characteristics, including time of day and day of week for trips, travel modes, previous trip purposes, personal demographics, and nearby place categories of trip ends. This study first examines the dataset of California Household Travel Survey (CHTS) between the years 2012 and 2013. After preprocessing and exploring the raw data, an activity matrix is constructed for each participant. The Jaccard similarity coefficient is employed to calculate matrix similarities between each pair of individuals. Moreover, given matrix similarity measures, a community social network is constructed for all participants. A community detection algorithm is further implemented to cluster travelers with similar travel behavior into the same groups. There are five clusters detected: non-working people with more shopping activities, non-working people with more recreation activities, normal commute working people, shorter working duration people, later working time people, and individuals needing to attend school. An image of activity map is built from each participant’s activity matrix. Finally, a deep learning approach with convolutional neural network is employed to classify travelers into corresponding groups according to their activity maps. The accuracy of classification reaches up to 97%. The proposed approach offers a new perspective for travel behavior analysis and traveler classification.