This paper proposes a dynamic vehicle routing problem (DVRP) model with nonstationary stochastic travel times under traffic congestion. Depending on the traffic conditions, the travel time between two nodes, particularly in a city, may not be proportional to distance and changes both dynamically and stochastically over time. Considering this environment, we propose a Markov decision process model to solve this problem and adopt a rollout-based approach to the solution, using approximate dynamic programming to avoid the curse of dimensionality. We also investigate how to estimate the probability distribution of travel times of arcs which, reflecting reality, are considered to consist of multiple road segments. Experiments are conducted using a real-world problem faced by Singapore logistics/delivery company and authentic road traffic information.