This paper presents an efficient distributed real-time control scheme for a set of batteries in power grids to follow a day-ahead dispatch plan. The problem of controlling the batteries is cast as a multistep optimization problem aiming to minimize the error in dispatch following while satisfying lookahead constraints of the individual batteries. To achieve a computationally efficient solution, Lagrangian decomposition is used to distribute the task into subproblems pertaining to each battery. Finally, each subproblem is reformulated as a Markov Decision Process and solved with the fitted Q-iteration. Communication among subproblems is achieved via the common Lagrange multiplier value. Simulation results demonstrate that this approach has a similar cost performance to the state-of-the-art model predictive control solution, while achieving better performance in terms of computation time.