This paper considers a passive localization scenario relying on a single transmitter, several receivers, and multiple moving targets to be located. The so-called “passive” targets equipped with RFID reflectors are capable of reflecting the signals from the transmitter to the receivers. Existing approaches assume that the transmitter and receivers are synchronous or quasi-synchronous, which is not always realistic in practical scenarios. Hence, an asynchronous wireless network is considered, where different clock offsets are assumed at different receivers. We propose a centralized expectation-maximization-based passive localization method for asynchronous receivers (EMpLaR) by treating the clock offsets as hidden variables. Thereby, the proposed algorithm makes use of Taylor expansions to arrive at a closed-form maximization. Furthermore, to improve the robustness to link failures and to reduce the energy consumption, we propose a distributed localization approach based on average consensus formulation to locate the target at each receiver. By applying a quadratic polynomial approximation of the function on which consensus has to be reached, both the computational complexity and the communications overhead are significantly reduced. The Cramér-Rao bound of the target location is derived as a benchmark of our proposed algorithms. Our simulation results show that the proposed centralized and distributed EMpLaR algorithms match the Cramér-Rao bound and significantly improve the localization performance compared with the conventional methods.