Video synthetic aperture radar (Video SAR) has sparked lots of research attention since it provides the ability of continuously observing and tracking the object of interest. Instead of tracking a target directly, in Video SAR, it is preferred to track its shadow due to the target's instable backscattering characteristic and defocused patterns. However, the current tracking algorithms do not work well since they always suffer from the interference caused by surrounding clutters and cannot locate the target precisely. To address these issues, we propose a robust shadow tracking algorithm for Video SAR in this letter. We equip our tracker with the spatial-temporal (ST) information and saliency-based detection mechanism (SD) against the distractions and background clutters. The proposed optimization formula could be solved efficiently using the alternating direction method of multipliers (ADMM) technique and fully carried in the Fourier domain at a low computational burden. Experiments have demonstrated that the proposed algorithm performs favorably against other state-of-the-art methods.