In previous studies, a visually servoed paired structured light system (ViSP) was proposed to estimate structural displacement with high accuracy at low cost. Although the performance of the system has been verified from various simulations and experimental tests, it has a limitation that high‐speed dynamic displacement cannot be measured properly due to the relatively low sampling rate. To speed up ViSP, in this paper, an inertial measurement unit (IMU) with a high sampling rate is concurrently used with ViSP in a Kalman filtering framework. By combining measured accelerations and angular velocities from the IMU with the estimated 6‐DOF displacement from ViSP, high‐speed structural movement can be monitored with high accuracy. In the proposed data fusion method, named ViSP‐IMU, random biases of the IMU are calibrated by using the estimated displacement results from ViSP‐IMU. We perform experimental tests and verify the efficacy of the proposed algorithm as a promising structural displacement estimation method which guarantees high accuracy and a high sampling rate. Also, the proposed bias compensation method can effectively reduce the displacement estimation error by minimizing the difference between estimated displacements from the IMU and ViSP‐IMU. Copyright © 2016 John Wiley & Sons, Ltd.