In order to compensate errors of inertial measurement unit which is the core of rotational inertial navigation system, self-calibration is utilized as an effective way to reduce navigation error. Error model of navigation solution and initial alignment is used to establish the relationship between navigation errors and inertial measurement unit (IMU) errors. A second order damper is added to the vertical velocity channel to suppress the divergence and then the vertical velocity error can be regarded as an effective observation to estimate the error parameters. Since the accuracy of the self-calibration method is susceptible to the positioning error of gimbals, total least squares (TLS) method is utilized in identification of the error parameters. Experimental results show that all of the twenty-one error parameters can be estimated with the proposed rotation scheme. Compared to least squares (LS) method, TLS method can improve the position accuracy of 8 h by 46.2%.