This article investigates an encoder-based relative attitude observation method for self-calibration in a dual-axis rotational inertial navigation system (RINS). First, the calibration error of the gyroscope parameters caused by the velocity and position observations in systematic calibration is analyzed. Second, a relative attitude observation method utilizing photoelectric encoders for gyroscope parameter estimation is proposed. We apply the proposed method in our designed dual-axis RINS to verify its effectiveness in this article. Experimental results show that the calibration accuracy of the gyroscope can be improved to 0.0005 in bias, and 0.98 ppm in scale factor using the proposed method. The proposed self-calibration method can be utilized as a high-accuracy self-calibration method in the dual-axis RINS when the vehicle is in static condition.