Inertial measurement unit (IMU)-based joint angle estimation is an increasingly mature technique that has a broad range of applications in clinics, biomechanics and robotics. However, the deviations of different IMUs’ reference frames, referring to IMUs’ individual orientations estimating errors, is still a challenge for improving the angle estimation accuracy due to conceptual confusion, relatively simple metrics and the lack of systematical investigation. In this paper, we clarify the determination of reference frame unification, experimentally study the time-varying characteristics of reference frames’ deviations and accordingly propose a novel method with a comprehensive metric to unify reference frames. To be specific, we firstly define the reference frame unification (RFU) and distinguish it with drift correction that has always been confused with the term RFU. Secondly, we design a mechanical gimbal-based experiment to study the deviations, where sensor-to-body alignment and rotation-caused differences of orientations are excluded. Thirdly, based on the findings of the experiment, we propose a novel method to utilize the consistency of the joint axis under the hinge-joint constraint, gravity acceleration and local magnetic field to comprehensively unify reference frames, which meets the nonlinear time-varying characteristics of the deviations. The results on ten human subjects reveal the feasibility of our proposed method and the improvement from previous methods. This work contributes to a relatively new perspective of considering and improving the accuracy of IMU-based joint angle estimation.