Robot-assisted rehabilitation for three-degree-of-freedom joints, such as hip and ankle, is significant for patients with motor function injuries. The control of such robots involves attitude control. To adapt to different disease stages, multi-mode hybrid control is considered to be one of the best choices. Passive mode is based on trajectory tracking control, while active mode is based on field-based assist-as-needed (AAN) control. The key to AAN control is the solution of the closest attitude point. However, the attitude point belongs to a special orthogonal group SO(3), and its topology is completely different from Euclidean space, which causes difficulties to the solution. Both passive and active control methods are affected by the inaccuracy of model parameters and external disturbances. Therefore, this paper proposes a multi-mode hybrid control method on SO(3). First, the expressions of trajectory tracking and contour tracking errors are proposed. To solve the contour tracking error, a feedback linearization algorithm based on sliding surface was used. A radial basis function (RBF) neural network was used for adaptive compensation. Subsequently, a controller for different modes was designed and its stability was analysed. Experiments were conducted using a hip exoskeleton, and the results verified the effectiveness of the proposed control method.