In this paper, an adaptive switching control scheme for the HSV attitude systems with unknown control direction and actuator faults is designed, which is capable of solving the problem of system instability. The purpose of the adaptive switching strategy is to keep the closed loop system in negative feedback control, and thus the Nussbaum technology is applied. The crucial point is that an adaptive switching strategy is needed to keep the negative feedback control for closed loop system. To address this issue, the Nussbaum technology is applied. As for the system uncertainties and actuator fault, the adaptive estimator and radial basis neural network (RBFNN) is used for the estimation thereof. All signals of closed-loop system are bounded, which is proved by the Lyapunov theorem of stability. At last, the effectiveness of the proposed fault-tolerant scheme is proved by the the simulation results.