We investigated the spin–orbit torque (SOT) switching of the single CoTb layer with or without the heavy metals (Pt and W) in different thickness to identify the self-torque effects generated in the CoTb layer. The generation of self-torque heavily depends on the thickness of CoTb. The SOT switching was not observed for a single CoTb layer with 3 nm in thickness, suggesting self-torque may be quite weak in a very thin film. A deterministic SOT switching can be obtained for a single 9 nm CoTb layer. The amplitude of the self-torque generated in CoTb is comparable to the Co/Pt case with the same switching polarity as the Co/Pt one. When 3 nm CoTb was deposited on Pt or W underlayer, the Jc is smaller in W/CoTb than that of Pt/CoTb, which can be attributable to a high spin hall angle of W and negligible self-torque of CoTb. On the other hand, when the CoTb is increased to 9 nm, the Jc of W/CoTb is significantly increased and becomes higher than that of Pt/CoTb. The enhanced Jc of W/CoTb may result from the opposite sign of spin orbit torque generated in W and CoTb. Since the self-torque generated in 9 nm CoTb becomes substantial, the competition of spin orbit torque between W and CoTb occurs, leading to a high Jc. Our findings may shed light to utilize self-torque induced in rare earth-transition metal alloy for new design of SOT devices.