In a rotating machinery system, a single fault of one component often causes damage to other related components, thus inducing compound faults. Without compound fault data to train intelligent models, the realization of decoupling diagnosis and accurate identification of unseen compound faults is not only of great practical significance for the safety management of equipment operation and maintenance, but also remains a challenging topic. Considering some shortcomings in the current intelligent diagnosis of compound faults, as well as the relatedness and difference between different fault features in compound fault signals, a hybrid task-adapted experts-based multi-task attention network (HeMTAN) model is investigated in this paper, which can be used for identify single faults and unseen compound faults in mechanical transmission systems. Firstly, variational mode decomposition is combined with Hilbert-Huang transform to obtain time–frequency graphs of time series signal as model input, so as to better characterize different fault features. Secondly, a hybrid task-adapted expert module is designed to extract the common and some private feature information of different learning tasks from different multi-perspective, and then the important information related to the specific learning task is further mined by the constructed private feature attention-based densely connected module. Finally, the diagnosis results can be obtained by fusing the outputs of the classifier of the two learning tasks. The performance of the investigated HeMTAN model is analyzed by the gearbox compound fault dataset and rolling bearing compound fault dataset, and the results demonstrate that the investigated HEMTAN method has significantly improved diagnosis accuracy and generalization performance.