At present, researches on brain fatigue recognition are still in the stage of single task and simple brain region network features, while researches on high-order brain functional network features and brain region state mechanisms during fatigue in multi-task scenarios are still insufficient, making it difficult to meet the needs of fatigue recognition under complex conditions. Therefore, this study utilized functional near-infrared spectroscopy (fNIRS) technology to explore the correlation and differences in the low-order and high-order brain functional network attributes of three task induced mental fatigue, and to explore the brain regions that have a major impact on mental fatigue. Self-training algorithms were used to identify the three levels of brain fatigue. The results showed that during the fatigue development, the overall connection strength of the endothelial cell metabolic activity and neural activity frequency bands of the low-order brain functional network first decreased and then increased, while the myogenic activity and heart rate activity frequency bands showed the opposite pattern. Network topology analysis indicated that from no fatigue to mild fatigue, the clustering coefficient of endothelial cell metabolic activity and myogenic activity frequency bands significantly decreased, while the characteristic path length of myogenic activity significantly increased; when experiencing severe fatigue, the small-world attribute of the neural frequency band significantly weakened. However, each frequency band maintained its small-world attribute, reflecting the self-optimization and adaptability of the network during the fatigue process. During mild fatigue, neuronal activity bands’ node degree, cluster coefficient, and efficiency rose in high-order brain networks, while low-order networks showed no significant changes. As fatigue progressed, the myogenic activity bands of high-order network properties dominated, but neural bands gained prominence in mild fatigue, approaching the level of myogenic bands in severe fatigue, indicating that brain fatigue orchestrated a shift from myogenic to neural dominance in frequency bands. In addition, during the process of fatigue, the four network attributes of the high-order network cluster composed of low-order nodes related to the prefrontal cortex region, left anterior motor region, motor assist region, and left frontal lobe eye movement region significantly increased, indicating that these brain regions had a significant impact on brain fatigue status. The accuracy of using both high-order and low-order features to identify fatigue levels reached 88.095%, indicating that the combined network features of both high-order and low-order fNIRS signals could effectively detect multi-level mental fatigue, providing innovative ideas for fatigue warning.