Fault diagnosis plays a critical role in ensuring the reliability and safety of industrial systems. Despite the success of semi-supervised learning in fault diagnosis, challenges persist in dealing with imbalanced datasets and out-of-distribution (OOD) samples, particularly in industrial settings where fault incidents are rare. The rarity of fault incidents in industrial contexts leads to dataset imbalances, while changes in conditions may introduce OOD samples. To address these challenges, this study proposes a novel semi-supervised fault diagnosis method that simultaneously considers the imbalanced distribution of fault categories and OOD samples. Firstly, the semi-supervised framework has been presented, which combines pseudo-labeling and consistency regularization techniques. Then, the self-adaptive threshold has been devised to balance confidence levels among fault classes using dataset-specific and class-specific thresholds, thereby enhancing the model's robustness in real-world scenarios. Additionally, the OOD detection strategy based on contrastive learning has been proposed, wherein predictions from two different views of each sample estimate its “likelihood” of belonging to in-distribution (ID) or OOD categories. This strategy improves system safety by alerting operators to potential anomalies. Comprehensive experimental validations and comparisons were conducted on two rotational machinery fault datasets, demonstrating that the proposed method achieves superior performance compared to the alternative methods and is applicable for practical application.