A diagnostic method for screw compressor faults using vibration signals was introduced and applied to a new type of helium oil-injected screw compressor in the 80L/h helium liquefier. The study revealed that the destruction of the oil film due to impurity particles could lead to rotor rubbing and poor meshing, compromising the reliability of these compressors. Based on frequency-domain analysis, the major vibration occurred at the meshing frequency of screw rotors and a series of its harmonics. The vibration of the characteristic frequency of bearing parts also indicated that the bearing had slight wear in the early stage. The vibration waveform’s time domain diagram exhibited clear clipping, and the axis trajectory shew disorder, which were common characteristic of rotor rubbing. When compressing helium with low molecular weight, the meshing clearance, tooth tip clearance, and end clearance of the rotor were significantly lower than those of traditional refrigerants or air media. Additionally, the thermal gap caused by thermoelastic deformation was smaller due to the large adiabatic index of helium. Therefore, rubbing faults between rotors and between rotors and casing were more likely to occur. The article recommended establishing a vibration signal monitoring system, optimizing the design of helium screw rotor profiles, and setting up reliability standards for screw compressor operation, such as limiting the vibration speed to ≤ 8 mm/s. Additionally, attention should be given to cleaning gas pipelines in cryogenic engineering and monitoring compressor vibration and noise signals during operation to prevent rotor damage due to particle impurities.