Wear is a failure factor that must be considered when alloys are in service, especially as moving parts. However, few current studies have addressed the tribological behavior of HEAs in service with tribo-pair systems consisting of materials in contact with them. Therefore, in this work, the tribological properties between Fe40Mn20Cr20Ni20 HEA with four typical counterfaces consisting of tribo-pair systems were investigated at room temperature, i.e., hard alloy (WC/Co), metal (GCr15 steel), ceramic (Si3N4) and plastic (POM). It was found that there were significant differences in the wear mechanisms between various tribo-pair systems. Moreover, adhesive wear dominates the wear extent of the alloys. The wear rate of the alloy after sliding with WC/Co or GCr15 even reached 20–30 times higher than that after sliding with Si3N4, indicating that the sliding contact of the alloy with WC/Co or GCr15 is not an optimal design during service. In addition, the frictional oxidation that occurs during sliding contact played good lubrication role. In tribo-pair systems consisting of Fe40Mn20Cr20Ni20 HEA with WC/Co, GCr15 or Si3N4, the average coefficient of friction exhibited a tendency to decrease with the normal force.