This brief proposes a novel approach to apply reliability centered maintenance (RCM) on a distance protection system in a smart grid. In the proposed approach, instead of focusing on individual equipment, the RCM is implemented for the whole protection system. The approach helps to calculate the reliability of protection systems by overcoming the limitation of their hidden function and also considering the effect of all system equipment on reliability. Markov process and improved Jaya algorithm (IJA) are used to calculate the reliability and find the optimal maintenance plan. The proposed approach has been applied to real data of smart grid distance protection systems, which results in a higher level of reliability and less cost. Furthermore, in order to validate the suggested technique and its efficiency, the simulation is carried out using a real-time digital simulators and hardware-in-the-loop testing equipment known as opal-RT technology.