The safety of energy infrastructure is currently attracting more and more attention. Once accidents happened or improper incident handling, uncontrolled release of huge energy will lead catastrophic consequences. The manuscript takes the hydrogen refueling station as a typical representative of the energy infrastructure. Once the hydrogen in hydrogen refueling stations (HRSs) leaks, it easily forms a combustible cloud and can explode by encountering a heat source. At such a time, effective emergency management can prevent and reduce the impact of accidents. In this paper, the “scene-task-ability” scenario construction method is first applied to HRS leakage and explosive accidents and used to illustrate the risk evolution process during the different stages of an accident. This method separates the emergency tasks of various emergency responders and evaluates their emergency response ability. The values of prevention, preparation, response and recovery are coupled using the time-ordered weighted averaging and time-ordered weighted geometric averaging hybrid operator, in order to derive a systematic and comprehensive evaluation of the safe emergency management of HRSs. Finally, based on the results of the emergency response ability evaluation, targeted and effective suggested measures are proposed to improve the emergency management level of emergency responders.