Graph matching (GM) plays a vital role in solving various computer vision tasks by establishing a node-to-node correspondence between graph-structured data. In this study, we propose a novel method based on affinity matrix for solving GM tasks. Specifically, a pruning strategy is proposed to remove inconsistent edges from an association graph that formulates the GM problem into a node ranking and selection problem. Moreover, an edge context information with respect to a cross-graph reference structure is proposed and combined with the local distance information to compute the affinities among the graphs’ nodes. Then, the re-weighted random walk technique is utilized to simulate random walks on the association graph and compute a quasi-stationary distribution in an iterative manner. Finally, the Hungarian algorithm is used to discretize the distribution and obtain an approximate matching between two graphs. The experimental results indicate that our proposed GM method is able to produce promising results in the presence of outliers and deformation noise as compared with other methods.