Feature point matching between two images is a fundamental and important process in machine vision. In many cases, mismatches are inevitable, and removing mismatches is an indispensable task. The existing methods attempt to find comprehensive constraints or sampling model to achieve better performance, which results in the increasingly complexity and may cause the weakness of the generality and scalability. To address this issue, a method called Local Topology similarity guided probabilistic Sampling consensus (LTS) is proposed. It constructs a topological network, then quantifies the mismatch probability in a concise approach based on comparing the topological relationship with neighbourhoods. Then, it detects and removes the mismatches by sampling guided by the mismatch probability. Compared with the state-of-the-art methods, LTS has an excellent performance in accuracy and robustness.