The distance-based critical node problem involves identifying a subset of nodes in a graph such that the removal of these nodes leads to a residual graph with the minimum distance-based connectivity. Due to its NP-hard nature, solving this problem using exact algorithms has proved to be highly challenging. Moreover, existing heuristic algorithms are typically time-consuming. In this work, we introduce a fast tri-individual memetic search approach to solve the problem. The proposed approach maintains a small population of only three individuals during the whole search. At each generation, it sequentially executes an inherit-repair recombination operator to generate a promising offspring solution, a fast betweenness centrality-based late-acceptance search to find high-quality local optima, and a simple population updating strategy to maintain a healthy population. Extensive experiments on both real-world and synthetic benchmarks show our method significantly outperforms state-of-the-art algorithms. In particular, it can steadily find the known optimal solutions for all 22 real-world instances with known optima in only one minute, and new upper bounds on the remaining 22 large real-world instances. For 54 synthetic instances, it finds new upper bounds on 36 instances, and matches the previous best-known upper bounds on 15 other instances in ten minutes. Finally, we investigate the usefulness of each key algorithmic ingredient.