Phosphatase of regenerating liver (PRL) is an oncogenic protein that promotes tumor progression by directly binding to cyclin M (CNNM) membrane proteins and inhibiting their Mg2+ efflux activity. In this study, we have developed a high-throughput screening system to detect the interactions between PRL and CNNM proteins based on homogenous time-resolved fluorescence resonance energy transfer (HTR-FRET, HTRF). We optimized the tag sequences attached to the recombinant proteins of the CNNM4 CBS domains and PRL3 lacking the carboxyl terminal CAAX motif, and successfully detected the interaction by observing the FRET signal in the mixture of the tagged proteins and fluorophore-conjugated antibodies. Moreover, we performed compound library screening using this system and discovered several compounds that could efficiently inhibit the PRL-CNNM interaction. Characterization of one candidate compound revealed that it was relatively stable compared with thienopyridone, a known inhibitor of the PRL-CNNM interaction. The candidate compound can also inhibit PRL function in cells: suppression of CNNM-dependent Mg2+ efflux, and has sufficient in vitro drug metabolism and pharmacokinetic properties. Overall, these results demonstrate the effectiveness of this screening system for identifying novel inhibitors of the PRL-CNNM interaction, which could contribute to the development of novel anti-cancer drugs.