Metallodrugs exhibiting distinct mechanisms of action compared with cisplatin hold promise for overcoming cisplatin resistance and improving the efficacy of anticancer drugs. In this study, a new series of rhodium (Rh)(III) complexes containing tris(triphenylphosphine)rhodium(I) chloride [(TPP)3RhCl] (TPP = triphenylphosphine, TPP=O = triphenylphosphine oxide) and 8-hydroxyquinoline derivatives (H-XR1–H-XR4), namely [Rh(XR1)2(TPP)Cl]·(TPP=O) (Yulin Normal University-1a [YNU-1a]), [Rh(XR2)2(TPP)Cl] (YNU-1b), [Rh(XR3)2(TPP)Cl] (YNU-1c), and [Rh(XR4)2(TPP)Cl] (YNU-1d), was synthesized and characterized via X-ray diffraction, mass spectrometry and IR. The cytotoxicity of the compounds YNU-1a–YNU-1d in Hep-G2 and HCC1806 human cancer cell lines and normal HL-7702 cell line was evaluated. YNU-1c exhibited cytotoxicity and selectivity in HCC1806 cells (IC50 = 0.13 ± 0.06 μM, selectivity factor (SF) = 384.6). The compounds YNU-1b and YNU-1c, which were selected for mechanistic studies, induced the activation of apoptotic pathways and mitophagy. In addition, these compounds released cytochrome c, cleaved caspase-3/pro-caspase-3 and downregulated the levels of mitochondrial respiratory chain complexes I/IV (M1 and M4) and ATP. The compound YNU-1c, which was selected for in vivo experiments, exhibited tumor growth inhibition (58.9 %). Importantly, hematoxylin and eosin staining and TUNEL revealed that HCC1806 tumor tissues exhibited significant apoptotic characteristics. YNU-1a–YNU-1d compounds are promising drug candidates that can be used to overcome cisplatin resistance.