The lack of tumor targeting ability and the ineffectiveness of single treatment modalities are two obstacles in the treatment of hepatocellular carcinoma (HCC). Herein, mesoporous silica nanoparticles (MSNs) with different sizes (MSN100 and MSN50) were synthesized and functionalized with polydopamine (PDA), hypericin B (HB), and arginine-glycine-aspartic acid (RGD) peptides for combined photothermal therapy (PTT) and photodynamic therapy (PDT) of HCC targeting αvβ3 integrin. MSN@PDA-RGD-HB were characterized by various techniques and showed selective drug release, high photothermal conversion efficiency, and reactive oxygen species (ROS) generation under laser irradiation. MSN100@PDA-RGD-HB exhibited higher cellular uptake and HCC toxicity than MSN50@PDA-RGD-HB in vitro, and specifically targeted HCC cells over normal cells. MSN100@PDA-RGD-HB demonstrated high biocompatibility in vivo, as confirmed by the hemolytic test and H&E staining. Importantly, MSN100@PDA-RGD-HB achieved a synergistic effect of PTT and PDT for HCC treatment in mice xenograft tumor model. Therefore, MSN100@PDA-RGD-HB has great potential for tumor therapy in the future.