Rosa rugosa, a valuable ornamental plant species, has high aromatic, ornamental and medicinal values. Diversification of flower color is one of the main purposes for ornamental horticultural breeding, especially the red phenotype. Understanding the molecular mechanism of red color formation regulated by O-methyltransferase (OMT) during anthocyanin biosynthesis is necessary. In this study, 38 candidate members belonging to the CCoAOMT (11) and COMT (27) subfamilies were systematically identified with average lengths of 227 and 360 amino acids, respectively. Two subfamily members had different chromosome distribution characteristics, exon/intron structures, and conserved motifs, consistent with the phylogenetic analysis result of 257 OMT members from six species. Cis-elements in the promoter region of RrCCoAOMTs were mainly involved in hormone and stress response. RrCCoAOMT1 was mainly expressed in petals, and its expression level was significantly decreased in red flowers with high relative percentage of cyanidin. Moreover, RrCCoAOMT1 was separated from other RrCCoAOMTs based on the multiple sequence alignment analysis. Silent expression of RrCCoAOMT1 caused the formation of red/deep-pink phenotypes on purple- and pink-flowered germplasms with an increase in the ratio of Cy3G5G to Pn3G5G. Overall, RrCCoAOMT1 was involved in the modification of red petal color in R. rugosa via changing the ratio of Cy3G5G to Pn3G5G, and may be transcriptionally regulated by MYB transcription factors. These findings provided a theoretical foundation for further studies on the regulation mechanism of red phenotype formation in R. rugosa, and provided important gene resources for flower color improvement breeding.