Protein arginine methyltransferases (PRMTs) catalyze the methylation of the terminal nitrogen atoms of the guanidino group of arginine of protein substrates. The aberrant expression of these methyltransferases is linked to various diseases, making them promising therapeutic targets. Currently, PRMT inhibitors are at different stages of clinical development, which validated their significance as drug targets. Structural Genomics Consortium (SGC) has reported several small fragment inhibitors as Class I PRMT inhibitors, which can be the starting point for rational drug development. Herein, we report the successful application of a fragment-based approach toward the discovery of selective Class I PRMT inhibitors. Structure-based ligand optimization was performed by strategic incorporation of fragment hits on the drug-like quinazoline core and subsequent fragment growth in the desired orientation towards identified hydrophobic shelf. A clear SAR was established, and the lead compounds 55 and 56 displayed potent inhibition of Class I PRMTs with IC50 values of 92 nM and 37 nM against PRMT4. We report the systematic development of potent Class I PRMT inhibitors with good potency and about 100-fold selectivity when tested against a panel of 31 human DNA, RNA, and protein lysine and arginine methyltransferases. These improved small molecules will provide new options for the development of novel potent and selective PRMT4 inhibitors.