Based on unselectively, several side effects and drug resistance of available anticancer agents, the development and research for novel anticancer agents is necessary. In this study, a new series of quinazoline-4(3H)-one derivatives having a thiol group at position 2 of the quinazoline ring (8a-8 h) were designed and synthesized as potential anticancer agents. The Chemical structures of all compounds were characterized by 1H-NMR, 13C-NMR, and Mass spectroscopy. The antiproliferative activity of all derivatives were determined against two cancer cell lines (MCF-7 and SW480) and one normal cell lines (MRC-5) by the MTT method. Cisplatin, Erlotinib and Doxorubicin were used as positive controls. The results of in vitro screening showed that 8a with an aliphatic linker to SH group was the most potent compound with IC50 values of 15.85 ± 3.32 and 17.85 ± 0.92 µM against MCF-7 and SW480 cell lines, respectively. 8a indicated significantly better potency compared to Erlotinib in the MCF-7 cell line. The cytotoxic results obtained from testing compound 8a on the normal cell line, revealing an IC50 value of 84.20 ± 1.72 µM, provide compelling evidence of its selectivity in distinguishing between tumorigenic and non-tumorigenic cell lines. Structure–activity relationship indicated that the variation in the anticancer activities of quinazoline-4(3H)-one derivatives was affected by different substitutions on the SH position. Molecular docking and MD simulation were carried out for consideration of the binding affinity of compounds against EGFR and EGFR-mutated. The binding energy of compounds 8a and 8c were calculated at -6.7 and − 5.3 kcal.mol− 1, respectively. Compounds 8a and 8c were found to establish hydrogen bonds and some other important interactions with key residue. The DFT analysis was also performed at the B3LYP/6–31 + G(d, p) level for compounds 8a, 8c and Erlotinib. Compound 8a was thermodynamically more stable than 8c. Also, the calculated theoretical and experimental data for the IR spectrum were in agreement. The obtained results delineated that the 8a can be considered an appropriate pharmacophore to develop as an anti-proliferative agent.