Agarwood essential oil (AEO) holds significant medicinal and economic value. While its antitumor activity is reported in the literature, the active ingredients remain unexplored. In this study, the chemical composition of AEO extracted by steam distillation (SD) and fungal fermentation-assisted SD (FFSD) was evaluated by gas chromatography-mass spectrometry (GC-MS). The antitumor activity was evaluated by the 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide (MTT) assay. Subsequently, the key active ingredients and targets of antitumor activity were evaluated using grey relational analysis and network pharmacology. The relative content of active ingredients in the AEOs extracted from different Aquilaria trees was compared. GC-MS tentatively identified 32 compounds, primarily sesquiterpenoids and aromatic compounds. Eight components with relative content differences were screened using orthogonal partial least squares-discriminant analysis (OPLS-DA), indicating that α-costal, guaiol, β-santalol may play important roles in the antitumor activity. Both SD and FFSD-extracted AEOs demonstrated the ability to inhibit the proliferation and survival of four cancer cell lines, with slightly stronger inhibitory effects observed in AEO extracted by SD. α-Costal, aristolone, 10-epi-γ-eudesmol, γ-eudesmol, agarospirol, hinesol, guaiol, β-santalol, and 6-isopropenyl-4,8a-dimethyl-1,2,3,5,6,7,8,8a-octahydro-naphthalen-2-ol were antitumor ingredients of AEO. The relative content of total active ingredients in agarwood (Aquilaria malaccensis) essential oil was higher, suggesting higher efficacy. This study confirmed the antitumor activity of AEO and identified its active ingredients. These findings provide a basis for further research on the antitumor mechanism of AEO.