BackgroundCancer has been one of the leading causes of mortality in this era. Ruta angustifolia L. Pers has been traditionally used as an abortifacient, antihelmintic, emmenagogue and ophthalmic. In Malaysia and Singapore, the local Chinese community used it for the treatment of cancer.MethodsIn this study, the methanol and fractionated extracts (hexane, chloroform, ethyl acetate and water) of R. angustifolia were tested for its cytotoxicity using the sulforhodamide (SRB) cytotoxicity assay against HCT-116, A549, Ca Ski and MRC5 cell lines. Chemical isolation was carried out by using the high performance liquid chromatography (HPLC) and the isolated compounds were tested for its cytotoxicity against A549 cell line. Cellular and nuclear morphological changes were observed in the cells using phase contrast microscopy and Hoechst/PI fluorescent staining. The externalisation of phosphatidylserine was observed through FITC-labelling Annexin V/PI assay whilst DNA fragmentation was observed through the TUNEL assay. Other indication of apoptosis occuring through the mitochondrial pathway were the attenuation of mitochondrial membrane potential and increase in ROS production. Activation of caspase 9 and 3 were monitored. Western blot analysis was done to show the expression levels of apoptotic proteins.ResultsThe chloroform extract (without chlorophyll) exhibited the highest cytotoxic activity with IC50 of 10.1 ± 0.15 μg/ml against A549 cell line. Further chemical investigation was thus directed to this fraction which led to the isolation of 12 compounds identified as graveoline, psoralen, kokusaginine, methoxysalen, bergapten, arborinine, moskachan B, chalepin, moskachan D, chalepensin, rutamarin and neophytadiene. Among these compounds, chalepin exhibited excellent cytotoxicity against A549 cell line with an IC50 value of 8.69 ± 2.43 μg/ml (27.64 μM). In western blot analysis, expression of p53, truncated Bid, Bax and Bak while the anti-apoptotic proteins Bcl-2, survivin, XIAP, Bcl-XL,cFLIP decreased in a time-dependent manner when A549 cells were treated with 36 μg/ml of chalepin. In addition, the level of PARP was found to decrease.ConclusionHence these findings indicated that chalepin-induced cell death might involve the intrinsic mitochodrial pathway resulting in the upregulation of pro-apoptotic proteins and downregulation of anti-apoptotic proteins. Thus, chalepin could be an excellent candidate for the development of an anticancer agent.Electronic supplementary materialThe online version of this article (doi:10.1186/s12906-016-1368-6) contains supplementary material, which is available to authorized users.