Hepatocellular carcinoma (HCC) is the dominant subtype of liver cancer with very low survival rate but the chemotherapy for HCC is still in grey zone due to the limited efficacy and high toxicity profile of approved drugs raising the heavy demand on drug development for HCC. The study aimed to establish a desirability based quantitative structure activity relation (QSAR) model to predict the activity of chemical compounds against one of HCC cell line (HepG2). Different support vector machine (SVM) models were constructed and ensembled to 10 virtual screening protocols. These protocols were validated by an external dataset in combination with decoys as interference. Results showed that ensemble models exhibited improved area under the Receiver Operating Characteristic Curve (ROC), sensitivity, and specificity compared to base models in training and test set. When being validated for virtual screening to recover known active molecules in the mixture with known inactive and decoy compounds, all virtual screening protocols have good performance with good Boltzmann-Enhanced Discrimination of ROC (BEDROC) and enrichment factor (EF). The best protocol with BEDROC of 0.63 and EF of 29.55 was suitable for further screening of active compounds against HepG2 cell line. HIGHLIGHTS Ensemble quantitative structure-activity relationship (QSAR) models improved the performance of single models Cytotoxicity prediction was transformed to desirability score as a general output for easily integration in multi-objective optimization Virtual models for screening of HepG2 cytotoxicity compounds were well validated GRAPHICAL ABSTRACT