Objective Hepatocellular carcinoma is a malignant disease occurring in the liver and is one of the main causes of death in cancer patients. Tumor cells are the main components of tumors and have a strong proliferative capacity. They are easily transferred to other parts of the body and can produce harmful substances that destroy the normal organ structure and endanger human life and health. In this study, we investigate the effect of circFOXO3 on the proliferation and invasion of hepatocellular carcinoma cells and its possible mechanism. Methods Human hepatocellular carcinoma cells BEL-7404, Hep G2, Hep 3B2.1–7, HuH-7, Li-7, and human normal hepatocytes HHL-5 were selected, and the expression level of circFOXO3 in the cell lines was determined by qRT-PCR. The cell line with low circFOXO3 expression level (HuH-7 cells) was used for follow-up experiments. HuH-7 liver cancer cells were divided into the control group (normal cultured), circFOXO3-NC group (transfected with circFOXO3 negative control), circFOXO3 mimic group (transfected with circFOXO3 mimic), PI3K activator group (20 μmol/L PI3K activator 740Y-P), and circFOXO3 mimic + PI3K activator group (transfected with circFOXO3 mimic + treated with PI3K activator 740Y-P). The qRT-PCR method was used to determine the expression level of circFOXO3 in HuH-7 liver cancer cells in each group, WB was used to detect the expression of apoptosis, invasion, and phosphoinositide 3-kinase/protein kinase B (PI3K/Akt) pathway related proteins in HuH-7 liver cancer cells in each group, the CCK-8 method was used to determine the viability of HuH-7 liver cancer cells in each group, flow cytometry was used to determine the apoptotic ability of HuH-7 liver cancer cells in each group, the transwell chamber experiment was used to determine the invasion ability of HuH-7 liver cancer cells in each group, and the scratch test was used to determine the migration ability of HuH-7 liver cancer cells in each group. Results circFOXO3 showed low expression in liver cancer cells; compared with the control group, the circFOXO3 expression and apoptosis rate of HuH-7 liver cancer cells in the circFOXO3 mimic group were significantly increased (P < 0.05) and the PI3K/Akt pathway-related protein expression, cell viability, invasion, and migration abilities were significantly reduced (P < 0.05); the apoptosis rate of HuH-7 liver cancer cells in the PI3K activator group was significantly reduced (P < 0.05) and the PI3K/Akt pathway related protein expression, cell viability, invasion and migration abilities were significantly increased (P < 0.05). Compared with the circFOXO3 mimic group, the apoptosis rate of HuH-7 liver cancer cells in the circFOXO3 mimic + PI3K activator group was significantly reduced (P < 0.05) and the PI3K/Akt pathway-related protein expression, cell viability, invasion and migration abilities were significantly increased (P < 0.05). Conclusion Highly expressed circFOXO3 can inhibit the proliferation and invasion of HuH-7 liver cancer cells, which may be achieved by inhibiting the PI3K/Akt pathway.