Fucoxanthin (FX), a non-provitamin-A carotenoid, is a well-known major xanthophyll contained in edible brown algae. The nanoencapsulation of FX was motivated due to its multiple activities. Here, nano-encapsulated-FX (nano-FX) was prepared according to our early method by using whey protein and flaxseed gum as the biomacromolecule carrier material, then in vivo antitumor effect and mechanism of nano-FX on xenograft mice were investigated. Thirty 4-week-old male BALB/c nude mice were fed adaptively for 7 days to establish xenograft tumor model with Huh-7 cells. The tumor-bearing mice consumed nano-FX (50, 25, and 12.5 mg kg−1) and doxorubicin hydrochloride (DOX, 1 mg kg−1) or did not consume (Control) for 21 days, n = 6. The tumor inhibition rates of nano-FX were as high as 54.67 ± 1.04 %. Nano-FX intervention promoted apoptosis and induced hyperchromatic pyknosis and focal necrosis in tumor tissue by down-regulating the expression of p-JNK, p-ERK, PI3Kp85α, p-AKT, p-p38MAPK, Bcl-2, CyclinD1 and Ki-67, while up-regulating the expression of cleaved caspase-3 and Bax. Nano-FX inhibited tumor growth and protected liver function of tumor bearing mice in a dose-dependent manner, up-regulate the level of apoptosis-related proteins, inhibit the MAPK-PI3K/Akt pathways, and promote tumor cell apoptosis.