BackgroundEndogenous substances of krill oil (KO) are lipophilic in nature and have clinical significance viz. DHA/EPA, phospholipids and astaxanthin. To improve the nanodispersibility of endogenous substances of KO, a self-nanoemulsifying system (SNE) was developed.ResultsTernary phase behaviour of KO was explored in ethanol (ET); propylene glycol, (PG); and PEG600 using Tween80 and Tween20 as surfactants. PEG600 induced the self-nanoemulsification of KO and yielded one phase region (OPR); dilution lines (KO/Smix fraction containing PEG600) traversed across OPR, produced a fully dilutable nanoemulsion system. PEG600-based nanoformulations (NFs) of KO underwent phase transformation via percolation behaviour in nanostructure domains (86–207 nm). PEG600 induced ternary phase behaviour of KO as revealed from rheological data (higher eta values), refractive index (nonlinear) and conductivity (bimodal) patterns. Induced phase transformation could be an interaction between aqueous phase and KO/Tween20 in PEG600 environment; generating highly viscous domains of low electrical conductivity. NFs offered antioxidant activity over corresponding coarse systems (p < 0.01) as measured using DPPH method. Optimized NFs (F4 and F6) inhibited the growth of skin cancer cell line (A431) in the range of 100–500 × dilutions.ConclusionPhase behaviour of KO was induced by PEG600, transforming the dilution pattern via generation of one phase region; however, ethanol and propylene glycol as co-solvents did not. PEG600-based NFs of KO possessed antioxidant as well as cytotoxic to skin cancer cell lines (A431).