The present study demonstrates a rhombohedral-to-orthorhombic phase transition of BiFe1-XCoXO3 (BFCXO, x = 0, 0.01, 0.03, and 0.05) nanoparticles induced by Co ion doping. With increasing concentration of Co dopant, XRD, Raman and IR patterns exhibit lattice shrinkage and increased lattice strain; SEM and TEM images reveal a corresponding reduction in grain size; XPS spectra indicate that the Fe2+ ions are readily oxidized by Co3+ to Fe3+ ions because to the greater oxidation–reduction potential of Fe3+/Fe2+ (1.3 eV) than that of Co3+/Co2+ (0.55 eV). Effective regulation of crystal structure and microstructure significantly alters the multiferroic properties of BFCXO samples. The Co-doped samples gradually transition from antiferromagnetism to ferromagnetism. Among them, BFC0.03O reaches its maximum values at Ms ~ 0.956 emu/g for magnetization saturation and Mr ~ 0.054 emu/g for remanent magnetization. Meanwhile, BFC0.03O exhibits better ferroelectric properties compared to its pure counterpart. This study presents an effective approach for controlling the structure and characteristics of BFO-based nanomaterials for multifunctional device applications.