This work presents a simple synthesis of tetraaza macrocyclic Schiff base ligand (C40H28N4), its complex [(C40H28N4)@Fe(II)], and a novel complex of magnetite Fe3O4NPs incorporated inside tetraaza macrocyclic cavity [(C40H28N4)@Fe3O4NPs] by the co-precipitation method in order to obtain well-dispersed nanoparticles, controlled size and prevent the aggregation of Fe3O4 NPs due to strong dipole–dipole magnetic attractions and interactions between particles. The characterization and structural identification were carried out by X-ray spectroscopy as well by 1H NMR, 13C and DEPT 135 NMR spectroscopy, FT-IR, UV–visible, EDX, TGA, and finally by VSM using both experimental and theoretical methods. XRD measurements indicate that the presence of Schiff's bases does not modify the crystal structure of the nanoparticles (11.05 nm). All the NMR spectra suggested the success of the synthesis of all structures with high purity. FTIR was used to illuminate the presence of Fe3O4NPs in tetrahedral and octahedral sites as well as their coordination with imine (C = N) of tetraaza macrocyclic. The UV–visible spectra and frontier molecular orbitals (FMOs) of the title compounds were calculated at TD-DFT/CAM-B3LYP-D3/6–311 G (d, p) level of theory. The corresponding calculated result yield shows a good agreement with the experimental data. The morphological characterization of the nanoparticles was carried out by SEM, which revealed that the shape of the NPs was generally spherical. The SEM images also show that the nanoparticles prepared by in situ with co-precipitation method were able to form stable complexes. Thermal characterization by TGA shows that there are 64% of the magnetite nanoparticles formed in situ, which corresponds to a grafting density of 25 mmol.g−1. The VSM analysis displays a similar magnetic hysteresis loop shape compared to free Fe3O4 NPs, which confirmed a ferromagnetic character due to the significant decrease in saturated magnetization.Graphical abstract