PurposeThe purpose of this paper is to present a new approach to optimizing the design of 3D magnetic circuits. This approach is based on topology optimization, where derivative calculations are performed using the continuous adjoint method. Thus, the continuous adjoint method for magnetostatics has to be developed in 3D and has to be combined with penalization, filtering and homotopy approaches to provide an efficient optimization code.Design/methodology/approachTo provide this new topology optimization code, this study starts from 2D magnetostatic results to perform the sensitivity analysis, and this approach is extended to 3D. From this sensitivity analysis, the continuous adjoint method is derived to compute the gradient of an objective function of a 3D topological optimization design problem. From this result, this design problem is discretized and can then be solved by finite element software. Thus, by adding the solid isotropic material with penalization (SIMP) penalization approach and developing a homotopy-based optimization algorithm, an interesting means for designing 3D magnetic circuits is provided.FindingsIn this paper, the 3D continuous adjoint method for magnetostatic problems involving an objective least-squares function is presented. Based on 2D results, new theoretical results for developing sensitivity analysis in 3D taking into account different parameters including the ferromagnetic material, the current density and the magnetization are provided. Then, by discretizing, filtering and penalizing using SIMP approaches, a topology optimization code has been derived to address only the ferromagnetic material parameters. Based on this efficient gradient computation method, a homotopy-based optimization algorithm for solving large-scale 3D design problems is developed.Originality/valueIn this paper, an approach based on topology optimization to solve 3D magnetostatic design problems when an objective least-squares function is involved is proposed. This approach is based on the continuous adjoint method derived for 3D magnetostatic design problems. The effectiveness of this topology optimization code is demonstrated by solving the design of a 3D magnetic circuit with up to 100,000 design variables.