Nonlinear magnetic network is an effective model to solve the magnetic field of the motor. For the strong nonlinearity of magnetic network parameters, it is difficult to solve the nonlinear magnetic network equations. To overcome the divergent issue for solving the nonlinear magnetic network equations, a new method, the homotopy continuation method, is proposed. Based on the benchmark model of Testing Electromagnetic Analysis Methods Problem 20 (TEAM P20), the two-dimensional nonlinear magnetic network model is established. Next, homotopy equations of nonlinear magnetic network equations are derived, and they are solved by the homotopy continuation method. So that solving the nonlinear magnetic network equations is transformed into solving a set of homotopy equations. Then, the magnetic flux density of iron core with different saturation levels is calculated using the simple iteration method, the relaxation iteration method, Newton-Raphson method, and the homotopy continuation method, respectively. The results and convergence performance of the four methods are compared. It is proved that the proposed method can effectively expand the convergence domain and simplify the calculation process. The magnetic flux density calculated by the homotopy continuation method is compared with the finite element method in the supersaturated state, and the results are in good agreement. The relative error of the magnetic flux density of the core is less than 5%, which verifies the correctness of the proposed algorithm.