PurposeThe purpose of this study is to develop a reliable finite element algorithm based on the transmission line method (TLM) to solve the nonlinear problem in electromagnetic field calculation.Design/methodology/approachIn this paper, the TLM has been researched and applied to solve nonlinear iteration in FEM. LU decomposition method and the Jacobi preconditioned conjugate gradient method have been investigated to solve the equations in transmission line finite element method (FEM-TLM). The algorithms have been developed in C++ language. The algorithm is applied to analyze the magnetic field of a long straight current-carrying wire and a single-phase transformer.FindingsFEM-TLM is more effective than traditional FEM in nonlinear iteration. The results of FEM-TLM have been compared and analyzed under different calculation scales. It is found that the LU decomposition method is more suitable for FEM-TLM because there is no need to repeatedly assemble the global coefficient matrix in the iterative solution process and it is not affected by the disturbance of the right-hand vector.Originality/valueAn effective algorithm is provided for solving nonlinear problems in the electromagnetic field, which can save a lot of computing costs. The efficiency of LU decomposition and CG method in FEM-TLM nonlinear iteration is investigated, which also makes a preliminary exploration for the research of FEM-TLM parallel algorithms.