Magnet tracking is an emerging tracking method for medical instruments and biological tissues, such as endoscopes and catheters. Traditional magnet tracking technology typically uses an optimization algorithm or heuristic algorithm to estimate the pose of a permanent magnet according to the measured magnetic induction intensity. However, this technology is limited by the high delay and the poor quality of the initial solution. This article proposes a new magnetic localization method, i.e., a hybrid feedforward neural network and Levenberg–Marquardt (hFFNN-LM) algorithm. FFNN provides an adjusted initial solution to the LM algorithm for solving the nonlinear inverse magnetic problem. The hFFNN-LM algorithm is promising for real-time magnetic tracking as it provides satisfactory tolerance for initially estimated parameters and fast search speed. The average single localization time is 1.07 ms. The experimental results show that the proposed method has a very low tracking delay compared to some other popular LM-based methods. The proposed hFFNN-LM algorithm achieves superior localization performances, i.e., the mean position and orientation errors are tested to be 0.70 mm and 0.90°, respectively.