The flight altitude has a large effect on the airborne electromagnetic (AEM) responses. Due to the dynamic environment of the aircraft, the recorded sensor altitudes may contain errors. Research demonstrates that the AEM responses caused by a several meters altitude errors can be larger than caused by some anomalous body. Ignoring these errors will create erroneous results in AEM data interpretation. Considering that there is not yet a published 3D AEM inversion method that takes into account the flight altitude, we develop in this paper a 3D inversion algorithm for AEM with the flight height treated as an inversion parameter. For the forward modeling we use the finite element method, while for the inversion we use the Gauss-Newton optimization method. To make our inversion works for variable flight altitudes, we propose a scheme of 3D Jacobean matrix calculation for both the resistivities and flight altitudes without much increasing the computational cost. The numerical simulation result confirms that the flight altitude really has a large effect on the AEM responses. The inversions of synthetic data show that our 3D inversion method can both recover the resistivity distribution in the underground and decrease the altitude errors recorded, while the field data inversion demonstrates that our method can deliver a better inversion model with a smaller data misfit.