We investigate entropy production in finitely slow transitions between nonequilibrium steady states in Markov jump processes using the improved adiabatic approximation method, which has a close relationship with the slow driving perturbation. This method provides systematic improvement of the adiabatic approximation on infinitely slow transitions from which we obtain nonadiabatic corrections. We analyze two types of excess entropy production and confirm that the leading adiabatic contribution reproduces known results, and then obtain nonadiabatic corrections written in terms of thermodynamic metrics defined in protocol parameter spaces. We also numerically study the resulting excess entropy production in a two-state system.