In this work we analyze the convergence of the high-order Enhanced DtN-FEM algorithm, described in our previous work (Nicholls and Nigam, J. Comput. Phys. 194:278–303, 2004), for solving exterior acoustic scattering problems in $${\mathbf{R}^{2}}$$. This algorithm consists of using an exact Dirichlet-to-Neumann (DtN) map on a hypersurface enclosing the scatterer, where the hypersurface is a perturbation of a circle, and, in practice, the perturbation can be very large. Our theoretical work had shown the DtN map was analytic as a function of this perturbation. In the present work, we carefully analyze the error introduced by virtue of using this algorithm. Specifically, we give a full account of the error introduced by truncating the DtN map at a finite order in the perturbation expansion, and study the well-posedness of the associated formulation. During computation, the Fourier series of the Dirichlet data on the artificial boundary must be truncated. To deal with the ensuing loss of uniqueness of solutions, we propose a modified DtN map, and prove well-posedness of the resulting problem. We quantify the spectral error introduced due to this truncation of the data. The key tools in the analysis include a new theorem on the analyticity of the DtN map in a suitable Sobolev space, and another on the perturbation of non-self-adjoint Fredholm operators.