Inspired by the classical control theory for linear dynamical systems, we present an analytical continuation algorithm with inherent error stabilization. Numerical error is stabilized quite the same way as controlling the tracking error dynamics in system theory. Simulations demonstrate the efficacy of the proposed method wherein the stabilized continuation method always converges to the unique solution (if it exists) or any feasible solution (in the case of non-unique solutions). Further, we find new applications for this method in solving second order linear and non-linear elliptic partial differential equations with linear and non-linear boundary conditions. Illustrative numerical examples deal with stiff non-linear systems arising out of various physical applications.