Damage induced localized failure is of vital importance to evaluate residual safety and to prevent catastrophic collapse of concrete structures. Aiming for the efficient and robust modeling of localized failure in quasi-brittle solids and structures, this paper addresses a thermodynamically consistent and numerically efficient localized damage model for concrete. The thermodynamic framework is presented in such a way that not only the continuum model for the bulk material, but also the localized model for the discontinuity, can be established in a consistent approach. For the latter, a localized Helmholtz free energy potential is postulated, mimicking the classical continuum damage model. Specifically, a localized damage variable is introduced to characterize degradation of the initially-rigid discontinuity with a well-defined reference stiffness. Consistent evolution law for the localized damage variable is derived from an appropriate traction-based failure criterion and its equivalent separation-based counterpart, e.g. the novel hyperbolic damage criterion introduced in this work. The proposed model can be regularized and in particular, upon the assumption of continuous stress field, an orthotropic damage model in the context of smeared crack methods is recovered. This coincidence not only justifies the proposed model, but also sheds new lights on other classical methods. Numerically, the proposed localized damage model is incorporated into the improved stable eXtended FEM. Compared to those plasticity-based localized models, an explicit numerical algorithm can be used to update the cohesive tractions transferred across the discontinuity with no iteration on the constitutive level. This feature, together with the well-conditioned system matrix ensured by the improved stable eXtended FEM, guarantees noticeable efficiency and robustness of the overall numerical performances. The proposed model is verified against several classical benchmark tests of concrete in both mode-I and mixed-mode failure. The numerical predictions agree well with experimental test data and those reported in the literature. Remarkably, all the numerical results are mesh-size and mesh-alignment independent, and no spurious stress locking is observed, showing validity of the proposed model in the modeling of localized failure in concrete.