Theoretical scaling arguments for turbulent premixed combustion indicate that the pressure-dilatation source of turbulent kinetic energy becomes significant at low Karlovitz numbers, leading to potential invalidation of commonly-used turbulence models developed for non-reacting flow. Based on these arguments, a critical Karlovitz number is defined, below which dilatation effects are expected to become significant. Velocity and scalar statistics are obtained from Direct Numerical Simulation (DNS) calculations of low Mach number spatially-evolving turbulent premixed planar jet flames. At fixed bulk Reynolds number and stoichiometric equivalence ratio, two simulations are performed at Karlovitz numbers above and below the critical Karlovitz number. Hydrogen combustion with detailed transport is modeled using a detailed nine-species chemical kinetic mechanism, and coflows of combustion products are used to ensure flame stability at uniform equivalence ratio. The analysis of these statistics focuses on three key areas. First, the influence of the velocity-pressure gradient source of turbulent kinetic energy is confirmed at a low Karlovitz number, and the unimportance of these effects is confirmed at a high Karlovitz number. Similar effects are observed for the chemical source term in the scalar variance budgets. Second, the degree of alignment between the Reynolds stress tensor (scalar flux) and the strain-rate tensor (scalar gradient), the foundation of a majority of the turbulence models used in reacting flows, is assessed with the DNS databases. Additionally, consistency of anisotropic Reynolds stress and strain-rate tensor invariants is assessed using invariant maps. While good alignment and consistency are obtained for statistics and invariants at a high Karlovitz number, both alignment and consistency degrade at a low Karlovitz number. Third, turbulence models formulated for non-reacting flow are modified algebraically in the Bray–Moss–Libby (BML) formalism for turbulent premixed combustion. A variable efficiency function is defined to capture the regime dependence of heat release effects in these models. Model performance is evaluated at Karlovitz numbers above and below the critical Karlovitz number using the DNS databases, and satisfactory prediction of counter-gradient transport in the flame-normal direction is obtained. However, heat release effects are also observed in the flame-parallel directions in the low-Karlovitz number simulation, and the models developed in the formalism for statistically planar flames fail to capture these effects. Furthermore, in the low-Karlovitz number case, redistributive effects are active on the shear components of the Reynolds stress, which are not considered in the BML formalism. More advanced turbulence models are therefore necessary for turbulent premixed jet flames below the critical Karlovitz number.