Direct numerical simulations of temporally developing, three-dimensional, CH4/CO2, CH4/O2, and CO2/O2 mixing layers are conducted at a supercritical pressure of 300 atm. To effectively model the supercritical regime, the employed formulation includes the compressible form of the governing equations, the cubic Peng–Robinson equation of state and a generalized formulation for heat and mass flux vectors derived from non-equilibrium thermodynamics and fluctuation theory. A linear inviscid stability analysis is also performed for each case, to determine its most unstable wavelength. Flow visualizations reveal the presence of high density gradient magnitude regions for all three mixing layers, with conditional averages indicating increased presence of heavier fluid species within these regions. No significant departures are observed from perfect gas behavior, with compressibility factors very close to unity for all three mixing cases. Applicability of presumed probability density function methods is examined for the three supercritical mixing layers. An a priori analysis is also conducted to investigate various simplifying assumptions employed in modeling various subgrid scale (SGS) flux models. Two additional terms are identified in the large eddy simulation equations, the gradient of SGS contribution of pressure in the momentum equation and the gradient of SGS contribution of heat flux in energy equation, whose magnitudes are similar and comparable with their respective resolved terms. The performance of the scale similarity model to represent these additional terms is investigated. Finally, the performance of Smagorinsky, gradient, and scale similarity models is also investigated.