A carrier-phase direct numerical simulation (CP-DNS) of a coal/ammonia co-firing flame in a turbulent mixing layer is performed with a detailed chemical reaction mechanism including NOx formation. The combustion characteristics and NOx formation mechanism of the coal/ammonia co-firing flame are investigated in detail through a conditional reaction pathway analysis. A new four-fuel-stream flamelet model is proposed to adapt to the piloted pulverized coal/ammonia co-combustion system, in which the complex mixing among the volatiles, char off-gases, ammonia, and the pilot stream is characterized with four mixture fractions. The flamelet solutions in the fourfold mixture fraction space are transformed into a uniform space to facilitate the access to the flamelet table. The performance of the flamelet model is evaluated through an a priori analysis, a combustion mode analysis, and a chemical timescale analysis. For the turbulent coal/ammonia co-firing flame studied, two distinct flame fronts within the shear layer are observed with the upper part being dominated by ammonia combustion and the lower part being governed by coal combustion. The conditional analyses show that NO species is mainly generated by the lower layer governed by pulverized coal combustion where the nitrogen-containing species from volatile combustion are highly involved in the NO formation. The NO concentration first increases with decreasing the fraction of ammonia in the local computational cell, and then decreases towards pure pulverized coal combustion, which is characterized with a newly introduced manifold coordinate. While the temperature and the major and intermediate species mass fractions in the coal/ammonia co-firing flame can be reasonably predicted by the proposed flamelet model, the NOx species mass fractions are not well predicted in the region where pulverized coal combustion dominates. It is clarified that the inaccurate prediction of the NOx species is not caused by interpolation errors, multiple combustion modes or the kinetics of NOx formation, but mainly attributed to the definition of the progress variable.Novelty and significance statementThe novelty of this research is the first DNS of a coal/ammonia co-firing flame in a turbulent mixing layer, and the proposal of a new flamelet tabulation method for predicting NOx formation in the coal/ammonia co-firing flame. It is significant because• Coal/ammonia co-firing promises to provide continuous, secure power supply at a much reduced carbon footprint compared to pure coal combustion;• The underlying physics governing the coal/ammonia co-firing are analyzed using carrier-phase DNS;• The performance of the newly proposed flamelet model in predicting the NOx species is evaluated based on the DNS dataset.