Litter of different species coexists in the natural ecosystem and may induce non-additive effects during decomposition. Identifying and quantifying the origins of species in litter mixtures is essential for evaluating the responses of each component species when mixed with co-occurring species and then unraveling the underlying mechanism of the mixing effects of litter decomposition. Here, we used near-infrared spectroscopy (NIRS) to predict the species composition and proportions of four-tree species foliage mixtures in association with litter crude ash and litter decomposition time. To simulate the whole mixed litter decomposition process in situ, a controlled mixture of four tree species litter leaves consisting of 15 tree species combinations and 193 artificial mixed-species samples were created for model development and verification using undecomposed pure tree species and decomposed litter of single tree species over one year. Two series of NIRS models were developed with the original mass and ash-free weight as reference values. The results showed that these NIRS models could provide an accurate prediction for the percentage of the component species from in the litter leaf mixture’s composition. The predictive ability of the near-infrared spectroscopy model declined marginally with the prolonged litter decomposition time. Furthermore, the model with ash-free litter mass as a reference exhibited a higher coefficient of determination (R2) and a lower standard error of prediction (RMSECV). Thus, our results demonstrate that NIRS presents great potential for not only predicting the organic composition and proportion in multi-species mixed samples in static conditions, but also for samples in dynamic conditions (i.e., during the litter decomposition process), which could facilitate evaluation of the species-specific responses and impacts on the interspecific interactions of co-occurring species in high-biodiversity communities.