A new soil moisture and soil temperature wireless sensor network (the SMN-SDR) consisting of 34 sites was established within the Shandian River Basin in 2018, located in a semi-arid area of northern China. In this study, in situ measurements of the SMN-SDR were used to evaluate 24 different soil moisture datasets grouped according to three categories: (1) single-sensor satellite-based products, (2) multi-sensor merged products, and (3) model-based products. Triple collocation analysis (TCA) was applied to all possible triplets to verify the reliability and robustness of the results. Impacts of different factors on the accuracy of soil moisture products were also investigated, including local acquisition time, physical surface temperature, and vegetation optical depth (VOD). The results reveal that the latest Climate Change Initiative (CCI) -combined product (v06.1, merging extra low-frequency passive microwave data) had the best agreement with in situ measurements from the SMN-SDR, with the lowest ubRMSE (< 0.04 m3/m3) and highest R (> 0.6). Among all single-sensor retrieved soil moisture products, the Soil Moisture Active Passive (SMAP) products performed best in terms of R (> 0.6) and ubRMSE (close to 0.04 m3/m3), with the SMAP-MDCA (Modified Dual Channel Algorithm) being slightly better than the baseline SCA-V (Single Channel Algorithm-Vertical polarization). Importantly, the newly developed SMAP-IB product, which does not use auxiliary data, delivered the best bias statistics and higher VOD values compared with the drier SMAP retrievals, suggesting that the low VOD values (underestimated vegetation effects) may be the major factor causing the dry bias of SMAP products in this study area. It was also found that TCA may systematically overestimate the correlation and underestimate the ubRMSE of soil moisture products as compared with ground-based metrics. TCA-based metrics may vary considerably when using different triplets, due to the TCA assumptions being violated even with the most conservative triplets (in this case an active product, a passive product, and a model-based product). Redundant TCA-based metrics from multiple independent triplets could be averaged to increase the accuracy of final TCA estimates. This study is the first to use in situ measurements from the SMN-SDR to conduct a comprehensive evaluation of commonly used, multi-source soil moisture products. These results are expected to further promote the improvement of satellite- and model-based soil moisture products.