The fouled ballast bed significantly impacts the operational condition of the ballasted track. Timely detection of the fouled ballast bed can greatly reduce maintenance workloads and capital expenditure. This study aims to assess the feasibility of employing infrared thermography (IRT) for detecting fouling. Nonetheless, existing research has been conducted under conditions noticeably different from the actual field. Moreover, the climatic conditions surrounding the ballasted track are beyond control. Hence, this paper innovatively utilized thermodynamic simulation to simulate as many conditions as possible. Firstly, temperatures and external meteorological data were obtained on a newly constructed railway line using an infrared thermal imager, temperature sensors, and a weather station. Thermodynamic inversion models were then established based on field structures to determine crucial thermodynamic parameters. The results demonstrate excellent alignment with the existing literature. Expanding upon this groundwork, the study explored the impact of various meteorological factors and weather conditions on the detection effectiveness of IRT. The results indicate that the solar radiation intensity (S) and air temperature (T) have the most significant effect on the surface temperature of the ballast bed (STB) and the temperature difference between the clean and fouled ballast beds (CF-TD). The rainfall (R) can change the thermodynamic properties of the ballast bed. Without considering R, the higher the S or T, the greater the CF-TD. Specifically, on sunny days, the maximum CF-TD can achieve 3.26 °C, while on cloudy days or at night, it is only 0.79 °C and 0.28 °C, respectively. Furthermore, the impact of wind speed (W) on the CF-TD was found to be disregarded in regions with low wind speeds but significant in regions with high wind speeds. In summary, IRT shows promise for rapid initial assessments of ballast bed conditions on sunny days, particularly following rain. While on cloudy days or at night, higher resolution infrared thermal imagers may be required, or other detection indicators may need to be investigated.