Understanding the properties of near-Earth asteroids (NEAs) is key for many aspects of planetary science, particularly planetary defense. Our current knowledge of NEA sizes and regolith properties is heavily dependent on simple thermal models. These models are often used to analyze data from missions such as NEOWISE because they are well suited to deal with large volumes of data. However, simple model results based on NEOWISE data may be inconsistent with results based on other types of observation in some cases. In this work, we seek to better understand these potential inconsistencies, as well as the situations for which they are most prevalent. We do this by comparing simple model results based on Infrared Telescope Facility SpeX data to similar results based on NEOWISE data. This is carried out for six NEAs that represent a range of spectral types, shapes, and rotation states. We find that models based on SpeX and NEOWISE data for these six objects are inconsistent in some cases, even though the SpeX results are consistent with other methods and observations. We find that objects observed at fainter magnitudes and objects with more primitive compositions are more likely to produce inconsistent fits. These results highlight the importance of better understanding the limitations of simple models as applied to large survey data sets like NEOWISE. This is particularly true as we move into an era where our understanding of the NEA population will be dominated by future large surveys such as NEO Surveyor.