Two-color (2C) pyrometry has long been used for flame temperature and soot concentration studies and is now becoming more widely used to measure backface temperatures of burning materials. With the obvious advantage of being a contact-free method that requires only minimal optical access, 2C pyrometry combined with high-speed acquisition is a promising diagnostic tool to obtain exceptional temporal and spatial resolution of thermally degrading samples. However, its conceptual simplicity relies on a set of basic assumptions that when violated can result in large errors. In this work, we used an experimental configuration representative for fire resistance testing for aerospace and naval applications to analyze the impact of camera parameters and test setup on the accuracy of the surface temperature results obtained. Two types of fiber reinforced polymer composites and a steel plate are used to investigate material specific aspects that affect the measurements. An improved workflow for camera calibration is presented that takes into account the actual experimental setup. The temperature and emissivity mapping obtained through in-situ IR measurements is compared to data acquired through thermocouples and post-fire hemispherical directional reflectance measurements at room temperature. Excellent agreement is found for post-processed 2C pyrometry measurements and TC probe measurements with an uncertainty of 30 °C at 500 °C. In-situ emissivity mappings of burnt specimens are consistent with data from the literature for degraded composite samples and oxidized steel plates. Our results remind the importance of proper post-processing and demonstrate the potentials of 2C pyrometry for fire testing applications. We also present a detailed error budget and suggestions for calibration measurements to improve measurement accuracy.