Deep learning approach for recognizing cold and warm thermal discomfort cues from videos

Abstract

Reducing building energy consumption while maintaining individual thermal comfort levels is a challenge. Energy-efficiency strategies that control indoor conditions, with the aim of reducing energy waste and increasing occupant comfort, require the capture of personal thermal comfort preferences in their feedback loop. However, direct feedback from surveys and invasive methods, which have been proposed to collect personal thermal comfort preferences, are impractical in real-world settings due to the decay of commitment, need for repeated feedback, and invasiveness. To address this challenge, this paper proposes a novel non-invasive approach for automated thermal discomfort detection using a deep learning-based model for thermal discomfort cue recognition. The model recognizes nine discomfort cues from video recordings of occupants in indoor work settings, which could be associated with the cold or warm thermal discomfort states. In developing the model, the architectures of two different residual neural networks (ResNets) that learn spatial and local information were tested using data augmentation and transfer learning techniques, and then compared with the performance of (1) traditional machine learning classifiers that used video frames mapped to the feature space of the ResNets; and (2) the ResNets extended with a long short-term memory (LSTM) layer that learns temporal information from the frame sequence. Approaches that accounted for temporal and spatial information outperformed those that only took spatial information into consideration. The final thermal discomfort cue recognition model achieved a weighted F1-measure of 85% in correctly classifying the cues. The proposed approach could help provide rapid and effortless feedback in building energy management systems to reduce building energy waste while providing comfortable indoor conditions towards increased occupant health and productivity.