Thermal Scene Research Articles

U-Net for temperature estimation from simulated infrared images in tokamaks

Surface temperature measurement is critical for the safety and proper operation of nuclear fusion reactors such as tokamaks, which operate at high temperature [100–3600 °C] in the presence of complex and unknown surface reflectance properties, causing a reflected flux that disturbs measurement interpretations. This paper describes a numerical approach based on machine learning techniques to estimate the surface temperature of in-vessel components from infrared images, despite the presence of reflections and the unknown emissivity of materials. Our contributions are two-fold: for the first time in the infrared domain, we generate a huge dataset of simulated images thanks to a novel GPU implementation of a Monte Carlo infrared ray tracer. This implementation offers the great advantage of enabling the generation of a very large amount of synthetic images for different plasma scenarios. Hence the second contribution of this paper: we propose an infrared inverse lighting model, based on a Convolutional Neural Network (CNN) trained with the images generated by our ray tracer. The proposed approach is applied on a numerical prototype of W Environment Steady state Tokamak (WEST), including complex simulated thermal scenes. Using synthetic test data, and without providing the surface optical properties, the mean temperature error predicted by the CNN on ITER-like wide angle view of WEST is evaluated to 9% with standard plasma scenarios and 22% in case of exotic thermal scenes.

Moving objects detection in thermal scene videos using unsupervised Bayesian classifier with bootstrap Gaussian expectation maximization algorithm

Moving objects detection in thermal scene videos using unsupervised Bayesian classifier with bootstrap Gaussian expectation maximization algorithm

The wide-angle infrared diagnostic for the first wall monitoring of the WEST tokamak

A tangential infrared thermography system is installed in the WEST tokamak to observe the thermal scene of the first wall and the divertor components through a wide-angle tangential view. The goal is to provide and monitor the heat load deposition on the plasma facing components. About one-sixth of the chamber is observed, including sectors of the baffle, the lower and upper divertors, one inner bumper, vertical displacement event (VDE) protections, one ICRH antenna, and other parts of the wall. It is used for real time machine protection by monitoring temperature thresholds in delimited region of interest, and for analysis of normal or specific heat load events during operation such as VDE, ELM, disruption or runaways. This wide-angle view uses one aspherical and one on-axis plane mirrors, a sapphire window and three lenses for the objective of the camera. The optical line is optimized for two wavelengths 1.7 and 4 µm. The field of view is 60° on a 512×640 pixels Focal Plane Array. The endoscope is fully actively cooled. The thermal scene is complex to interpret given the fully metallic and radiative environment and the uncertainties on the emissivity which is angular-dependent and changes with the surface properties. This involves significant inaccuracy on the recovery of the real temperature. In this context, an interpretation by modeling approach is better suited, based on ray-tracing simulations taking into account the optical properties of materials. For instance, this allowed discriminating reflections patterns from real thermal events in the wide-angle view. A description of the wide-angle infrared diagnostic and its performances is presented as well as experimental measurements obtained in the WEST Tokamak.

Edge-Aware Guidance Fusion Network for RGB–Thermal Scene Parsing

RGB–thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high-level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB–thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed information in the prediction map and then embed the prior edge information in the feature maps. To effectively fuse the RGB and thermal information, we propose a multimodal fusion module that guarantees adequate cross-modal fusion. Considering the importance of high-level semantic information, we propose a global information module and a semantic information module to extract rich semantic information from the high-level features. For decoding, we use simple elementwise addition for cascaded feature fusion. Finally, to improve the parsing accuracy, we apply multitask deep supervision to the semantic and boundary maps. Extensive experiments were performed on benchmark datasets to demonstrate the effectiveness of the proposed EGFNet and its superior performance compared with state-of-the-art methods. The code and results can be found at https://github.com/ShaohuaDong2021/EGFNet.

MFFENet: Multiscale Feature Fusion and Enhancement Network For RGB–Thermal Urban Road Scene Parsing

Compared with traditional handcrafted features, deep learning has greatly improved the performance of scene parsing. However, it remains challenging under various environmental conditions caused by imaging limitations. Thermal imaging cameras have several advantages over cameras for the visible spectrum, such as operation in total darkness, robustness to shadow effects, insensitivity to illumination variations, and strong ability to penetrate smog and haze. These advantages of thermal imaging cameras make them ideal for the scene parsing of semantic objects in daytime and nighttime. In this paper, we propose a novel multiscale feature fusion and enhancement network (MFFENet) for accurate parsing of RGB–thermal urban road scenes even when the quality of the available RGB data is compromised. The proposed MFFENet consists of two encoders, a feature fusion layer, and a multi-label supervision layer. We concatenate the multi-scale features with the features that contain global semantic information. Furthermore, we explore the cross-modal fusion of RGB and thermal features at multiple stages, rather than fusing them once at the low or high stage. Then, we propose a spatial attention mechanism module that provides a higher weight to (focuses more on) the foreground area, allowing MFFENet to emphasize foreground objects. Finally, multi-label supervision is introduced to optimize parameters of the proposed MFFENet. Experimental results confirm that the proposed MFFENet outperforms similar high-performing methods.

Behavioral Parameter Field for Human Abnormal Behavior Recognition in Low-Resolution Thermal Imaging Video

In recent years, thermal imaging cameras are widely used in the field of intelligent surveillance because of their special imaging characteristics and better privacy protection properties. However, due to the low resolution and fixed location for current thermal imaging cameras, it is difficult to effectively identify human behavior using a single detection method based on skeletal keypoints. Therefore, a self-update learning method is proposed for fixed thermal imaging camera scenes, called the behavioral parameter field (BPF). This method can express the regularity of human behavior patterns concisely and directly. Firstly, the detection accuracy of small targets under low-resolution video is improved by optimizing the YOLOv4 network to obtain a human detection model under thermal imaging video. Secondly, the BPF model is designed to learn the human normal behavior features at each position. Finally, based on the learned BPF model, we propose to use metric modules, such as cosine similarity and intersection over union matching, to accomplish the classification of human abnormal behaviors. In the experimental stage, the living scene of the indoor elderly living alone is applied as our experimental case, and a variety of detection models are compared to the proposed method for verifying the effectiveness and practicability of the proposed behavioral parameter field in the self-collected thermal imaging dataset for the indoor elderly living alone.

Identification of radiant source in an enclosure by reduced model

We propose an original method to recover from a few measurement points the integrity of the temperature field of a furnace heated by a radiant thermal source. The radiant thermal source is first identified via a low order reduced model based on based on AROMM (Amalgam Reduced Order Modal Model) method which preserves the integrity of the geometry. The minimization is performed via a trust-region reflective least squares algorithm implemented in MATLAB “lsqcurvefit” function. From that identified heat flux, the integrity of the thermal field is then recovered by direct simulation thanks to a reduced model of higher rank to have a better precision. The treated application is a complex titanium piece heated by two radiant panels placed in a furnace. With four measuring points, the temperature of the whole thermal scene is retrieved at all times with an average error around 1 K on the studied object.

Low complexity visual UAV track navigation using long‐wavelength infrared

AbstractVisual navigation is a commonly researched alternative to the use of global navigation satellite systems in challenging environments where satellite signals are not available. However, the vast majority of visual navigation techniques studied to date require scene illumination of some form. In this study, we use a low‐resolution long‐wave infrared (LWIR) image sensor sensitive to thermal emissivity within an optical flow processing engine to extend a low complexity track‐based navigation scheme for fixed wing aircraft to operate at night. A mixture of closed and open loop flight experiments conducted on a small UAV integrated with the new sensor demonstrate: accurate track‐based navigation in visual darkness; that the LWIR sensor performs equivalently to the benchmark optical flow sensor during daylight and continues to operate in low light; and that the LWIR sensor is able to detect suitable textures for operation at night across a wide span of altitudes. These results demonstrate utility of optical flow algorithms with low‐resolution thermal scenes as a novel aircraft navigation sensor for day and night operation.

Mid-infrared electroluminescence from type-II In(Ga)Sb quantum dots

There exists significant interest in the demonstration and development of alternative mid-infrared emitters, with future applications for thermal scene projection, low-cost infrared sensing, and possible long-wavelength quantum communication applications. Type-II In(Ga)Sb quantum dots grown in InAs matrices have the potential to serve as a viable material system for wavelength-flexible, mid-infrared sources. Here, we dramatically expand the range of potential applications of these mid-infrared quantum emitters through the demonstration of surface-emitting electrically pumped mid-infrared light-emitting diodes with active regions utilizing type-II In(Ga)Sb quantum dots. Two device structures were studied, the first iteration being a single In(Ga)Sb insertion layer within a simple PIN structure and the second being a design engineered for improved room temperature emission with the addition of lattice matched AlAsSb cladding at the anode to block electrons and five layers of In(Ga)Sb dots to increase the effective volume of active material. Samples were grown by molecular beam epitaxy and the electrical and optical properties for each design were characterized as a function of temperature.

Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging.

The ability to monitor the respiratory rate, one of the vital signs, is extremely important for the medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake everyday activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Alternatively, contactless digital image sensor based remote-photoplethysmography (PPG) can be used. However, remote PPG requires an ambient source of light, and does not work properly in dark places or under varying lighting conditions. Recent advances in thermographic systems have shrunk their size, weight and cost, to the point where it is possible to create smart-phone based respiration rate monitoring devices that are not affected by lighting conditions. However, mobile thermal imaging is challenged in scenes with high thermal dynamic ranges (e.g. due to the different environmental temperature distributions indoors and outdoors). This challenge is further amplified by general problems such as motion artifacts and low spatial resolution, leading to unreliable breathing signals. In this paper, we propose a novel and robust approach for respiration tracking which compensates for the negative effects of variations in the ambient temperature and motion artifacts and can accurately extract breathing rates in highly dynamic thermal scenes. The approach is based on tracking the nostril of the user and using local temperature variations to infer inhalation and exhalation cycles. It has three main contributions. The first is a novel Optimal Quantization technique which adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking. The second is the Thermal Gradient Flow method that computes thermal gradient magnitude maps to enhance the accuracy of the nostril region tracking. Finally, we introduce the Thermal Voxel method to increase the reliability of the captured respiration signals compared to the traditional averaging method. We demonstrate the extreme robustness of our system to track the nostril-region and measure the respiratory rate by evaluating it during controlled respiration exercises in high thermal dynamic scenes (e.g. strong correlation (r = 0.9987) with the ground truth from the respiration-belt sensor). We also demonstrate how our algorithm outperformed standard algorithms in settings with different amounts of environmental thermal changes and human motion. We open the tracked ROI sequences of the datasets collected for these studies (i.e. under both controlled and unconstrained real-world settings) to the community to foster work in this area.

3D thermography for improving temperature measurements in thermal vacuum testing

The application of thermography to thermal vacuum (TV) testing of spacecrafts is becoming a vital additional tool in the mapping of structures during thermal cycles and thermal balance (TB) testing. Many of the customers at the European Space Agency (ESA) test centre, European Space Research and Technology Centre (ESTEC), The Netherlands, now make use of a thermal camera during TB-TV campaigns. This complements the use of embedded thermocouples on the structure, providing the prospect of monitoring temperatures at high resolution and high frequency. For simple flat structures with a well-defined emissivity, it is possible to determine the surface temperatures with reasonable confidence. However, for most real spacecraft and sub-systems, the complexity of the structure’s shape and its test environment creates inter-reflections from external structures. This and the additional complication of angular and spectral variations of the spacecraft surface emissivity make the interpretation of the radiation detected by a thermal camera more difficult in terms of determining a validated temperature with high confidence and well-defined uncertainty. One solution to this problem is: to map the geometry of the test specimen and thermal test environment; to model the surface temperatures and emissivity variations of the structures and materials; and to use this model to correct the apparent temperatures recorded by the thermal camera. This approach has been used by a team from NPL (National Physical Laboratory), Psi-tran, and PhotoCore, working with ESA, to develop a 3D thermography system to provide a means to validate thermal camera temperatures, based on a combination of thermal imaging photogrammetry and ray-tracing scene modeling. The system has been tested at ESTEC in ambient conditions with a dummy spacecraft structure containing a representative set of surface temperatures, shapes, and spacecraft materials, and with hot external sources and a high power lamp as a sun simulator. The results are presented here with estimated temperature measurement uncertainties and defined confidence levels according to the internationally accepted Guide to Uncertainty of Measurement as used in the IEC/ISO17025 test and measurement standard. This work is understood to represent the first application of well-understood thermal imaging theory, commercial photogrammetry software, and open-source ray-tracing software (adapted to realize the Planck function for thermal wavebands and target emission), and to produce from these elements a complete system for determining true surface temperatures for complex spacecraft-testing applications.

Thermal analysis of droplet flow: Numerical, analytical and experimental investigations

This article reports the thermal behavior analysis of droplet flow for a scenario in which the hydrodynamic and thermal effects are coupled in a cylindrical miniaturized channel; this analysis was performed by comparing numerical, analytical and experimental results. The 3D numerical modeling of the droplet flow was performed using THETIS, a multi-fluid Navier–Stokes solver. A 1D analytical model in Lagrangian space based on a thin-fin thermal approximation was developed. Infrared thermography measurements of droplet flows were used to experimentally determine the temperature fields inside droplets, and these results were compared with those obtained from the simulations and the analytical approach. The originality of this work is the choice of a particular thermal scene to mimic a chemical situation in order to improve understanding and to illustrate the predominant exchanges by model simplification. Furthermore, the results of this study are important for understanding the heat transfer occurring inside droplets for lab-on-chip applications.

Infrared Devices And Techniques (Revision)

Abstract The main objective of this paper is to produce an applications-oriented review covering infrared techniques and devices. At the beginning infrared systems fundamentals are presented with emphasis on thermal emission, scene radiation and contrast, cooling techniques, and optics. Special attention is focused on night vision and thermal imaging concepts. Next section concentrates shortly on selected infrared systems and is arranged in order to increase complexity; from image intensifier systems, thermal imaging systems, to space-based systems. In this section are also described active and passive smart weapon seekers. Finally, other important infrared techniques and devices are shortly described, among them being: non-contact thermometers, radiometers, LIDAR, and infrared gas sensors.

Human search with a limited field of view: effects of scan speed, aperture size, and target conspicuity

We investigate how human visual search performance with field of view (FOV) restrictions depends on scan speed, zoom factor, target location, and conspicuity. First, observers search a FOV that moves at a constant speed along a predefined path over a visual search scene (the field of regard, or FOR), and we measure the effects of scan speed, zoom factor, FOV size, target location, and conspicuity on search performance (detection time and probability). Then observers search either visual or thermal scenes by freely moving a joystick-controlled FOV over the FOR, while freely selecting the zoom factor in some conditions, and we measure the effects of the scan path characteristics and the use of different strategies on search performance. Search performance depends on the effective FOV size (the area where targets are expected to occur), but is largely independent of the display area. Both for visual and thermal imagery, various search performance measures correlate with target conspicuity. The results of both experiments suggest that observers can optimize their performance by adjusting their scanning behavior both to their visual limitations and to the expected target conspicuity. We present the outlines of a simple search model based on these current findings.

Modeling diurnal variation of ground thermal radiance images using energy balance model and endmember composing technique

Modeling and analyzing dynamic changes of land thermal radiance scenes play an important role in thermal remote sensing. In this paper, the diurnal variation of ground surface thermal scene is mainly discussed. Firstly, based on the land surface energy balance equation, the diurnal variation of land surface temperatures (LSTs) over bare land covers were simulated by an analytical thermal model with second harmonic terms, and the diurnal LST variation of vegetation canopy was simulated using the Cupid model. Secondly, normalized difference vegetation index (NDVI), normalized difference water index (NDWI), and ratio resident-area index (RRI) were used to evaluate the endmember abundance of four land cover types including vegetation, bare soil, impervious and water area, which were calculated from IKONOS visible and near infrared (VNIR) bands. Finally, the thermal radiance scenes at various times and view angles were modeled based on the linear-energy-mixing hypothesis. The results showed that the simulated daily LST variations for vegetated and bare surfaces are correlated with the measured values with a maximum standard deviation of 2.7°C, that land thermal radiant textures with high-resolution are restored from the linear-energy-mixing method, and that the information abundance of the scene are related to the distribution of land cover, the imaging time, and the view angle.

Erratum: Towards high-speed imaging of infrared photons with bio-inspired nanoarchitectures

Nature Photon. 6, 195–200 (2012); published online 12 February 2012; corrected after print 11 July 2012. In the version of this Article originally published, the measured temperature difference ΔT was at the detector plane, whereas the definition of noise-equivalent temperature difference (NETD) in equation (2) requires ΔT to be a thermal scene temperature difference.

Development and validation of an automatic thermal imaging process for assessing plant water status

Leaf temperature is a physiological trait that can be used for monitoring plant water status. Nowadays, by means of thermography, canopy temperature can be remotely determined. In this sense, it is crucial to automatically process the images. In the present work, a methodology for the automatic analysis of frontal images taken on individual trees was developed. The procedure can be used when cameras take at the same time thermal and visible scenes, so it is not necessary to reference the images. In this way, during the processing in batch, no operator participated. The procedure was developed by means of a non supervised classification of the visible image from which the presence of sky and soil could be detected. In case of existence, a mask was performed for the extraction of intermediate pixels to calculate canopy temperature by means of the thermal image. At the same time, sunlit and shady leaves could be detected and isolated. Thus, the procedure allowed to separately determine canopy temperature either of the more exposed part of the canopy or of the shaded portion. The methodology developed was validated using images taken in several regulated deficit irrigation trials in Persimmon and two citrus cultivars (Clementina de Nules and Navel Lane-Late). Overall, results indicated that similar canopy temperatures were calculated either by means of the automatic process or the manual procedure. The procedure developed allows to drastically reduce the time needed for image analysis also considering that no operator participation was required. This tool will facilitate further investigations in course for assessing the feasibility of thermography for detecting plant water status in woody perennial crops with discontinuous canopies. Preliminary results reported indicate that the type of crop evaluated has an important influence in the results obtained from thermographic imagery. Thus, in Persimmon trees there were good correlations between canopy temperature and plant water status while, in Clementina de Nules and Navel Lane-Late citrus cultivars canopy temperature differences among trees could not be related with tree-to-tree variations in plant water status.

Towards Intelligent Video Understanding Applied to Plasma Facing Component Monitoring

AbstractIn this paper, we promote intelligent plasma facing component video monitoring for both real‐time purposes (machine protection issues) and post event analysis purposes (plasma‐wall interaction understanding). We propose a vision‐based system able to automatically detect and classify into different pre‐defined categories thermal phenomena such as localized hot spots or transient thermal events (e.g. electrical arcing) from infrared imaging data of PFCs. This original computer vision system is made intelligent by endowing it with high‐level reasoning (i.e. integration of a priori knowledge of thermal event spatiotemporal properties to guide the recognition), self‐adaptability to varying conditions (e.g. different thermal scenes and plasma scenarios), and learning capabilities (e.g. statistical modelling of event behaviour based on training samples) (© 2011 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Detecting land cover change at the Jornada Experimental Range, New Mexico with ASTER emissivities

Multispectral thermal infrared remote sensing of surface emissivities can detect and monitor long term land vegetation cover changes over arid regions. The technique is based on the link between spectral emissivities within the 8.5–9.5 μm interval and density of sparsely covered terrains. The link exists regardless of plant color, which means that it is often possible to distinguish bare soils from senescent and non-green vegetation. This capability is typically not feasible with vegetation indices. The method is demonstrated and verified using ASTER remote sensing observations between 2001 and 2003 over the Jornada Experimental Range, a semi-arid site in southern New Mexico, USA. A compilation of 27 nearly cloud-free, multispectral thermal infrared scenes revealed spatially coherent patterns of spectral emissivities decreasing at rates on the order of 3% per year with R 2 values of ∼ 0.82. These patterns are interpreted as regions of decreased vegetation densities, a view supported by ground-based leaf area index transect data. The multi-year trend revealed by ASTER's 90-m resolution data are independently confirmed by 1-km data from Terra MODIS. Comparable NDVI images do not detect the long-term spatially coherent changes in vegetation. These results show that multispectral thermal infrared data, used in conjunction with visible and near infrared data, could be particularly valuable for monitoring land cover changes.

Modelling surface energy fluxes over maize using a two-source patch model and radiometric soil and canopy temperature observations

Models estimating surface energy fluxes over partial canopy cover with thermal remote sensing must account for significant differences between the radiometric temperatures and turbulent exchange rates associated with the soil and canopy components of the thermal pixel scene. Recent progress in separating soil and canopy temperatures from dual angle composite radiometric temperature measurements has encouraged the development of two-source (soil and canopy) approaches to estimating surface energy fluxes given observations of component soil and canopy temperatures. A Simplified Two-Source Energy Balance (STSEB) model has been developed using a “patch” treatment of the surface flux sources, which does not allow interaction between the soil and vegetation canopy components. A simple algorithm to predict the net radiation partitioning between the soil and vegetation is introduced as part of the STSEB patch modelling scheme. The feasibility of the STSEB approach under a full range in fractional vegetation cover conditions is explored using data collected over a maize (corn) crop in Beltsville Maryland, USA during the 2004 summer growing season. Measurements of soil and canopy component temperatures as well as the effective composite temperature were collected over the course of the growing season from crop emergence to cob development. Comparison with tower flux measurements yielded root-mean-square-difference values between 15 and 50 W m − 2 for the retrieval of the net radiation, soil, sensible and latent heat fluxes. A detailed sensitivity analysis of the STSEB approach to typical uncertainties in the required inputs was also conducted indicating greatest model sensitivity to soil and canopy temperature uncertainties with relative errors reaching ∼ 30% in latent heat flux estimates. With algorithms proposed to infer component temperatures from bi-angular satellite observations, the STSEB model has the capability of being applied operationally.