Thermal Image Segmentation Research Articles

U-Net architecture for high-resolution thermal image segmentation: Enhancing CFD models in smart HVAC design

This paper presents an enhanced U-Net architecture for accurately segmenting high-resolution thermal images to facilitate the design of intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems using Computational Fluid Dynamics (CFD) models. Indoor thermal comfort for occupants will improve under the same energy load by imposing comfort demands. Segmented data is fed into CFD simulations using a U-Net variant for thermal imagery. Integrating the segmented thermal data into the CFD simulations increases the model’s accuracy by 15 % for segmentation and 20% for prediction when compared to a traditional U-Net architecture. Experimental findings demonstrate the potential to achieve energy savings of up to 25 % in the simulated scenario by using the trained and validated model on a dataset of 1,500 high-resolution thermal images of various building styles. Compared to existing methods, the proposed method saves 30% in total simulation time for complex building models. Deep learning enables building the next generation of HVAC systems by pushing the boundaries of energy-efficient and environmentally friendly buildings. This study shows that advanced computer vision can be leveraged to create more innovative and sustainable built environments, leading to drastic improvements in energy efficiency.

Dual-Tree Complex Wavelet Pooling and Attention-Based Modified U-Net Architecture for Automated Breast Thermogram Segmentation and Classification.

Thermography is a non-invasive and non-contact method for detecting cancer in its initial stages by examining the temperature variation between both breasts. Preprocessing methods such as resizing, ROI (region of interest) segmentation, and augmentation are frequently used to enhance the accuracy of breast thermogram analysis. In this study, a modified U-Net architecture (DTCWAU-Net) that uses dual-tree complex wavelet transform (DTCWT) and attention gate for breast thermal image segmentation for frontal and lateral view thermograms, aiming to outline ROI for potential tumor detection, was proposed. The proposed approach achieved an average Dice coefficient of 93.03% and a sensitivity of 94.82%, showcasing its potential for accurate breast thermogram segmentation. Classification of breast thermograms into healthy or cancerous categories was carried out by extracting texture- and histogram-based features and deep features from segmented thermograms. Feature selection was performed using Neighborhood Component Analysis (NCA), followed by the application of machine learning classifiers. When compared to other state-of-the-art approaches for detecting breast cancer using a thermogram, the proposed methodology showed a higher accuracy of 99.90% for VGG16 deep features with NCA and Random Forest classifier. Simulation results expound that the proposed method can be used in breast cancer screening, facilitating early detection, and enhancing treatment outcomes.

Semantic segmentation of longitudinal thermal images for identification of hot and cool spots in urban areas

This work presents the analysis of semantically segmented, longitudinally, and spatially rich thermal images collected at the neighborhood scale to identify hot and cool spots in urban areas. An infrared observatory was operated over a few months to collect thermal images of different types of buildings on the educational campus of the National University of Singapore. A subset of the thermal image dataset was used to train state-of-the-art deep learning models to segment various urban features such as buildings, vegetation, sky, and roads. It was observed that the U-Net segmentation model with ‘resnet34’ CNN backbone has the highest mIoU score of 0.99 on the test dataset, compared to other models such as DeepLabV3, DeeplabV3+, FPN, and PSPnet. The masks generated using the segmentation models were then used to extract the temperature from thermal images and correct for differences in the emissivity of various urban features. Further, various statistical measures of the temperature extracted using the predicted segmentation masks are shown to closely match the temperature extracted using the ground truth masks. Finally, the masks were used to identify hot and cool spots in the urban feature at various instances of time. This forms one of the very few studies demonstrating the automated analysis of thermal images, which can be of potential use to urban planners for devising mitigation strategies for reducing the urban heat island (UHI) effect, improving building energy efficiency, and maximizing outdoor thermal comfort.

An Edge-Guided Deep Learning Solar Panel Hotspot Thermal Image Segmentation Algorithm

To overcome the deficiencies in segmenting hot spots from thermal infrared images, such as difficulty extracting the edge features, low accuracy, and a high missed detection rate, an improved Mask R-CNN photovoltaic hot spot thermal image segmentation algorithm has been proposed in this paper. Firstly, the edge image features of hot spots were extracted based on residual neural networks. Secondly, by combining the feature pyramid structure, an edge-guided feature pyramid structure was designed, and the hot spot edge features were injected into a Mask R-CNN network. Thirdly, an infrared spatial attention module was introduced into the Mask R-CNN network when feature extraction and the infrared features of the detected hot spots were enhanced. Fourthly, the size ratio of the candidate frames was adjusted self-adaptively according to the structural characteristics of the aspect ratio of the hot spots. Finally, the validation experiments were conducted, and the results demonstrated that the hot spot contours of thermal infrared images were enhanced through the algorithm proposed in this paper, and the segmentation accuracy was significantly improved.

Feet Segmentation for Regional Analgesia Monitoring Using Convolutional RFF and Layer-Wise Weighted CAM Interpretability

Regional neuraxial analgesia for pain relief during labor is a universally accepted, safe, and effective procedure involving administering medication into the epidural. Still, an adequate assessment requires continuous patient monitoring after catheter placement. This research introduces a cutting-edge semantic thermal image segmentation method emphasizing superior interpretability for regional neuraxial analgesia monitoring. Namely, we propose a novel Convolutional Random Fourier Features-based approach, termed CRFFg, and custom-designed layer-wise weighted class-activation maps created explicitly for foot segmentation. Our method aims to enhance three well-known semantic segmentation (FCN, UNet, and ResUNet). We have rigorously evaluated our methodology on a challenging dataset of foot thermal images from pregnant women who underwent epidural anesthesia. Its limited size and significant variability distinguish this dataset. Furthermore, our validation results indicate that our proposed methodology not only delivers competitive results in foot segmentation but also significantly improves the explainability of the process.

Digital infrared thermal imaging system based breast cancer diagnosis using 4D U-Net segmentation

Medical Research field has been taken continuous efforts to develop an efficient method for detecting breast cancer, but the goal has still not yet achieved. To overcome this issue, a 4D U-Net segmentation using digital infrared (IR) thermal imaging system is proposed in this manuscript for the diagnosis of breast cancer (DBC-4D U-Net-DITI). Initially, the digital infrared thermal images are taken from DMR-IR data set as input, and the imageries are pre-processed to maintain local features and compress the dynamic range of image based upon Altered Phase Preserving Dynamic Range Compression (APPDRC) approach by removing the speckle noise. Then, the image segmentation is carried out with the help of 4D U-Net for obtaining the segmented digital infrared thermal image. The 4D U-Net weight parameters are optimized with Glowworm Swarm Optimization Algorithm (GSOA). The segmented regions of digital infrared thermal images are fed to Binarized Spiking Neural Network (BSNN) for classifying the pathology stage as No spread, Early Stage, Localized, Regional and Distant. The proposed approach is executed in MATLAB. The performance of proposed approach attains better accuracy of 39.01%, 28.34%, and 37.45%, better precision of 17.12%, 24.12% and 32.07% when compared to existing approaches like chaotic salp swarm algorithm (CSSA) based segmentation of thermal images for breast cancer identification (DBC-CSSA-DITI), marine-predators-algorithm based segmentation of thermal images for the diagnosis of breast cancer (DBC-MPA-DITI) and diagnosis of breast cancer based upon CNN using thermal imageries (DBC-CNN-DITI) respectively.

Automated segmentation and classification of hand thermal images in rheumatoid arthritis using machine learning algorithms: A comparison with quantum machine learning technique

The aims and objectives of the study were to i) perform image segmentation using a color-based k-means clustering algorithm and feature extraction using binary robust invariant scalable key points (BRISK), maximum stable extremal regions (MSER), features from accelerated segment test (FAST), Harris, and orientated FAST and rotated BRIEF (ORB); ii) compare the performance of classical machine learning techniques such as LogitBoost, Bagging, and SVM with a quantum machine learning technique. For the proposed study, 240 hand thermal images were acquired in the dorsal view and ventral view of both the right and left-hand regions of RA and normal subjects. The hot spot regions from the thermograms were segmented using a color-based k-means clustering technique. The features from the segmented hot spot region were extracted using different feature extraction methods. Finally, normal and RA groups were categorized using LogitBoost, Bagging, and support vector machine (SVM) classifiers. The proposed study used two testing methods, such as 10-fold cross-validation and a percentage split of 80–20%. The LogitBoost classifier outperformed with an accuracy of 93.75% using the 10-fold cross-validation technique compared to other classifiers. Also, the quantum support vector machine (QSVM) obtained a prediction accuracy of 92.7%. Furthermore, the QSVM model reduces the computational cost and training time of the model to classify the RA and normal subjects. Thus, thermograms with classical machine learning and quantum machine learning algorithms could be considered a feasible technique for classifying normal and RA groups.

MPSA: A multi-level pixel spatial attention network for thermal image segmentation based on Deeplabv3+ architecture

The thermal image segmentation has been widely concerned to alleviate the limitations of image segmentation caused by imaging of visible spectrum under challenge environmental conditions. The existing thermal image segmentation methods are not easy to obtain high quality segmented results, because thermal images are lack of color information, edges are unclear, details are not obvious. To this end, we proposed a multi-level pixel spatial attention network for thermal image segmentation. Specifically, we designed a pixel space attention module on each layer of the backbone network(MPAM), which recovers the more spatial detailed and maintains more semantic information. Then, we designed an edge extraction module (EEM) and a small target extraction module (STEM), which enhances the edge and small target features of the network by explicitly modeling the edge and small target features. Finally, the edge and small target features were fused with the output features of backbone, and the specialized loss functions was used to supervise them. Experimental results on SCUT-SEG, SODA and STI-Cityscpaes dataset demonstrate that our approach is slightly improved by 2.2% compared with other the state-of-art algorithms in the same scene.

A method for improving semantic segmentation using thermographic images in infants

BackgroundRegulation of temperature is clinically important in the care of neonates because it has a significant impact on prognosis. Although probes that make contact with the skin are widely used to monitor temperature and provide spot central and peripheral temperature information, they do not provide details of the temperature distribution around the body. Although it is possible to obtain detailed temperature distributions using multiple probes, this is not clinically practical. Thermographic techniques have been reported for measurement of temperature distribution in infants. However, as these methods require manual selection of the regions of interest (ROIs), they are not suitable for introduction into clinical settings in hospitals. Here, we describe a method for segmentation of thermal images that enables continuous quantitative contactless monitoring of the temperature distribution over the whole body of neonates.MethodsThe semantic segmentation method, U-Net, was applied to thermal images of infants. The optimal combination of Weight Normalization, Group Normalization, and Flexible Rectified Linear Unit (FReLU) was evaluated. U-Net Generative Adversarial Network (U-Net GAN) was applied to thermal images, and a Self-Attention (SA) module was finally applied to U-Net GAN (U-Net GAN + SA) to improve precision. The semantic segmentation performance of these methods was evaluated.ResultsThe optimal semantic segmentation performance was obtained with application of FReLU and Group Normalization to U-Net, showing accuracy of 92.9% and Mean Intersection over Union (mIoU) of 64.5%. U-Net GAN improved the performance, yielding accuracy of 93.3% and mIoU of 66.9%, and U-Net GAN + SA showed further improvement with accuracy of 93.5% and mIoU of 70.4%.ConclusionsFReLU and Group Normalization are appropriate semantic segmentation methods for application to neonatal thermal images. U-Net GAN and U-Net GAN + SA significantly improved the mIoU of segmentation.

Automated Segmentation and Classification of Hand Thermal Images in Rheumatoid Arthritis Using Machine Learning Algorithms: A Comparison with Quantum Machine Learning Technique

Automated Segmentation and Classification of Hand Thermal Images in Rheumatoid Arthritis Using Machine Learning Algorithms: A Comparison with Quantum Machine Learning Technique

Electrical Thermal Image Semantic Segmentation: Large-Scale Dataset and Baseline

The fault diagnosis of electrical equipment plays a vital role in the safe operation of the power system. The task of electrical thermal image semantic segmentation is to segment all electrical equipment in thermal images, which is a key step in automatic fault diagnosis of electrical equipment. However, there lacks a large-scale dataset in this research field. Therefore, we contribute a large-scale dataset for electrical thermal image semantic segmentation. It contains 4839 thermal images and 17 types of electrical equipment. We provide pixel-level annotations to facilitate performance evaluation of different semantic segmentation algorithms. To provide a strong baseline, we propose a Cross Guidance Network (CGNet), which jointly infers semantic segmentation maps and edge extraction results in an end-to-end learning framework, for electrical thermal image semantic segmentation. Extensive experiments on our launched dataset demonstrate the effectiveness of the proposed CGNet, and it also achieves the best performance on the general thermal image segmentation dataset. We will release our code and dataset at https://github.com/guo49/CGNet-pytorch.

Double Thresholding with Sine Entropy for Thermal Image Segmentation

Traditional thresholding methods are often used for image segmentation of real images. However, due to distinct characteristics of infrared thermal images, it is difficult to ensure an optimal image segmentation using the traditional thresholding algorithms, and therefore, sometimes this can lead to over-segmentation, missing object information, and/or spurious responses in the output. To overcome these issues, we propose a new thresholding technique that makes use of the sine entropy-based criterion. Moreover, we build a double thresholding technique that makes use of two thresholds to get the final image thresholding result. Besides, we introduce the sine entropy concept as a supplement of the Shannon entropy in creating threshold-dependent criterion derived from the grayscale histogram. We found that the sine entropy is more robust in interpreting the strength of the long-range correlation in the gray levels compared to the Shannon entropy. We have experimented with our method on several infrared thermal images collected from standard image databases to describe the performance. On comparing with the state-of-art methods, the qualitative results from the experiments show that the proposed method achieves the best performance with an average sensitivity of 0.98 and an average misclassification error of 0.01, and second-best performance with an average sensitivity of 0.99 and an average specificity of 0.93.

Turbulent Thermal Image Velocimetry at the Immediate Fire and Atmospheric Interface

AbstractWe present novel in‐field vegetation fire observations and the analyses using brightness temperatures recorded by longwave infrared camera and thermal image velocimetry. The brightness temperatures from a wind‐driven stubble wheat fire were obtained with a 60 frames per second (fps) video acquisition. Multi‐level sonic anemometers on a 10m in‐fire tower were used for measurements of turbulent velocity and air temperatures, while fuel level air and flame temperatures were collected by an array of thermocouples (TCs). The camera's image pixel resolution was adequate to discern thermal structures in accordance with the in‐fire TC measurements and its spacing distances. The in‐situ and remotely measured flaming zone dynamics were derived using two different methodologies, Thermal Image Velocimetry (TIV) and Image Segmentation (IS). The results highlight spatial and spectral information of coherent turbulent and mean velocity structures. The power spectra decomposition of the TIV showed similar spectral characteristics to the sonic speed measurements during the fire passage under the tower with a similar inertial subrange slope. This result reveals plausible evidence of interaction between the flaming zone and wind turbulence for a prescribed rapidly moving stubble wheat fire. This research presents a new field measurement methodology for understanding fire‐atmospheric interactions between the flaming zone and the immediate overlying atmospheric turbulent boundary layer.

MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation

In this letter, we propose a multi-spectral unsupervised domain adaptation for thermal image semantic segmentation. The proposed framework aims to address the data scarcity problem and boost segmentation performance in the thermal domain with the help of existing large-scale RGB datasets and segmentation knowledge from an RGB image segmentation network. We also enhance the generalization capability of our thermal segmentation network with pixel-level domain adaptation bridging day and night thermal image domains. With our framework, a thermal image segmentation network can achieve high performance without any ground-truth labels by exploiting successive multi-spectral knowledge transfers including RGB-to-RGB, RGB-to-Thermal, and Thermal-to-Thermal adaptations. Moreover, we provide a real-world RGB-Thermal semantic segmentation dataset with 950 manually annotated Cityscapes-style ground-truth labels in 19 classes. Experimental results on real-world datasets demonstrate the effectiveness and robustness of the proposed framework quantitatively and qualitatively.

TheLNet270v1 - A Novel Deep-Network Architecture for the Automatic Classification of Thermal Images for Greenhouse Plants.

The real challenge for separating leaf pixels from background pixels in thermal images is associated with various factors such as the amount of emitted and reflected thermal radiation from the targeted plant, absorption of reflected radiation by the humidity of the greenhouse, and the outside environment. We proposed TheLNet270v1 (thermal leaf network with 270 layers version 1) to recover the leaf canopy from its background in real time with higher accuracy than previous systems. The proposed network had an accuracy of 91% (mean boundary F1 score or BF score) to distinguish canopy pixels from background pixels and then segment the image into two classes: leaf and background. We evaluated the classification (segment) performance by using more than 13,766 images and obtained 95.75% training and 95.23% validation accuracies without overfitting issues. This research aimed to develop a deep learning technique for the automatic segmentation of thermal images to continuously monitor the canopy surface temperature inside a greenhouse.

Разработка мобильного комплекса для тепловизионной видео-цифровой диагностики заболеваний коров

The principles of creating a mobile integrated system for thermal imaging video-digital diagnostics of cow diseases, a block diagram of a mobile integrated system and a sequence of operations are described. The parameters of thermal imaging equipment for detecting temperature anomalies have been determined. Methods of segmentation of thermal images, histogram analysis and new methods of phase portrait analysis are selected. The features of the formation and processing of thermal images for monitoring inflammatory processes in animals have been determined.

Variational level set and fuzzy clustering for enhanced thermal image segmentation and damage assessment

One of fundamental steps in the infrared thermographic process is the accurate segmentation of defects displayed on thermal images. State-of-the art segmentation algorithms are still inefficient to background noise, which may cause poor damage detection. In this study, an infrared image segmentation algorithm combined with artificial intelligence-based technology such as the variational level set and fuzzy clustering algorithm is proposed to enhance the quality of thermal images for damage assessment. Local Shannon entropy and fuzzy membership functions are introduced into the external clustering energy in order to make the algorithm robust to the clustering segmentation of noisy images. A higher-order derivative edge detection operator is used in the regularization energy to solve the singularity in the evolution of the level set function. An internal penalty energy is also introduced into the energy functional to avoid the re-initialization of the level set function and reduce the computational time. Experimental results on thermographic data are shown to demonstrate the efficiency and robustness of the proposed methodology.

Deep Instance Segmentation of Laboratory Animals in Thermal Images

In this paper we focus on the role of deep instance segmentation of laboratory rodents in thermal images. Thermal imaging is very suitable to observe the behaviour of laboratory animals, especially in low light conditions. It is an non-intrusive method allowing to monitor the activity of animals and potentially observe some physiological changes expressed in dynamic thermal patterns. The analysis of the recorded sequence of thermal images requires smart algorithms for automatic processing of millions of thermal frames. Instance image segmentation allows to extract each animal from a frame and track its activity and thermal patterns. In this work, we adopted two instance segmentation algorithms, i.e., Mask R-CNN and TensorMask. Both methods in different configurations were applied to a set of thermal sequences, and both achieved high results. The best results were obtained for the TensorMask model, initially pre-trained on visible light images and finally trained on thermal images of rodents. The achieved mean average precision was above 90 percent, which proves that model pre-training on visible images can improve results of thermal image segmentation.

Multi-modal neural networks with multi-scale RGB-T fusion for semantic segmentation

A novel deep-learning-based method for semantic segmentation of RGB and thermal images is introduced. The proposed method employs a novel neural network design for multi-modal fusion based on multi-resolution patch processing. A novel decoder module is introduced to fuse the RGB and thermal features extracted by separate encoder streams. Experimental results on synthetic and real-world data demonstrate the efficiency of the proposed method compared with state-of-the-art methods.

Segmentation of thermal infrared images of cucumber leaves using K-means clustering for estimating leaf wetness duration

Leaf wetness duration (LWD) is a critical parameter used to predict plant disease, but its determination under actual field conditions is a major challenge. In this study, a method for determining LWD using thermal infrared imaging was developed and applied to cucumber plants grown in a solar greenhouse. Thermal images of the plant leaves were captured using an infrared scanning camera, and a leaf wetness area segmentation method consisting of two procedures was applied. First, a color space conversion was performed automatically by an image-processing algorithm. Then, the K-means clustering algorithm was applied to enable the segmentation of the wetness area on the thermal image. Subsequently, to enable overall thermal image analysis, an initial leaf wetness threshold (LWT) of 5% was defined (where wetness values higher than 5% indicated that the leaf was in a wet state). The results of comparative experiments conducted using thermal images of plant leaves captured using an infrared scanning camera and human visual observation indicated that the estimated LWD values were generally higher than the observed LWD values, because slight leaf wetness condensations were overlooked by the human eye but detected by the infrared scanning camera. While these differences were not found to be statistically significant in this study, the proposed method for determining LWD using thermal infrared imaging may provide a new LWD detection method for cucumber and other plants grown in solar greenhouses. Keywords: thermal imaging, K-means clustering algorithm, leaf wetness duration, cucumber DOI: 10.25165/j.ijabe.20201303.4301 Citation: Wen D M, Ren A X, Ji T, Flores-Parra I M, Yang X T, Li M. Segmentation of thermal infrared images of cucumber leaves using K-means clustering for estimating leaf wetness duration. Int J Agric & Biol Eng, 2020; 13(3): 161–167.