Thermal Modalities Research Articles

Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this paper, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed TINet to adaptively fuse the complementary features extracted from RGB and thermal images. Firstly, we design an illumination-guided feature weighting module to guide the network in leaning towards a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.

Dilated high-resolution network driven RGB-T multi-modal crowd counting

Crowd counting aims to estimate the number of pedestrians in a scene. However, the problems of insufficient illumination and large-scale variation affect the accuracy of crowd counting. In this paper, a dilated high-resolution network (DHRNet) driven RGB-T multi-modal crowd counting model is proposed to address the above problems. In terms of the importance of RGB and thermal modalities, a thermal-main and RGB-auxiliary strategy is chosen instead of treating them equally as in previous works. In terms of the fusion of RGB and thermal modalities, a cross-modal fusion module is designed and embedded in the input feature level of DHRNet. In terms of DHRNet output feature utilization, a multilayer perceptron regression head is proposed to predict high-quality density maps. The experimental results on public datasets show that our proposed network significantly outperforms state-of-the-art methods and reaches a new level of performance. The ablation studies verify the effectiveness of the thermal-main and RGB-auxiliary strategy and the proposed modules.

Mask-guided modality difference reduction network for RGB-T semantic segmentation

By exploiting the complementary information of RGB modality and thermal modality, RGB-thermal (RGB-T) semantic segmentation is robust to adverse lighting conditions. When fusing features from RGB images and thermal images, the existing methods design different feature fusion strategies, but most of these methods overlook the modality differences caused by different imaging mechanisms. This may result in insufficient usage of complementary information. To address this issue, we propose a novel Mask-guided Modality Difference Reduction Network (MMDRNet), where the mask is utilized in the image reconstruction to ensure that the modality discrepancy within foreground regions is minimized. Doing so enables the generation of more discriminative representations for foreground pixels, thus facilitating the segmentation task. On top of this, we present a Dynamic Task Balance (DTB) method to balance the modality difference reduction task and semantic segmentation task dynamically. The experimental results on the MFNet dataset and the PST900 dataset demonstrate the superiority of the proposed mask-guided modality difference reduction strategy and the effectiveness of the DTB method.

Detection and Recognition of Driver Distraction Using Multimodal Signals

Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we performed experiments with visual, physiological, and thermal information to explore potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual, physiological, and thermal groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the three modalities on identifying different types of driving distractions.

DEFNet: Dual-Branch Enhanced Feature Fusion Network for RGB-T Crowd Counting

Most existing crowd counting approaches use limited information of RGB (red–green–blue) images and fail to suitably extract potential pedestrians in unconstrained scenarios. Moreover, complementary depth maps do not provide information of locations where people are more likely to be present. However, by incorporating optical and thermal information, the recognition of pedestrians may be enhanced considerably. In fact, thermal imaging information is robust to weather and lighting scenarios, and information from targets can be extracted even at nighttime. By combining RGB and thermal imaging information, we propose a dual-branch enhanced feature fusion network (DEFNet) for RGB-T (RGB and thermal) crowd counting. In DEFNet, an intensive data-enhancement module fuses complementary features of the same sizes from the RGB and thermal modalities, thus combining various rich receptive fields and generating powerful fused RGB-T features. These features describe both spatial structures and appearance details, highlighting information of crowd location. Then, an efficient dilation fusion module applies convolutions to the RGB -T features to obtain flexible and specific features, effectively eliminating the influence of background on the crowd information for density map prediction. Finally, high- and low-level features are used to efficiently obtain density maps through a fusion decoding module. Experimental results on an RGB-T crowd counting dataset indicate that the proposed DEFNet outperforms existing approaches. Furthermore, DEFNet can be generalized to handle RGB and depth data.

High-contrast spectroscopic photoacoustic characterization of thermal tissue ablation in the visible spectrum.

High-contrast tissue characterization of thermal ablation has been desired to evaluate therapeutic outcomes accurately. This paper presents a photoacoustic (PA) characterization of thermal tissue ablation in the visible spectrum, in which higher light absorbance can produce spectral contrast starker than in the near-infrared range. Ex vivo experiments were performed to measure visible PA spectra (480-700 nm) from fresh porcine liver tissues that received a thermal dose in a range of cumulative equivalent minutes at 43°C (CEM43). The local hemoglobin lobe area between 510-600 nm and wholespectral area under the curve were evaluated to represent the transition of hemoglobin into methemoglobin (MetHb) in the target tissue. The thermal process below an estimated therapeutic CEM43 threshold (80-340 minutes) presented a progressive elevation of the PA spectrum and an eventual loss of local hemoglobin peaks in the visible spectrum, closer to the MetHb spectrum. Interestingly, an excessive CEM43 produced a substantial drop in the PA spectrum. In the spectral analysis, the visible spectrum yielded 13.9-34.1 times higher PA sensitivity and 1.42 times higher contrast change than at a near-infrared wavelength. This novel method of PA tissue characterization in the visible spectrum could be a potential modality to evaluate various thermal therapeutic modalities at high-contrast resolution.

MFENet: Multitype fusion and enhancement network for detecting salient objects in RGB-T images

Recent progress in salient object detection (SOD) has been fueled substantially by the development of convolutional neural networks. However, several SOD methods do not fully exploit information from different modalities, consequently performing only marginally better than methods using a single modality. Therefore, we propose a multitype fusion and enhancement network (MFENet), following three steps “Encoder- Pre-decoder- Decoder” for RGB-thermal (RGB-T) SOD by completely exploiting the advantages of the RGB and thermal modalities through feature integration and enhancement. To better fuse two modalities' features, we have designed the cross-modality fusion module (CMFM) in the encoder part. As shallow features describe details and deep features provide semantic information, a multiscale interactive refinement module is designed in the pre-decoder part to complement multilevel features. Additionally, to further sharpen salient objects, we have proposed a high-level, low-level module that takes inputs from adjacent layers for gradual translation into a saliency map in the decoder part. This module provides semantic information for shallower features and the boundaries of salient objects can be gradually sharpened with subtle details. Extensive experiments show the effectiveness and robustness of the proposed MFENet and its substantial improvement over state-of-the-art RGB-T SOD methods. The codes and results will be available at: https://github.com/wujunyi1412/MFENet_DSP.

Quantitative Sensory Testing in Late-Onset ATTRv Presymptomatic Subjects: A Single Center Experience.

Backgrounds Hereditary transthyretin amyloidosis (ATTRv) presymptomatic subjects undergo multidisciplinary evaluation to detect, as early as possible, a subclinical involvement of multisystem disease. Quantitative sensory testing (QST) that investigates and discriminates the function of C, Aδ and Aβ fibers is included as an instrumental test to monitor nerve fiber function. The purpose of this study was to evaluate the role of QST in the context of the multidisciplinary evaluation in late onset carriers. Methods Four-teen presymptomatic (namely carriers) were enrolled. Subjects underwent thermal [cold and warm detection threshold (CDT, WDT), cold and heat pain (CP and HP)] and tactile QST in four body sites: foot dorsum, distal lateral leg, distal thigh, hand dorsum. Results Overall, presymptomatic subject showed a significant difference in all thermal QST findings compared to the control group. All subjects had at least one altered thermal QST finding; the sites more frequently altered were foot and leg, whilst the thermal modalities which were more frequently abnormal were CDT, WDT and CP. Conclusions Our study highlights the importance of performing thermal QST in subjects carrying TTR mutation, given the high frequency of abnormal findings. Notably, performing both innocuous and painful stimulation in foot and/or leg increases the chance of detecting nerve fiber dysfunction. Moreover, the investigation of the hand may provide useful information in monitoring disease progression before the Predicted Age of Disease Onset (PADO).

Electroencephalographic characteristics of children and adolescents with chronic musculoskeletal pain.

The pathophysiology of pediatric musculoskeletal (MSK) pain is unclear, contributing to persistent challenges to its management. This study hypothesizes that children and adolescents with chronic MSK pain (CPs) will show differences in electroencephalography (EEG) features at rest and during thermal pain modalities when compared with age-matched controls. One hundred forty-two CP patients and 45 age-matched healthy controls (HCs) underwent a standardized thermal tonic heat and cold stimulations, while a 21-electrode headset collected EEG data. Cohorts were compared with respect to their EEG features of spectral power, peak frequency, permutation entropy, weight phase-lag index, directed phase-lag index, and node degree at 4 frequency bands, namely, delta (1-4 Hz), theta (4-8 Hz), alpha (8-13 Hz), and beta (13-30 Hz), at rest and during the thermal conditions. At rest, CPs showed increased global delta (P = 0.0493) and beta (P = 0.0002) power in comparison with HCs. These findings provide further impetus for the investigation and prevention of long-lasting developmental sequalae of early life chronic pain processes. Although no cohort differences in pain intensity scores were found during the thermal pain modalities, CPs and HCs showed significant difference in changes in EEG spectral power, peak frequency, permutation entropy, and network functional connectivity at specific frequency bands (P < 0.05) during the tonic heat and cold stimulations. This suggests that EEG can characterize subtle differences in heat and cold pain sensitivity in CPs. The complementation of EEG and evoked pain in the clinical assessment of pediatric chronic MSK pain can better detect underlying pain mechanisms and changes in pain sensitivity.

CSA-Net: Cross-modal scale-aware attention-aggregated network for RGB-T crowd counting

Albeit recent cross-modal crowd counting methods have achieved promising performance, most of them only focus on how to combine the RGB modality and thermal modality, and ignore the adverse effect of the scale variation issue on the cross-modal feature fusion process, where the scale variation is extensively investigated in unimodal crowd counting but less-studied in RGB-T crowd counting. Therefore, this paper aims to explore the aggregated multi-scale context in both the modality-specific and modality-shared feature extraction. Specially, we firstly introduce a novel combining way of the multi-scale analysis and attention aggregation to realize a deeply aggregated scale-aware feature representation. Then based on this combining way, the Scale-aware channel attention aggregation(SCAA) module and Scale-aware Cross-modal Feature Aggregation(SCFA) module are designed for the modality-specific feature extraction and cross-modal feature fusion process. Finally, based on SCAA module and SCFA module, we construct our cross-modal crowd counting architecture, and importantly, the backbone structure of the counting model could be substituted by other structures and thus the SCAA and SCFA are plug-and-play modules. Extensive experiments on the challenging RGB-T and RGB-D crowd counting benchmarks demonstrate the proposed method realizes the state-of-the-art RGB-T counting performance and could also be extended into the RGB-D crowd counting task.

Synergetic Thermal Therapy for Cancer: State-of-the-Art and the Future.

As a safe and minimal-invasive modality, thermal therapy has become an effective treatment in cancer treatment. Other than killing the tumor cells or destroying the tumor entirely, the thermal modality results in profound molecular, cellular and biological effects on both the targeted tissue, surrounding environments, and even the whole body, which has triggered the combination of the thermal therapy with other traditional therapies as chemotherapy and radiation therapy or new therapies like immunotherapy, gene therapy, etc. The combined treatments have shown encouraging therapeutic effects both in research and clinic. In this review, we have summarized the outcomes of the existing synergistic therapies, the underlying mechanisms that lead to these improvements, and the latest research in the past five years. Limitations and future directions of synergistic thermal therapy are also discussed.

CGMDRNet: Cross-Guided Modality Difference Reduction Network for RGB-T Salient Object Detection

How to explore the interaction between the RGB and thermal modalities is the key success of the RGB-T saliency object detection (SOD). Most of the existing methods integrate multi-modality information by designing various fusion strategies. However, the modality gap between the RGB and thermal features will lead to unsatisfactory performances by simple feature concatenation. To solve this problem, we innovatively propose a cross-guided modality difference reduction network (CGMDRNet) to achieve intrinsic consistency feature fusion via reducing the modality differences. Specifically, we design a modality difference reduction (MDR) module, which is embedded in each layer of the backbone network. The module uses a cross-guided strategy to reduce the modality difference between the RGB and thermal features. Then, a cross-attention fusion (CAF) module is designed to fuse cross-modality features with small modality differences. In addition, we use a transformer-based feature enhancement (TFE) module to enhance the high-level feature representation that contributes more to performance. Finally, the high-level features guide the fusion of low-level features to obtain a saliency map with clear boundaries. Extensive experiments on three public RGB-T datasets show that the proposed CGMDRNet achieves competitive performance compared with state-of-the-art (SOTA) RGB-T SOD models.

Burden And Rehabilitation Care Of Chronic Ankle Instability In Military Tactical Athletes. A Retrospective Population Study.

PURPOSE: To assess estimated population-level burden of Chronic Ankle Instability (CAI) and rehabilitation care patterns in the US military health system from 2000 to 2019. METHODS: The Military Health System Data Repository was used to identify cases with an outpatient diagnosis of lateral ankle sprain (LAS) and associated rehabilitation care in all active-duty service members from 2000 to 2019. Individuals with one care episode were classified as index LAS; two care episodes separated by ≥365 days as “possible CAI;” and cases with ≥3 care episodes (>365 days from index) as CAI. Multinomial logistic regression assessed the effects of sex, time in service, service branch, age at index sprain, and number of deployments on the diagnostic classification. RESULTS: A total of 422,978 military tactical athletes sought care for LAS during the study epoch (22,262 cases annually), with 81.8% of the cases (n = 346,119) classified as having an index LAS, 12.8% (n = 54,133) as possible CAI, and 5.4%, (n = 22,726) as CAI. When pooled, the cases classified as either possible CAI or CAI comprised 18.2% of all lateral ankle injuries (n = 76,859). Several significant factors were associated with injury chronification: female sex (OR: 1.05 to 1.25), age < 30 years (OR: 2.13 to 3.30), >7 years of service (OR: 2.17 to 5.38), and ≥ 2 deployments (OR 1.13 to 1.20). There were no other significant associations. Only 12.3% of individuals with LAS, 24.9% with possible CAI, and 59.0% with CAI were evaluated by a rehabilitation specialist. Rehabilitation care consisted primarily of therapeutic or aquatic exercises (3.5 to 5.1 units/episode), thermal modalities (1.8 to 2.2 units/episode), and functional/sensorimotor re-education interventions (1.6 to 2.2 units per episode). Of those provided rehabilitation care, 71% of index LAS, 68% of possible CAI, and 90% of the CAI group received care beyond 30 days. CONCLUSION: There was a substantial burden of LAS and CAI observed in military tactical athletes, with several salient factors associated with injury chronification. Only a small proportion were provided rehabilitation care. To our knowledge, this is the first study to estimate the epidemiological burden of CAI in the military population.

Industrial vision and automation

Abstract This volume of Measurement Science and Technology contains a Special Feature on Industrial Vision and Automation. The special feature puts together research articles within the common denominator of Industrial Vision &amp; Automation, especially for evaluation of various materials. This tried to demonstrate the breadth of applications for which one can use various vision systems, together with very recent research developments, some clear demonstrations of the method at work in applications and some of the necessary background theory that underpins the basic vision modalities. This special feature based on the adopted imaging methodology and the principles involved. Further articles in this Special Feature are sub-classified in to optical, thermal, X-ray and acoustical vision modalities as presented below with a brief description of the accepted manuscripts in each sub-section. Optical Vision Systems: An alternative technique is proposed to improve the visual detection rate of longitudinal conveyor belt tears by Gongxian Wang et al [1]; Automatic defect detection in multi-crystalline solar wafers using deep learning is presented by Du-Ming Tsai et al [2] Thermo-Vision Systems: An efficient denoising CAE with U-net architecture is proposed for thermal data to solve the problem of an insufficient dataset has been demonstrated in the work of Xiaoyuan Li et al [3]; Exploratory factor analysis for defect identification with active thermography is presented by Kai-Lun Huang et al [4]; X-ray Vision Systems: Coded aperture x-ray computed tomography `by sparsity-driven deterministic sampling strategy is demonstrated by Munnu Sonkar et al [5]; Acoustical Vision Systems: Phase velocity measurement of dispersive wave modes by Gaussian peak-tracing using f-k transform is presented by Bikash Ghose et al [6]; Coplanar electrical/ultrasonic dual-modality tomography for water continuous gas/oil/water three-phase distribution imaging is demonstrated by Guanghui Liang et al [7]; Defect visualization of cylindrical and conical CFRP lattice structures using rotational ultrasonic propagation imager is highlighted by J H An et al [8]; We are sure that readers will find this as a valuable special feature and a useful reference resource. Finally, we would like to thank all the authors for their contributions, the referees for their effort to thoroughly review the manuscripts by providing their valuable feedback to enhance the quality and the entire Editorial Board of Measurement Science and Technology for their constant support.

Non-surgical Treatment of Metastatic Liver Tumors with Microwave Ablation.

With all the measures taken, the trend of liver cancer cases has increased worldwide, as a consequence, this is accompanied by an increase in mortality by 43% (10.3 per 100,000 in 2016 (USA)) [1, 2].&#x0D; Liver cancer is also associated with the lowest 5-year survival rate among all types of cancer (19%) [1].&#x0D; Liver cancer (including intrahepatic bile duct cancer) was the ninth leading cause of cancer death in 2000 and rose to sixth in 2016 [3].&#x0D; This shows that the identification of safe and effective treatments for liver cancer is urgent now.&#x0D; Microwave ablation is a thermal ablation modality that has particular applicability in treating hepatic malignancies primary tumors or metastases. Microwaves can generate very high temperatures in short time periods, potentially leading to improved treatment efficiency in larger ablation zones.&#x0D; We will show a patient male 55 years old. Post-surgery of colon cancer 6 months ago. Normal exam. There is e liver metastasis in the right lobe subcapsular 22mm. The patient was a good candidate for MWA.&#x0D; Conclusions&#x0D; MWA is an effective and safe alternative in patients/tumors that are not suitable for resection. Survival and recurrence outcomes after MWA are significantly improved with significantly shorter hospital stays and operative times, with little or no intraoperative blood loss and minor complications. Its use should be extended more and more in the field of treatment of these patients depending on the therapeutic indications

Early Prediction of Hemodynamic Shock in Pediatric Intensive Care Units With Deep Learning on Thermal Videos.

Shock is one of the major killers in intensive care units, and early interventions can potentially reverse it. In this study, we advance a noncontact thermal imaging modality for continuous monitoring of hemodynamic shock working on 1,03,936 frames from 406 videos recorded longitudinally upon 22 pediatric patients. Deep learning was used to preprocess and extract the Center-to-Peripheral Difference (CPD) in temperature values from the videos. This time-series data along with the heart rate was finally analyzed using Long-Short Term Memory models to predict the shock status up to the next 6 h. Our models achieved the best area under the receiver operating characteristic curve of 0.81 ± 0.06 and area under the precision-recall curve of 0.78 ± 0.05 at 5 h, providing sufficient time to stabilize the patient. Our approach, thus, provides a reliable shock prediction using an automated decision pipeline that can provide better care and save lives.

Adaptive Fusion CNN Features for RGBT Object Tracking

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory

Recently, automated surveillance cameras can change a visible sensor and a thermal sensor for all-day operation. However, existing single-modal pedestrian detectors mainly focus on detecting pedestrians in only one specific modality (i.e., visible or thermal), so they cannot cope with other modal inputs. In addition, recent multispectral pedestrian detectors have shown remarkable performance by adopting multispectral modalities, but they also have limitations in practical applications (e.g., different Field-of-View (FoV) and frame rate). In this paper, we introduce a versatile pedestrian detector that shows robust detection performance in any single modality. We propose a multisensory-matching contrastive loss to reduce the difference between the visual representation of pedestrians in the visible and thermal modalities. Moreover, for the robust detection on a single modality, we design a Multispectral Recalling (MSR) Memory. The MSR Memory enhances the visual representation of the single modal features by recalling that of the multispectral modalities. To guide the MSR Memory to store the multispectral modal contexts, we introduce a multispectral recalling loss. It enables the pedestrian detector to encode more discriminative features with a single input modality. We believe our method is a step forward detector that can be applied to a variety of real-world applications. The comprehensive experimental results verify the effectiveness of the proposed method.

Edge-Aware Guidance Fusion Network for RGB–Thermal Scene Parsing

RGB–thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high-level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB–thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed information in the prediction map and then embed the prior edge information in the feature maps. To effectively fuse the RGB and thermal information, we propose a multimodal fusion module that guarantees adequate cross-modal fusion. Considering the importance of high-level semantic information, we propose a global information module and a semantic information module to extract rich semantic information from the high-level features. For decoding, we use simple elementwise addition for cascaded feature fusion. Finally, to improve the parsing accuracy, we apply multitask deep supervision to the semantic and boundary maps. Extensive experiments were performed on benchmark datasets to demonstrate the effectiveness of the proposed EGFNet and its superior performance compared with state-of-the-art methods. The code and results can be found at https://github.com/ShaohuaDong2021/EGFNet.

Pixelated bicontrollable metasurface absorber tunable in complete X band

A pixelated bicontrollable metasurface tunable in the complete X band was designed with its meta-atoms comprising yttrium iron garnet (YIG)-patched pixels and conductive rubber (CR)-patched pixels on a metal-backed silicon substrate. Maximum absorptance of a normally incident plane wave greater than 0.99 is achievable and tunability with absorptance exceeding 0.9 is realizable, with the help of an external magnetostatic field and temperature. The YIG-patched pixels provide magnetostatic control modality with sensitivity of and the CR-patched pixels provide thermal control modality with sensitivity of . The proposed metasurface absorber can be used to improve the performance of weather radar, police speed radar, and direct broadcast television.