Pixel Level Research Articles

CrackCLIP: Adapting Vision-Language Models for Weakly Supervised Crack Segmentation

Weakly supervised crack segmentation aims to create pixel-level crack masks with minimal human annotation, which often only differentiate between crack and normal no-crack patches. This task is crucial for assessing structural integrity and safety in real-world industrial applications, where manually labeling the location of cracks at the pixel level is both labor-intensive and impractical. Addressing the challenges of labeling uncertainty, this paper presents CrackCLIP, a novel approach that leverages language prompts to augment the semantic context and employs the Contrastive Language–Image Pre-Training (CLIP) model to enhance weakly supervised crack segmentation. Initially, a gradient-based class activation map is used to generate pixel-level coarse pseudo-labels from a trained crack patch classifier. The estimated coarse pseudo-labels are utilized to fine-tune additional linear adapters, which are integrated into the frozen image encoders of CLIP to adapt the CLIP model to the specialized task of crack segmentation. Moreover, specific textual prompts are crafted for crack characteristics, which are input into the frozen text encoder of CLIP to extract features encapsulating the semantic essence of the cracks. The final crack segmentation is determined by comparing the similarity between text prompt features and visual patch token features. Comparative experiments on the Crack500, CFD, and DeepCrack datasets demonstrate that the proposed framework outperforms existing weakly supervised crack segmentation methods, and the pre-trained vision-language model exhibits strong potential for crack feature learning, thereby enhancing the overall performance and generalization capabilities of the proposed framework.

Regional Image Quality Scoring for 2-D Echocardiography Using Deep Learning.

To develop and compare methods to automatically estimate regional ultrasound image quality for echocardiography separate from view correctness. Three methods for estimating image quality were developed: (i) classic pixel-based metric: the generalized contrast-to-noise ratio (gCNR), computed on myocardial segments (region of interest) and left ventricle lumen (background), extracted by a U-Net segmentation model; (ii) local image coherence: the average local coherence as predicted by a U-Net model that predicts image coherence from B-mode ultrasound images at the pixel level; (iii) deep convolutional network: an end-to-end deep-learning model that predicts the quality of each region in the image directly. These methods were evaluated against manual regional quality annotations provided by three experienced cardiologists. The results indicated poor performance of the gCNR metric, with Spearman correlation to annotations of ρ = 0.24. The end-to-end learning model obtained the best result, ρ = 0.69, comparable to the inter-observer correlation, ρ = 0.63. Finally, the coherence-based method, with ρ = 0.58, out-performed the classical metrics and was more generic than the end-to-end approach. The deep convolutional network provided the most accurate regional quality prediction, while the coherence-based method offered a more generalizable solution. gCNR showed limited effectiveness in this study. The image quality prediction tool is available as an open-source Python library at https://github.com/GillesVanDeVyver/arqee.

P0040 Measurement of collagen concentration throughout the intestinal layers using Deep Learning: implications in Inflammatory Bowel Disease (IBD)

Abstract Background IBD, particularly Crohn’s disease (CD), is marked by recurrent episodes of inflammation, leading to fibrosis and collagen deposition throughout the intestinal layers.1,2 In this study, we aim to quantify collagen levels across the distinct layers of the gut and examine their correlation with clinical characteristics and disease outcomes. Methods A total of 190 IBD patients were enrolled, including 98 with Ulcerative Colitis (UC) and 92 with CD (60% male; median age: 42 years, IQR: 29–48; follow-up: 110 months, IQR: 39-181), plus 73 controls. Biopsies were collected, stained with Sirius Red, and digitized. A total of 612 biopsies were included. Intestinal layers in biopsy images from the derivation group were manually annotated to train a mask Region-based Convolution Neural Network (mask R-CNN), which uses ResNet-101 backbone. After the detection of intestinal layers by the trained model, K-means was employed at the pixel level, to segment and quantify Collagen Proportionate Area (CPA) regions in each layer. Follow-up biopsies (>6 months) were available for 85 patients. Multivariate Cox regression was used to assess the risk of adverse outcomes, including variables with p<0.05 from the univariate analysis. Results Submucosal collagen was significantly correlated with mucosal CPA (r=0.36; p<0.001) and CPA in the muscular layer (r=0.54; p<0.001), as well as with C-reactive protein (r=0.38; p=0.001) and erythrocyte sedimentation rate (r=0.53; p=0.005). In CD, submucosal CPA showed a positive association with disease activity as measured by the Crohn’s Disease Activity Index (CDAI) (r=0.31; p=0.005), endoscopic activity assessed by the Simple Endoscopic Score for CD (SES-CD) (r=0.32; p=0.004), and Nancy histologic activity score (r=0.25; p=0.046). In UC, the Full Mayo score showed a trend with submucosal CPA, but it did not reach significance (r=0.13; p=0.09). Patients requiring treatment with biologics had higher baseline submucosal CPA levels (mean difference: 12; 95%CI: -0.6 to 24.6; p=0.06). Patients treated with biologics at follow-up biopsies (19/85) experienced a significantly greater reduction in submucosal CPA (-21.1; 95% CI: -38.3 to -3.9) compared to those not receiving biologics (-1.7; 95% CI: -9.4 to 6.1; p = 0.039). Baseline submucosal CPA was independently associated with an increased risk of surgery in CD patients during follow-up (aHR=1.03; 95%CI: 1.002–1.06; p=0.048), along with B2/B3 disease behavior (p=0.003) and elevated CDAI (p=0.004) (Table I). Conclusion Submucosal collagen levels correlate with clinical, endoscopic, and histologic activity in CD, potentially serving as a prognostic marker for long-term outcomes. Emerging therapies may help reduce collagen accumulation and its associated complications.

Binocular Video-Based Automatic Pixel-Level Crack Detection and Quantification Using Deep Convolutional Neural Networks for Concrete Structures

Crack detection and quantification play crucial roles in assessing the condition of concrete structures. Herein, a novel real-time crack detection and quantification method that leverages binocular vision and a lightweight deep learning model is proposed. In this methodology, the proposed method based on the following four modules is adopted: a lightweight classification algorithm, a high-precision segmentation algorithm, a semi-global block matching algorithm (SGBM), and a crack quantification technique. Based on the crack segmentation results, a framework is developed for quantitative analysis of the major geometric parameters, including crack length, crack width, and crack angle of orientation at the pixel level. Results indicate that, by incorporating channel attention and spatial attention mechanisms in the MBConv module, the detection accuracy of the improved EfficientNetV2 increased by 1.6% compared with the original EfficientNetV2. Results indicate that using the proposed quantification method can achieve low quantification errors of 2%, 4.5%, and 4% for the crack length, width, and angle of orientation, respectively. The proposed method can contribute to crack detection and quantification in practical use by being deployed on smart devices.

Selecting Relevant Features for Random Forest-Based Crop Type Classifications by Spatial Assessments of Backward Feature Reduction

AbstractRandom Forest (RF) is a widely used machine learning algorithm for crop type mapping. RF’s variable importance aids in dimension reduction and identifying relevant multisource hyperspectral data. In this study, we examined spatial effects in a sequential backward feature elimination setting using RF variable importance in the example of a large-scale irrigation system in Punjab, Pakistan. We generated a reference classification with RF applied to 122 SAR and optical features from time series data of Sentinel‑1 and Sentinel‑2, respectively. We ranked features based on variable importance and iteratively repeated the classification by excluding the least important feature, assessing its agreement with the reference classification. McNemar’s test identified the critical point where feature reduction significantly affected the RF model’s predictions. Additionally, spatial assessment metrics were monitored at the pixel level, including spatial confidence (number of classifications agreeing with the reference map) and spatial instability (number of classes occurring during feature reduction). This process was repeated 10 times with ten distinct stratified random sampling splits, which showed similar variable rankings and critical points. In particular, VH SAR data was selected when cloud-free optical observations were unavailable. Omitting 80% of the features resulted in an insignificant loss of only 2% overall accuracy, while spatial confidence decreased by 5%. Moreover, the crop map at the critical point exhibited an increase in spatial instability from a single crop to 1.28. McNemar’s test and the spatial assessment metrics are recommended for optimized feature reduction benchmarks and identifying areas requiring additional ground data to improve the results.

Automated recognition and segmentation of lung cancer cytological images based on deep learning.

Compared with histological examination of lung cancer, cytology is less invasive and provides better preservation of complete morphology and detail. However, traditional cytological diagnosis requires an experienced pathologist to evaluate all sections individually under a microscope, which is a time-consuming process with low interobserver consistency. With the development of deep neural networks, the You Only Look Once (YOLO) object-detection model has been recognized for its impressive speed and accuracy. Thus, in this study, we developed a model for intraoperative cytological segmentation of pulmonary lesions based on the YOLOv8 algorithm, which labels each instance by segmenting the image at the pixel level. The model achieved a mean pixel accuracy and mean intersection over union of 0.80 and 0.70, respectively, on the test set. At the image level, the accuracy and area under the receiver operating characteristic curve values for malignant and benign (or normal) lesions were 91.0% and 0.90, respectively. In addition, the model was deemed suitable for diagnosing pleural fluid cytology and bronchoalveolar lavage fluid cytology images. The model predictions were strongly correlated with pathologist diagnoses and the gold standard, indicating the model's ability to make clinical-level decisions during initial diagnosis. Thus, the proposed method is useful for rapidly localizing lung cancer cells based on microscopic images and outputting image interpretation results.

Towards a more objective and high-throughput spheroid invasion assay quantification method

Multicellular spheroids embedded in 3D hydrogels are prominent in vitro models for 3D cell invasion. Yet, quantification methods for spheroid cell invasion that are high‐throughput, objective and accessible are still lacking. Variations in spheroid sizes and the shapes of the cells within render it difficult to objectively assess invasion extent. The goal of this work is to develop a high-throughput quantification method of cell invasion into 3D matrices that minimizes sensitivity to initial spheroid size and cell spreading and provides precise integrative directionally-dependent metrics of invasion. By analyzing images of fluorescent cell nuclei, invasion metrics are automatically calculated at the pixel level. The initial spheroid boundary is segmented and automated calculations of the nuclear pixel distances from the initial boundary are used to compute common invasion metrics (i.e., the change in invasion area, mean distance) for the same spheroid at a later timepoint. We also introduce the area moment of inertia as an integrative metric of cell invasion that considers the invasion area as well as the pixel distances from the initial spheroid boundary. Further, we show that principal component analysis can be used to quantify the directional influence of a stimuli to invasion (e.g., due to a chemotactic gradient or contact guidance). To demonstrate the power of the analysis for cell types with different invasive potentials and the utility of this method for a variety of biological applications, the method is used to analyze the invasiveness of five different cell types. In all, implementation of this high‐throughput quantification method results in consistent and objective analysis of 3D multicellular spheroid invasion. We provide the analysis code in both MATLAB and Python languages as well as a GUI for ease of use for researchers with a range of computer programming skills and for applications in a variety of biological research areas such as wound healing and cancer metastasis.

An Unsupervised Remote Sensing Image Change Detection Method Based on RVMamba and Posterior Probability Space Change Vector

Change vector analysis in posterior probability space (CVAPS) is an effective change detection (CD) framework that does not require sound radiometric correction and is robust against accumulated classification errors. Based on training samples within target images, CVAPS can generate a uniformly scaled change-magnitude map that is suitable for a global threshold. However, vigorous user intervention is required to achieve optimal performance. Therefore, to eliminate user intervention and retain the merit of CVAPS, an unsupervised CVAPS (UCVAPS) CD method, RFCC, which does not require rigorous user training, is proposed in this study. In the RFCC, we propose an unsupervised remote sensing image segmentation algorithm based on the Mamba model, i.e., RVMamba differentiable feature clustering, which introduces two loss functions as constraints to ensure that RVMamba achieves accurate segmentation results and to supply the CSBN module with high-quality training samples. In the CD module, the fuzzy C-means clustering (FCM) algorithm decomposes mixed pixels into multiple signal classes, thereby alleviating cumulative clustering errors. Then, a context-sensitive Bayesian network (CSBN) model is introduced to incorporate spatial information at the pixel level to estimate the corresponding posterior probability vector. Thus, it is suitable for high-resolution remote sensing (HRRS) imagery. Finally, the UCVAPS framework can generate a uniformly scaled change-magnitude map that is suitable for the global threshold and can produce accurate CD results. The experimental results on seven change detection datasets confirmed that the proposed method outperforms five state-of-the-art competitive CD methods.

Radar Signal Sorting Method with Mimetic Image Mapping Based on Antenna Scan Pattern via SOLOv2 Network

Aiming at the problems, in which the traditional radar signal sorting method has high requirements for manual experience and poor adaptability, and considering the differences in received power caused by radar beam scanning under long-term observation, an end-to-end signal sorting method based on the instance segmentation network SOLOv2 and using an antenna scan pattern (ASP) is proposed in this letter. Firstly, the interleaved pulse sequences of multiple radar signals with various inter-pulse modulation types, scan patterns, and gain patterns are simulated, mimetic image mapping is constructed to visualize the interleaved pulse sequences as mimetic point graphs, and the index relationship between pulses and pixel points is recorded. Subsequently, the SOLOv2 instance segmentation network is used to segment the mimetic point graph at the pixel level, thereby clustering the discrete pixel points in the image. Finally, based on the index relationship recorded during the construction of the mimetic image mapping, the clustering results of points in the image are traced back to the clustering of pulses, achieving end-to-end intelligent radar signal sorting. Through simulation experiments, it was verified that, compared with YOLOv8-based, U-Net-based, and traditional signal sorting methods, the sorting accuracy of the proposed method increased by 9.26%, 11.17%, and 24.55% in the scenario of five signals with 30% missing pulse ratio (MPR), and increased by 13.33%, 18.88%, and 23.94% in the scenario of five signals with 30% spurious pulse ratio (SPR), respectively. The results show that by introducing the stable parameter, namely ASP, the proposed method can achieve signal sorting with highly overlapping parameters and adapt to non-ideal conditions with measurement errors, missing pulses, and spurious pulses.

An Overview of Pixel-Level Multi-Modal Medical Image Fusion: Modalities, Methods and Recent Trends

Image fusion is a methodology of registering and merging data from manifold sources of the same location taken at different time, from various sensors, or at separate positions. It improves the quality of an image, reduces randomness and redundancy thereby attracting wide variety of application areas. Medical image fusion generates a final fused image which is more descriptive and instructive than the independent source images. It is emerging as a powerful research area as it progresses the clinical accuracy by combining images of different modalities for diagnosis and evaluation of medical problems. Firstly, an extensive survey of image fusion levels and traditional image fusion approaches are presented to give thorough view of the image fusion process. In addition, the merits and demerits of various fusion methods are presented. Then, the existing metrics used to quantify the performance of image fusion results are summarized. Next, the three benchmark datasets are considered for experimental analysis of various image fusion methods and quantitative analysis is done using standard metrics followed by discussion. Eventually, this review paper concludes that although numerous fusion methods outperformed the fusion process but still a lot of challenges are associated in the field of medical imaging. Hence, the technical challenges are highlighted, and the patent paper is concluded giving insight into several future directions in the research field. Hence, efforts are made on summarizing main approaches in image fusion process at pixel level that would produce valuable help to developers and researchers owing to explore this research field.

Analysis of Spatiotemporal Variation Characteristics and Influencing Factors of Grassland Vegetation Coverage in the Qinghai–Tibet Plateau from 2000 to 2023 Based on MODIS Data

Changes in grassland fractional vegetation coverage (FVC) are important indicators of global climate change. Due to the unique characteristics of the Tibetan Plateau ecosystem, variations in grassland coverage are crucial to its ecological stability. This study utilizes the Google Earth Engine (GEE) platform to retrieve long-term MODIS data and analyzes the spatiotemporal distribution of grassland FVC across the Qinghai–Tibet Plateau (QTP) over 24 years (2000–2023). The grassland growth index (GI) is used to evaluate the annual grassland growth at the pixel level. GI is an important indicator for measuring grassland growth status, which can effectively measure the changes in grassland growth in each year relative to the base year. FVC trends are monitored using Sen-Mann-Kendall slope estimation, the coefficient of variation, and the Hurst exponent. Geographic detectors and partial correlation analysis are then applied to explore the contribution rates of key driving factors to FVC. The results show: (1) From 2000 to 2023, FVC exhibited an overall upward trend, with an annual growth rate of 0.0881%. The distribution of FVC on the QTP follows a pattern of higher values in the east and lower values in the west; (2) Over the past 24 years, 54.05% of the total grassland area has shown a significant increase, 23.88% has remained stable, and only a small portion has shown a significant decrease. The overall trend is expected to continue with minimal variability, covering 82.36% of the total grassland area. The overall grassland GI suggests a balanced state of growth; (3) precipitation (Pre) and soil moisture (SM) are the main single factors affecting FVC changes in grasslands on the Tibetan Plateau (q = 0.59 and 0.46). In the interaction detection, in addition to the highest interaction between Pre and other factors, the interaction between SM and other factors also showed a significant impact on the changes in FVC of the QTP grassland; partial correlation analysis of hydrothermal factors and FVC of the QTP grassland. It shows that precipitation has a stronger correlation with QTP grassland FVC changes than temperature. This study has enhanced our understanding of grassland vegetation change and its driving factors on the QTP and quantitatively described the relationship between vegetation change and driving factors, which is of great significance for maintaining the sustainable development of grassland ecosystems.

An innovative methodology for segmenting vessel like structures using artificial intelligence and image processing

Innovation is currently driving enhanced performance and productivity across various fields through process automation. However, identifying intricate details in images can often pose challenges due to morphological variations or specific conditions. Here, artificial intelligence (AI) plays a crucial role by simplifying the segmentation of images. This is achieved by training algorithms to detect specific pixels, thereby recognizing details within images. In this study, an algorithm incorporating modules based on Efficient Sub-Pixel Convolutional Neural Network for image super-resolution, U-Net based Neural baseline for image segmentation, and image binarization for masking was developed. The combination of these modules aimed to identify capillary structures at pixel level. The method was applied on different datasets containing images of eye fundus, citrus leaves, printed circuit boards to test how well it could segment the capillary structures. Notably, the trained model exhibited versatility in recognizing capillary structures across various image types. When tested with the Set 5 and Set 14 datasets, a PSNR of 37.92 and SSIM of 0.9219 was achieved, surpassing significantly other image superresolution methods. The enhancement module processes the image using three different varaiables in the same way, which imposes a complexity of O(n) and takes 308,734 ms to execute; the segmentation module evaluates each pixel against its neighbors to correctly segment regions of interes, generating an quadratic complexity and taking 687,509 ms to execute; the masking module makes several runs through the whole image and in several occasions it calls processes of complexity at 581686 microseconds to execute, which makes it not only the most complex but also the most exhaustive part of the program. This versatility, rooted in its pixel-level operation, enables the algorithm to identify initially unnoticed details, enhancing its applicability across diverse image datasets. This innovation holds significant potential for precisely studying certain structures’ characteristics while enhancing and processing images with high fidelity through AI-driven machine learning algorithms.

Liquid-crystal-based diffractive optical elements with high Pancharatnam-Berry phase accuracy for holographic displays fabricated using an optimized liquid crystal on silicon device

A rapid and accurate photoalignment technique was proposed for the fabrication of liquid crystal Pancharatnam-Berry phase diffractive optical elements (LC PB-DOEs). The in-plane orientation of LCs was precisely manipulated through the polarized illumination of an optimized liquid crystal on silicon (LCOS) device. LCOS and thereafter spatial light modulator (SLM) can generate polarization patterns at pixel level at will. The quality of such alignment was improved significantly by minimizing the phase flicker of the phase-only LCOS SLM. This was confirmed by the increase of the measured quality of the holographic images reconstructed using our DOE in terms of structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) at 30% and 5%, respectively. Furthermore, a bi-focal LC PB-lens was fabricated and used as a high quality Fourier lens in holographic display to validate the usefulness of such LC PB-DOEs. This work illustrated a ubiquitous approach of fabricating different types of lightweight and thin form factor DOEs of random phase patterns at pixel level with low cost and high throughput.

Testing Hubel and Wiesel's "ice-cube" model of functional maps at cellular resolution in macaque V1.

Hubel and Wiesel's ice-cube model proposed that V1 orientation and ocular dominance functional maps intersect orthogonally to optimize wiring efficiency. Here, we revisited this model and additional arrangements at both cellular and pixel levels in awake macaques using two-photon calcium imaging. The recorded response fields of view were similar in size to hypercolumns, each containing up to 2,000 identified neurons and representing full periods of orientation preferences and ocular dominance. We estimated each neuron/pixel's orientation, ocular dominance, and spatial frequency preferences, constructed respective functional maps, computed geometric gradients of feature preferences, and calculated intersection angles among these gradients. At the cellular level, the intersection angles among functional maps were nearly evenly distributed. Nonetheless, pixel-based maps after Gaussian smoothing displayed orientation-ocular dominance and orientation-spatial frequency orthogonality, as well as ocular dominance-spatial frequency parallelism, in alignment with previous results, even though the trends were weak and highly variable. However, these Gaussian smoothing effects were not observed in cellular maps, indicating that the pixel-based trends may not accurately represent the relationships among feature-tuning properties of V1 neurons. We suggest that the widely distributed intersections among cellular maps can ensure that multiple stimulus features are represented within a hypercolumn, and no pair of features is represented with the least economical wiring (e.g. parallel intersections).

Detection and identification of un-uniformed shape text from blurred video frames

<p><span lang="EN-GB">The identification and recognition of text from video frames have received a lot of attention recently, that makes many computer vision-based applications conceivable. In this study, we modify the picture mask and the original identification of the mask region convolution neural network and permit detection in three levels, including holistic, sequence, and at the level of pixels. To identify the texts and determine the text forms, semantics at the pixel and holistic levels can be used. With masking and detection, existences of the character and the word are separated and recognised. In addition, text detection using the results of 2-D feature space instance segmentation is done. Moreover, we explore text recognition using an attention-based optical character recognition (OCR) method with mask</span><span lang="EN-US"> r</span><span lang="EN-GB">egion convolution neural networks (R-CNN) to address and detect the problem of smaller and blurrier texts at the sequential level. Using attribute maps of the word occurrences in sequence to seq, the OCR method calculates the character sequence. At last, a fine-grained learning strategy is proposed to constructs models at word level using the annotated datasets, resulting in the training of a more precise and reliable model. The well-known benchmark datasets ICDAR 2013 and ICDAR 2015 are used to test our suggested methodology.</span></p>

Heterogeneous many-core optimization for Monte Carlo path-tracing on new generation Sunway HPC system

We present swRender, a new parallel rendering pipeline based on the new Sunway many-core architecture (SW26010P) for the Monte Carlo path-tracing algorithm. Previous parallel rendering schemes are unsuitable for our task due to issues such as vast differences in hardware architectures and bottlenecks in I/O communication efficiency. To that end, we create a new two-level parallel tile rendering framework to fully utilize the Sunway computing resources, a practical tile-grouping load-balancing method to maintain the framework’s stability, and a novel many-core acceleration optimization to improve the rendering performance at the pixel level. Our method achieves (1) an average speedup of 16x in multiple benchmarks when compared to the baseline path-tracing model on the Sunway architecture, and (2) an average speedup of 2x when compared to state-of-the-art CPU, co-processor, and GPU-based parallel rendering approaches. Moreover, we scale swRender to run on 15 million cores and obtain high scalable parallel efficiency of 92%.

Anomaly detection in sensor data via encoding time series into images

Detecting anomalies in multivariate time series data is crucial for maintaining the optimal functionality of control system equipment. While existing research has made significant strides in this area, the increasing complexity of industrial environments poses challenges in accurately capturing the interactions between variables. Therefore, this paper introduces an innovative anomaly detection approach that extends one-dimensional time series into two-dimensions to capture the spatial correlations within the data. Unlike traditional approaches, we utilize the Gramian Angular Field to encode the correlations between different sensors at specific time points into images, enabling precise learning of spatial information across multiple variables. Subsequently, we construct an adversarial generative model to accurately identify anomalies at the pixel level, facilitating precise localization of abnormal points. We evaluate our method using five open-source datasets from various fields. Our method outperforms state-of-the-art anomaly detection techniques across all datasets, showcasing its superior performance. Particularly, our method achieves a 11.5% increase in F1 score on the high-dimensional WADI dataset compared to the baseline method. Additionally, we conduct thorough effectiveness analysis, parameter impact experiments, significant statistical analysis, and burden analysis, confirming the efficacy of our approach in capturing both the temporal dynamics and spatial relationships inherent in multivariate time series data.

A novel framework for dynamic and quantitative mapping of damage severity due to compound Drought–Heatwave impacts on tea Plantations, integrating Sentinel-2 and UAV images

In 2022, China experienced a historically rare compound drought–heatwave (CDH) event, which had more severe impacts on vegetation compared with individual extreme events. However, quantitatively mapping the damage severity of CDH on tea tree using satellite data remains a significant challenge. Here we proposed a novel framework for dynamic and quantitative mapping of tea trees damage severity caused by CDH in 2022 using Sentinel-2 and Unmanned Aerial Vehicle (UAV) data. The Extreme Gradient Boosting (XGBoost) was selected as the optimal machine learning algorithm to extract tea plantations using Sentinel-2 data from XGBoost, Random Forest (RF), Logistic regression (LR), and Naive Bayes. The User’s Accuracy and Producer’s Accuracy for the extraction of tea plantations are 92.20 % and 93.51 %, respectively. UAV images with 2.5 cm spatial resolution were utilized to detect the tea trees damaged caused by the CDH in 2022. A new index, named the CDH damage severity index (CDH_DSI), was proposed to quantitatively evaluate the damage severity of CDH on tea trees at pixel level, with a spatial resolution of 10 m x 10 m. Based on the results of tea plantations and damaged tea trees detection, UAV-derived CDH_DSI was calculated and used as ground truth data. Then, The XGBoost was selected as the optimal CDH_DSI prediction model from XGBoost, RF, and LR with the Sentnel-2 derived vegetation indices and spectral reflectance as predictors. The coefficient of determination was 0.81 and root mean squared error was 7.61 %. Finally, dynamic and quantitative CDH_DSI maps were generated with the optimal CDH_DSI prediction model. The results show that 50 percent of tea plantations in Wuyi were damaged by the prolonged CDH event in 2022. These results can be attributed to precipitation deficits and heatwaves. Given that more severe CDH events are projected for the future, quantifying their impacts can provide decision-making support for disaster mitigation and prevention.

Entangled appearance and motion structures network for multi-object tracking and segmentation

The object segmentation mask’s observation sequence shows the trend of changes in the object’s observable geometric form, and predicting them may assist in solving various difficulties in multi-object tracking and segmentation (MOTS). With this aim, we propose the entangled appearance and motion structures network (EAMSN), which can predict the object segmentation mask at the pixel level by integrating VAE and LSTM. Regardless of the surroundings, each EAMSN keeps complete knowledge about the sequence of probable changes in the seen map of the object and its related dynamics. It suggests that EAMSN understands the item meaningfully and is not reliant on instructive examples. As a result, we propose a novel MOTS algorithm. By employing different EAMSNs for each kind of item and training them offline, ambiguities in the segmentation mask discovered for that object may be recovered, and precise estimation of the real boundaries of the object at each step. We analyze our tracker using the KITTI MOTS and MOTS challenges datasets, which comprise car and pedestrian objects, to illustrate the usefulness of the suggested technique. As a result, we developed distinct EAMSNs for cars and pedestrians, trained using the MODELNET40 and Human3.6 M datasets, respectively. The discrepancy between training and testing data demonstrates that EAMSN is not dependent on training data. Finally, we compared our strategy to a variety of other ways. Compared to the published findings, our technique gets the best overall performance.

Real-World Scene Image Enhancement with Contrastive Domain Adaptation Learning

Image enhancement methods leveraging learning-based approaches have demonstrated impressive results when trained on synthetic degraded-clear image pairs. However, when deployed in real-world scenarios, such models often suffer significant performance degradation due to the inherent domain gap between synthetic and real degradations. To bridge this gap, we propose a novel Two-stage Contrastive Domain Adaptation image Enhancement (TCDAE) framework consisting of two key strategies: (1) Synthetic-to-Real Domain Transfer Learning (S2R-DTL) that effectively translates images from the synthetic degraded domain to the real degraded domain, aligning the domains at the pixel level, and (2) Degraded-to-Clear Domain Transfer Learning (D2C-DTL) that further adapts the enhancement model from the synthetic to the real domain by translating images from the real degraded domain to the real clean domain in both supervised and unsupervised branches. A unique aspect of our approach is the integration of a Domain Noise Contrastive Estimation (DoNCE) loss in both learning strategies. This specialized loss formulation enables TCDAE to robustly translate images across domains, even in scenarios lacking strong positive examples. Consequently, our framework can generate enhanced images with natural, realistic appearances akin to real clear images. Comprehensive experiments on real-world degraded scenes across diverse tasks, including dehazing, deraining, and deblurring, demonstrate the superiority of TCDAE over state-of-the-art methods, achieving improved visual quality, quantitative metrics, and downstream task performance.