Segmentation Map Research Articles

SGW-YOLOv8n: An Improved YOLOv8n-Based Model for Apple Detection and Segmentation in Complex Orchard Environments

This study addresses the problem of detecting occluded apples in complex unstructured environments in orchards and proposes an apple detection and segmentation model based on improved YOLOv8n-SGW-YOLOv8n. The model improves apple detection and segmentation by combining the SPD-Conv convolution module, the GAM global attention mechanism, and the Wise-IoU loss function, which enhances the accuracy and robustness. The SPD-Conv module preserves fine-grained features in the image by converting spatial information into channel information, which is particularly suitable for small target detection. The GAM global attention mechanism enhances the recognition of occluded targets by strengthening the feature representation of channel and spatial dimensions. The Wise-IoU loss function further optimises the regression accuracy of the target frame. Finally, the pre-prepared dataset is used for model training and validation. The results show that the SGW-YOLOv8n model significantly improves relative to the original YOLOv8n in target detection and instance segmentation tasks, especially in occlusion scenes. The model improves the detection mAP to 75.9% and the segmentation mAP to 75.7% and maintains a processing speed of 44.37 FPS, which can meet the real-time requirements, providing effective technical support for the detection and segmentation of fruits in complex unstructured environments for fruit harvesting robots.

TCDDU-Net: combining transformer and convolutional dual-path decoding U-Net for retinal vessel segmentation

Accurate segmentation of retinal blood vessels is crucial for enhancing diagnostic efficiency and preventing disease progression. However, the small size and complex structure of retinal blood vessels, coupled with low contrast in corresponding fundus images, pose significant challenges for this task. We propose a novel approach for retinal vessel segmentation, which combines the transformer and convolutional dual-path decoding U-Net (TCDDU-Net). We propose the selective dense connection swin transformer block, which converts the input feature map into patches, introduces MLPs to generate probabilities, and performs selective fusion at different stages. This structure forms a dense connection framework, enabling the capture of long-distance dependencies and effective fusion of features across different stages. The subsequent stage involves the design of the background decoder, which utilizes deformable convolution to learn the background information of retinal vessels by treating them as segmentation objects. This is then combined with the foreground decoder to form a dual-path decoding U-Net. Finally, the foreground segmentation results and the processed background segmentation results are fused to obtain the final retinal vessel segmentation map. To evaluate the effectiveness of our method, we performed experiments on the DRIVE, STARE, and CHASE datasets for retinal vessel segmentation. Experimental results show that the segmentation accuracies of our algorithms are 96.98, 97.40, and 97.23, and the AUC metrics are 98.68, 98.56, and 98.50, respectively.In addition, we evaluated our methods using F1 score, specificity, and sensitivity metrics. Through a comparative analysis, we found that our proposed TCDDU-Net method effectively improves retinal vessel segmentation performance and achieves impressive results on multiple datasets compared to existing methods.

Accuracy of endocardial abnormal electrogram characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation for ventricular tachycardia ablation

Abstract Background Substrate-based catheter ablation of ventricular tachycardia (VT) has become the mainstay therapy for patients with scar-related VT. Correct identification and timing of abnormal near-field EGMs and discerning it from far-field EGMs, is key for procedural success. The novel CARTO™ Late Activation Mapping (LAM) module provides fast and accurate annotation of near-field EGMs including late abnormal ventricular activation. For the algorithm to succeed, a late boundary (LB) annotation line should be positioned immediately behind the normal near-field EGM, however, since the timing of normal local near-field activation varies greatly throughout the endocardium, there is currently no consensus for the optimal LB positioning. We hypothesized that 17-AHA segmental annotation of the endocardial LB followed by scar-dependent sensitivity settings, will improve the accuracy of the near-field EGM characterization. Purpose To evaluate the accuracy of endocardial abnormal EGM characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation as compared to conventional mid-QRS LB positioning. Methods Five high-resolution substrate voltage maps were obtained from 5 patients (80% male, 68±7 years old, 80% ischemic cardiomyopathy) that underwent a catheter-based ablation for scar-related VT. A personalized CT-derived anatomical reconstruction of the 17-segment AHA model was co-registered offline with the electroanatomic voltage map, using ADAS3D software. Per segment, individual LB timings were set at the highest sensitivity. A gold standard annotation map was created by two independent experts after manual scrutiny of individual EGMs. Next, the expert map was compared to automatic maps created from segmental LB and mid-QRS LB positions at normal, high, highest or mixed (normal/high or normal/highest) sensitivities, focusing on EGM voltage and activation accuracy. Results A mean of 2041±1028 points per map were analyzed. Correct correlation of EGM annotation was found on average in 91% (activation) and 81% (voltage) for all segmental LB positioned maps (segLB), and in 83% (activation) and 77% (voltage) for all mid-QRS LB positioned maps (mid-QRSLB). The highest correlation with expert map for both activation and voltage annotation was found in the mixed sensitivity normal/highest map with segmental LB positioning (activation 98%, voltage 88%). The correlation on activation was significantly superior in the segLB map when compared to the normal sensitivity maps (p=0.0008 for mid-QRSLB and p=0.02 for segLB). For correlation on voltage, the segLBmap with mixed sensitivity was significantly superior compared to three mid-QRSLB maps at normal (p=0.04), high and highest (p=0.01) sensitivity. Conclusion Individualized, 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation provides a possible accurate and clinically-relevant framework for substrate assessment. Prospective validation is needed.

GRID-FAST: A Grid-based Intersection Detection for Fast Semantic Topometric Mapping

This article introduces a novel approach to constructing a topometric map that allows for efficient navigation and decision-making in mobile robotics applications. The method generates the topometric map from a 2D grid-based map. The topometric map segments areas of the input map into different structural-semantic classes: intersections, pathways, dead ends, and pathways leading to unexplored areas. This method is grounded in a new technique for intersection detection that identifies the area and the openings of intersections in a semantically meaningful way. The framework introduces two levels of pre-filtering with minimal computational cost to eliminate small openings and objects from the map which are unimportant in the context of high-level map segmentation and decision making. The topological map generated by GRID-FAST enables fast navigation in large-scale environments, and the structural semantics can aid in mission planning, autonomous exploration, and human-to-robot cooperation. The efficacy of the proposed method is demonstrated through validation on real maps gathered from robotic experiments: 1) a structured indoor environment, 2) an unstructured cave-like subterranean environment, and 3) a large-scale outdoor environment, which comprises pathways, buildings, and scattered objects. Additionally, the proposed framework has been compared with state-of-the-art topological mapping solutions and is able to produce a topometric and topological map with up to 92% fewer nodes than the next best solution. The method proposed in this article has been implemented in the robotics framework ROS and is open-sourced. The code is available at: https://github.com/LTU-RAI/GRID-FAST.

Integrated multicenter deep learning system for prognostic prediction in bladder cancer

Precise survival risk stratification is crucial for personalized therapy in bladder cancer (BCa). This study developed and validated an end-to-end deep learning system using histological slides to predict overall survival (OS) risk in BCa patients. We employed the BlaPaSeg tile classifier to generate tissue probability heatmaps and segmentation maps, trained two prognostic networks, MacroVisionNet and UniVisionNet, and explored six potential BCa prognostic biomarkers. Across all cohorts, the AUC for BlaPaSeg ranged from 0.9906 to 0.9945, while the C-index varied from 0.655 to 0.834 for MacroVisionNet and 0.661 to 0.853 for UniVisionNet. After covariate adjustment, the hazard ratio (HR) values for high-risk groups were 1.97 to 5.06 in MacroVisionNet and 2.13 to 4.01 in UniVisionNet. The high-risk Coloc (Tumor Co-localization score) and IMTS (Integrated Muscle Tumor Score) groups illustrated a higher death risk with HR values from 1.41 to 10.16. The system improves BCa survival prediction and supports refined patient management.

Dual Energy CT for Deep Learning-Based Segmentation and Volumetric Estimation of Early Ischemic Infarcts.

Ischemic changes are not visible on non-contrast head CT until several hours after infarction, though deep convolutional neural networks have shown promise in the detection of subtle imaging findings. This study aims to assess if dual-energy CT (DECT) acquisition can improve early infarct visibility for machine learning. The retrospective dataset consisted of 330 DECTs acquired up to 48h prior to confirmation of a DWI positive infarct on MRI between 2016 and 2022. Infarct segmentation maps were generated from the MRI and co-registered to the CT to serve as ground truth for segmentation. A self-configuring 3D nnU-Net was trained for segmentation on (1) standard 120kV mixed-images (2) 190keV virtual monochromatic images and (3) 120kV + 190keV images as dual channel inputs. Algorithm performance was assessed with Dice scores with paired t-tests on a test set. Global aggregate Dice scores were 0.616, 0.645, and 0.665 for standard 120kV images, 190keV, and combined channel inputs respectively. Differences in overall Dice scores were statistically significant with highest performance for combined channel inputs (p < 0.01). Small but statistically significant differences were observed for infarcts between 6 and 12h from last-known-well with higher performance for larger infarcts. Volumetric accuracy trended higher with combined inputs but differences were not statistically significant (p = 0.07). Supplementation of standard head CT images with dual-energy data provides earlier and more accurate segmentation of infarcts for machine learning particularly between 6 and 12h after last-known-well.

Retinal fundus image super-resolution based on generative adversarial network guided with vascular structure prior

Many ophthalmic and systemic diseases can be screened by analyzing retinal fundus images. The clarity and resolution of retinal fundus images directly determine the effectiveness of clinical diagnosis. Deep learning methods based on generative adversarial networks are used in various research fields due to their powerful generative capabilities, especially image super-resolution. Although Real-ESRGAN is a recently proposed method that excels in processing real-world degraded images, it suffers from structural distortions when super-resolving retinal fundus images are rich in structural information. To address this shortcoming, we first process the input image using a pre-trained U-Net model to obtain a structural segmentation map of the retinal vessels and use the segmentation map as the structural prior. The spatial feature transform layer is then used to better integrate the structural prior into the generation process of the generator. In addition, we introduce channel and spatial attention modules into the skip connections of the discriminator to emphasize meaningful features and accordingly enhance the discriminative power of the discriminator. Based on the original loss functions, we introduce the L1 loss function to measure the pixel-level differences between the segmentation maps of retinal vascular structures in the high-resolution images and the super-resolution images to further constrain the super-resolution images. Simulation results on retinal image datasets show that our improved algorithm results have a better visual performance by suppressing structural distortions in the super-resolution images.

Reinforcement learning-based anatomical maps for pancreas subregion and duct segmentation.

The pancreas is a complex abdominal organ with many anatomical variations, and therefore automated pancreas segmentation from medical images is a challengingapplication. In this paper, we present a framework for segmenting individual pancreatic subregions and the pancreatic duct from three-dimensional (3D) computed tomography (CT) images. A multiagent reinforcement learning (RL) network was used to detect landmarks of the head, neck, body, and tail of the pancreas, and landmarks along the pancreatic duct in a selected target CT image. Using the landmark detection results, an atlas of pancreases was nonrigidly registered to the target image, resulting in anatomical probability maps for the pancreatic subregions and duct. The probability maps were augmented with multilabel 3D U-Net architectures to obtain the final segmentationresults. To evaluate the performance of our proposed framework, we computed the Dice similarity coefficient (DSC) between the predicted and ground truth manual segmentations on a database of 82 CT images with manually segmented pancreatic subregions and 37 CT images with manually segmented pancreatic ducts. For the four pancreatic subregions, the mean DSC improved from 0.38, 0.44, and 0.39 with standard 3D U-Net, Attention U-Net, and shifted windowing (Swin) U-Net architectures, to 0.51, 0.47, and 0.49, respectively, when utilizing the proposed RL-based framework. For the pancreatic duct, the RL-based framework achieved a mean DSC of 0.70, significantly outperforming the standard approaches and existing methods on different datasets. The resulting accuracy of the proposed RL-based segmentation framework demonstrates an improvement against segmentation with standard U-Netarchitectures.

Novel Concepts of Bipolar Picture Fuzzy Graphs With Applications in Image Segmentation and Road Maps Designs

Novel Concepts of Bipolar Picture Fuzzy Graphs With Applications in Image Segmentation and Road Maps Designs

Attention-guided hierarchical fusion U-Net for uncertainty-driven medical image segmentation

Small inaccuracies in the system components or artificial intelligence (AI) models for medical imaging could have significant consequences leading to life hazards. To mitigate those risks, one must consider the precision of the image analysis outcomes (e.g., image segmentation), along with the confidence in the underlying model predictions. U-shaped architectures, based on the convolutional encoder–decoder, have established themselves as a critical component of many AI-enabled diagnostic imaging systems. However, most of the existing methods focus on producing accurate diagnostic predictions without assessing the uncertainty associated with such predictions or the introduced techniques. Uncertainty maps highlight areas in the predicted segmented results, where the model is uncertain or less confident. This could lead radiologists to pay more attention to ensuring patient safety and pave the way for trustworthy AI applications. In this paper, we therefore propose the Attention-guided Hierarchical Fusion U-Net (named AHF-U-Net) for medical image segmentation. We then introduce the uncertainty-aware version of it called UA-AHF-U-Net which provides the uncertainty map alongside the predicted segmentation map. The network is designed by integrating the Encoder Attention Fusion module (EAF) and the Decoder Attention Fusion module (DAF) on the encoder and decoder sides of the U-Net architecture, respectively. The EAF and DAF modules utilize spatial and channel attention to capture relevant spatial information and indicate which channels are appropriate for a given image. Furthermore, an enhanced skip connection is introduced and named the Hierarchical Attention-Enhanced (HAE) skip connection. We evaluated the efficiency of our model by comparing it with eleven well-established methods for three popular medical image segmentation datasets consisting of coarse-grained images with unclear boundaries. Based on the quantitative and qualitative results, the proposed method ranks first in two datasets and second in a third. The code can be accessed at: https://github.com/AfsanaAhmedMunia/AHF-Fusion-U-Net

The Study of Echocardiography of Left Ventricle Segmentation Combining Transformer and Convolutional Neural Networks.

Accurate prediction of echocardiographic parameters is essential for diagnosis and treatment of cardiac disease, especially for segmentation of the left ventricle to obtain measurements such as left ventricular ejection fraction and volume. However, manually outlining left ventricle on echocardiographic images is a time-consuming and physician experience-dependent task. Therefore, it is crucial to develop an accurate and efficient automatic segmentation tool. Therefore, we aimed to explore a model to perform echocardiography of left ventricle segmentation by combining transformer and convolutional neural networks (CNN).ResNet-50 was used in CNN branch. The encoder-decoder architecture was used for transformer branch, which was fused to the corresponding feature maps of the CNN branches. Fusion module was used to effectively combine feature information from the CNN and transformer. Bridge attention used to increase sensitivity and prediction accuracy of model. The entire network was trained end-to-end using the binary cross-entropy with logits loss L.In this work, we propose an automatic left ventricular (LV) segmentation model based on Transformer and CNN that efficiently captures global dependencies and spatial details and create a fusion module using CBAM that fuses Transformer and CNN features. In addition, attention is also computed using multi-level fusion features to obtain the final attention segmentation map. The model was trained and evaluated on a large cardiac image dataset, EchoNet-Dynamic, with test dice coefficient of 92.4%.The results show that our model can better segment left ventricle. We also tested our model on clinical patient ultrasound images, and visualization results proved effectiveness of the model.

Картографирование сегментов экономической экосистемы территории

Relevance. The rapid introduction of digital technologies into economic activity has marked a new paradigm of economic analysis – the analysis of economic ecosystems. The relevance of the structural study of the economic ecosystems of territories is due to both the weak elaboration of this topic in the world and domestic scientific literature, and the practical need for the formation of scientifically based strategies for the socio-economic development of territories. Goal. The purpose of this study is to develop a methodology for mapping the economic ecosystem of the territory. Materials and methods. The object of this study is the economic ecosystem of the territory. The subject of the study is economic relations for the development of the economic ecosystem of the territory on the basis of mapping its elements. The information base of the research was scientific publications indexed in the Scopus and RSCI databases, as well as the author's results on public management of the development of economic ecosystems. The method of logical system analysis is used as a research method. Results. As a result of the conducted research, a scheme of a map of the industry and sphere of activity of the economic ecosystem is proposed, including indicators, participants, products of activity, development trends, assessment experts and conclusions on the development of this sphere of economic activity. An algorithm for mapping the economic ecosystem of the territory has been developed, which includes the stages of segmentation, the formation of maps of industries, the assessment of development projects and the formation of a strategy for the socio-economic development of the territory. Using the example of the development of the Tyumen region, the applicability of mapping segments of the economic ecosystem of the territory for public management of the development of various fields of activity is shown. The novelty of this study lies in the development of the author's approach to mapping segments of the economic eco-system of the territory, which develops the theory of modeling the structure and development of elements of economic ecosystems.

Using computer vision to classify, locate and segment fire behavior in UAS-captured images

The widely adaptable capabilities of artificial intelligence, in particular deep learning and computer vision have led to significant research output regarding flame and smoke detection. The composition of flame and smoke, also described as fire behavior, can be considerably different depending on factors like weather, fuels, and the specific landscape fire is being observed on. The ability to detect definable classes of fire behavior using computer vision has not been explored and could be helpful given it often dictates how firefighters respond to fire situations. To test whether types of fire behavior could be reliably classified, we collected and labeled a unique unmanned aerial system (UAS) image dataset of fire behavior classifications to be trained and validated using You Only Look Once (YOLO) detection models. Our 960 labeled images were sourced from over 21 h of UAS video collected during prescribed fire operations covering a large region of Texas and Louisiana, United States. National Wildfire Coordinating Group (NWCG) fire behavior observations and descriptions served as a reference for determining fire behavior classes during labeling. YOLOv8 models were trained on NWCG Rank 1–3 fire behavior descriptions in grassland, shrubland, forested, and combined fire regimes within our study area. Models were first trained and validated on classifying isolated image objects of fire behavior, and then separately trained to locate and segment fire behavior classifications in UAS images. Models trained to classify isolated image objects of fire behavior consistently performed at a mAP of 0.808 or higher, with combined fire regimes producing the best results (mAP = 0.897). Most segmentation models performed relatively poorly, except for the forest regime model at a box (locate) and mask (segment) mAP of 0.59 and 0.611, respectively. Our results indicate that classifying fire behavior with computer vision is possible in different fire regimes and fuel models, whereas locating and segmenting fire behavior types around background information is relatively difficult. However, it may be a manageable task with enough data, and when models are developed for a specific fire regime. With an increasing number of destructive wildfires and new challenges confronting fire managers, identifying how new technologies can quickly assess wildfire situations can assist wildfire responder awareness. Our conclusion is that levels of abstraction deeper than just detection of smoke or flame are possible using computer vision and could make even more detailed aerial fire monitoring possible using a UAS.

Skin Lesion Segmentation through Generative Adversarial Networks with Global and Local Semantic Feature Awareness

The accurate segmentation of skin lesions plays an important role in the diagnosis and treatment of skin cancers. However, skin lesion areas are rich in details and local features, including the appearance, size, shape, texture, etc., which pose challenges for the accurate localization and segmentation of the target area. Unfortunately, the consecutive pooling and stride convolutional operations in existing convolutional neural network (CNN)-based solutions lead to the loss of some spatial information and thus constrain the accuracy of lesion region segmentation. In addition, using only the traditional loss function in CNN cannot ensure that the model is adequately trained. In this study, a generative adversarial network is proposed, with global and local semantic feature awareness (GLSFA-GAN) for skin lesion segmentation based on adversarial training. Specifically, in the generator, a multi-scale localized feature fusion module and an effective channel-attention module are designed to acquire the multi-scale local detailed information of the skin lesion area. In addition, a global context extraction module in the bottleneck between the encoder and decoder of the generator is used to capture more global semantic features and spatial information about the lesion. After that, we use an adversarial training strategy to make the discriminator discern the generated labels and the segmentation prediction maps, which assists the generator in yielding more accurate segmentation maps. Our proposed model was trained and validated on three public skin lesion challenge datasets involving the ISIC2017, ISIC2018, and HAM10000, and the experimental results confirm that our proposed method provides a superior segmentation performance and outperforms several comparative methods.

Subjective performance assessment protocol for visual explanations-based face verification explainability

The integration of Face Verification (FV) systems into multiple critical moments of daily life has become increasingly prevalent, raising concerns regarding the transparency and reliability of these systems. Consequently, there is a growing need for FV explainability tools to provide insights into the behavior of these systems. FV explainability tools that generate visual explanations, e.g., saliency maps, heatmaps, contour-based visualization maps, and face segmentation maps, show promise in enhancing FV transparency by highlighting the contributions of different face regions to the FV decision-making process. However, evaluating the performance of such explainability tools remains challenging due to the lack of standardized assessment metrics and protocols. In this context, this paper proposes a subjective performance assessment protocol for evaluating the explainability performance of visual explanation-based FV explainability tools through pairwise comparisons of their explanation outputs. The proposed protocol encompasses a set of key specifications designed to efficiently collect the subjects’ preferences and estimate explainability performance scores, facilitating the relative assessment of the explainability tools. This protocol aims to address the current gap in evaluating the effectiveness of visual explanation-based FV explainability tools, providing a structured approach for assessing their performance and comparing with alternative tools. The proposed protocol is exercised and validated through an experiment conducted using two distinct heatmap-based FV explainability tools, notably FV-RISE and CorrRISE, taken as examples of visual explanation-based explainability tools, considering the various types of FV decisions, i.e., True Acceptance (TA), False Acceptance (FA), True Rejection (TR), and False Rejection (FR). A group of subjects with variety in age, gender, and ethnicity was tasked to express their preferences regarding the heatmap-based explanations generated by the two selected explainability tools. The subject preferences were collected and statistically processed to derive quantifiable scores, expressing the relative explainability performance of the assessed tools. The experimental results revealed that both assessed explainability tools exhibit comparable explainability performance for FA, TR, and FR decisions with CorrRISE performing slightly better than FV-RISE for TA decisions.

Semi-supervised ViT knowledge distillation network with style transfer normalization for colorectal liver metastases survival prediction

Colorectal liver metastases (CLM) affect almost half of all colon cancer patients and the response to systemic chemotherapy plays a crucial role in patient survival. While oncologists typically use tumor grading scores, such as tumor regression grade (TRG), to establish an accurate prognosis on patient outcomes, including overall survival (OS) and time-to-recurrence (TTR), these traditional methods have several limitations. They are subjective, time-consuming, and require extensive expertise, which limits their scalability and reliability. Additionally, existing approaches for prognosis prediction using machine learning mostly rely on radiological imaging data, but recently histological images have been shown to be relevant for survival predictions by allowing to fully capture the complex microenvironmental and cellular characteristics of the tumor. To address these limitations, we propose an end-to-end approach for automated prognosis prediction using histology slides stained with Hematoxylin and Eosin (H&E) and Hematoxylin Phloxine Saffron (HPS). We first employ a Generative Adversarial Network (GAN) for slide normalization to reduce staining variations and improve the overall quality of the images that are used as input to our prediction pipeline. We propose a semi-supervised model to perform tissue classification from sparse annotations, producing segmentation and feature maps. Specifically, we use an attention-based approach that weighs the importance of different slide regions in producing the final classification results. Finally, we exploit the extracted features for the metastatic nodules and surrounding tissue to train a prognosis model. In parallel, we train a vision Transformer model in a knowledge distillation framework to replicate and enhance the performance of the prognosis prediction. We evaluate our approach on an in-house clinical dataset of 258 CLM patients, achieving superior performance compared to other comparative models with a c-index of 0.804 (0.014) for OS and 0.735 (0.016) for TTR, as well as on two public datasets. The proposed approach achieves an accuracy of 86.9% to 90.3% in predicting TRG dichotomization. For the 3-class TRG classification task, the proposed approach yields an accuracy of 78.5% to 82.1%, outperforming the comparative methods. Our proposed pipeline can provide automated prognosis for pathologists and oncologists, and can greatly promote precision medicine progress in managing CLM patients.

Scalable video transformer for full-frame video prediction

Vision Transformers (ViTs) have shown success in many low-level computer vision tasks. However, existing ViT models are limited by their high computation and memory cost when generating high-resolution videos for tasks like video prediction. This paper presents a scalable video transformer for full-frame video predication. Specifically, we design a backbone transformer block for our video transformer. This transformer block decouples the temporal and channel features to reduce the computation cost when processing large-scale spatial–temporal video features. We use transposed attention to focus on the channel dimension instead of the spatial window to further reduce the computation cost. We also design a Global Shifted Multi-Dconv Head Transposed Attention module (GSMDTA) for our transformer block. This module is built upon two key ideas. First, we design a depth shift module to better incorporate the cross-channel or temporal information from video features. Second, we introduce a global query mechanism to capture global information to handle large motion for video prediction. This new transformer block enables our video transformer to predict a full frame from multiple past frames at the resolution of 1024 × 512 with 12 GB VRAM. Experiments on various video prediction benchmarks demonstrate that our method with only RGB input outperforms state-of-the-art methods that require additional data, like segmentation maps and optical flows. Our method exceeds the state-of-the-art RGB-only methods by a large margin (1.2 dB) in PSNR. Our method is also faster than state-of-the-art video prediction transformers.

A Novel End-to-End Deep Learning Framework for Chip Packaging Defect Detection.

As semiconductor chip manufacturing technology advances, chip structures are becoming more complex, leading to an increased likelihood of void defects in the solder layer during packaging. However, identifying void defects in packaged chips remains a significant challenge due to the complex chip background, varying defect sizes and shapes, and blurred boundaries between voids and their surroundings. To address these challenges, we present a deep-learning-based framework for void defect segmentation in chip packaging. The framework consists of two main components: a solder region extraction method and a void defect segmentation network. The solder region extraction method includes a lightweight segmentation network and a rotation correction algorithm that eliminates background noise and accurately captures the solder region of the chip. The void defect segmentation network is designed for efficient and accurate defect segmentation. To cope with the variability of void defect shapes and sizes, we propose a Mamba model-based encoder that uses a visual state space module for multi-scale information extraction. In addition, we propose an interactive dual-stream decoder that uses a feature correlation cross gate module to fuse the streams' features to improve their correlation and produce more accurate void defect segmentation maps. The effectiveness of the framework is evaluated through quantitative and qualitative experiments on our custom X-ray chip dataset. Furthermore, the proposed void defect segmentation framework for chip packaging has been applied to a real factory inspection line, achieving an accuracy of 93.3% in chip qualification.

Vehicle Ego-Trajectory Segmentation Using Guidance Cues

Computer vision has significantly influenced recent advancements in autonomous driving by providing cutting-edge solutions for various challenges, including object detection, semantic segmentation, and comprehensive scene understanding. One specific challenge is ego-vehicle trajectory segmentation, which involves learning the vehicle’s path and describing it with a segmentation map. This can play an important role in both autonomous driving and advanced driver assistance systems, as it enhances the accuracy of perceiving and forecasting the vehicle’s movements across different driving scenarios. In this work, we propose a deep learning approach for ego-trajectory segmentation that leverages a state-of-the-art segmentation network augmented with guidance cues provided through various merging mechanisms. These mechanisms are designed to direct the vehicle’s path as intended, utilizing training data obtained with a self-supervised approach. Our results demonstrate the feasibility of using self-supervised labels for ego-trajectory segmentation and embedding directional intentions within the network’s decisions through image and guidance input concatenation, feature concatenation, or cross-attention between pixel features and various types of guidance cues. We also analyze the effectiveness of our approach in constraining the segmentation outputs and prove that our proposed improvements bring major boosts in the segmentation metrics, increasing IoU by more than 12% and 5% compared with our two baseline models. This work paves the way for further exploration into ego-trajectory segmentation methods aimed at better predicting the behavior of autonomous vehicles.

Segmentation of supragranular and infragranular layers in ultra-high-resolution 7T ex vivo MRI of the human cerebral cortex.

Accurate labeling of specific layers in the human cerebral cortex is crucial for advancing our understanding of neurodevelopmental and neurodegenerative disorders. Building on recent advancements in ultra-high-resolution ex vivo MRI, we present a novel semi-supervised segmentation model capable of identifying supragranular and infragranular layers in ex vivo MRI with unprecedented precision. On a dataset consisting of 17 whole-hemisphere ex vivo scans at 120 $\mu $m, we propose a Multi-resolution U-Nets framework that integrates global and local structural information, achieving reliable segmentation maps of the entire hemisphere, with Dice scores over 0.8 for supra- and infragranular layers. This enables surface modeling, atlas construction, anomaly detection in disease states, and cross-modality validation while also paving the way for finer layer segmentation. Our approach offers a powerful tool for comprehensive neuroanatomical investigations and holds promise for advancing our mechanistic understanding of progression of neurodegenerative diseases.