Encoder-decoder Architecture Research Articles

Interactive Surgical Training in Neuroendoscopy: Real-Time Anatomical Feature Localization Using Natural Language Expressions.

This study addresses challenges in surgical education, particularly in neuroendoscopy, where the demand for optimized workflow conflicts with the need for trainees' active participation in surgeries. To overcome these challenges, we propose a framework that accurately identifies anatomical structures within images guided by language descriptions, facilitating authentic and interactive learning experiences in neuroendoscopy. Utilizing the encoder-decoder architecture of a conventional transformer, our framework processes multimodal inputs (images and language descriptions) to identify and localize features in neuroendoscopic images. We curate a dataset from recorded endoscopic third ventriculostomy (ETV) procedures for training and evaluation. Utilizing evaluation metrics, including "R@n," "IoU= θ," "mIoU," and top-1 accuracy, we systematically benchmark our framework against state-of-the-art methodologies. The framework demonstrates excellent generalization, surpassing the compared methods with 93.67 % accuracy and 76.08 % mIoU on unseen data. It also exhibits better computational speed compared with other methods. Qualitative results affirms the framework's effectiveness in precise localization of referred anatomical features within neuroendoscopic images. The framework's adeptness at localizing anatomical features using language descriptions positions it as a valuable tool for integration into future interactive clinical learning systems, enhancing surgical training in neuroendoscopy. The exemplary performance reinforces the framework's potential in enhancing surgical education, leading to improved skills and outcomes for trainees in neuroendoscopy.

The Study of Echocardiography of Left Ventricle Segmentation Combining Transformer and Convolutional Neural Networks.

Accurate prediction of echocardiographic parameters is essential for diagnosis and treatment of cardiac disease, especially for segmentation of the left ventricle to obtain measurements such as left ventricular ejection fraction and volume. However, manually outlining left ventricle on echocardiographic images is a time-consuming and physician experience-dependent task. Therefore, it is crucial to develop an accurate and efficient automatic segmentation tool. Therefore, we aimed to explore a model to perform echocardiography of left ventricle segmentation by combining transformer and convolutional neural networks (CNN).ResNet-50 was used in CNN branch. The encoder-decoder architecture was used for transformer branch, which was fused to the corresponding feature maps of the CNN branches. Fusion module was used to effectively combine feature information from the CNN and transformer. Bridge attention used to increase sensitivity and prediction accuracy of model. The entire network was trained end-to-end using the binary cross-entropy with logits loss L.In this work, we propose an automatic left ventricular (LV) segmentation model based on Transformer and CNN that efficiently captures global dependencies and spatial details and create a fusion module using CBAM that fuses Transformer and CNN features. In addition, attention is also computed using multi-level fusion features to obtain the final attention segmentation map. The model was trained and evaluated on a large cardiac image dataset, EchoNet-Dynamic, with test dice coefficient of 92.4%.The results show that our model can better segment left ventricle. We also tested our model on clinical patient ultrasound images, and visualization results proved effectiveness of the model.

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis.

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

NA-segformer: A multi-level transformer model based on neighborhood attention for colonoscopic polyp segmentation

In various countries worldwide, the incidence of colon cancer-related deaths has been on the rise in recent years. Early detection of symptoms and identification of intestinal polyps are crucial for improving the cure rate of colon cancer patients. Automated computer-aided diagnosis (CAD) has emerged as a solution to the low efficiency of traditional methods relying on manual diagnosis by physicians. Deep learning is the latest direction of CAD development and has shown promise for colonoscopic polyp segmentation. In this paper, we present a multi-level encoder-decoder architecture for polyp segmentation based on the Transformer architecture, termed NA-SegFormer. To improve the performance of existing Transformer-based segmentation algorithms for edge segmentation on colon polyps, we propose a patch merging module with a neighbor attention mechanism based on overlap patch merging. Since colon tract polyps vary greatly in size and different datasets have different sample sizes, we used a unified focal loss to solve the problem of category imbalance in colon tract polyp data. To assess the effectiveness of our proposed method, we utilized video capsule endoscopy and typical colonoscopy polyp datasets, as well as a dataset containing surgical equipment. On the datasets Kvasir-SEG, Kvasir-Instrument and KvasirCapsule-SEG, the Dice score of our proposed model reached 94.30%, 94.59% and 82.73%, with an accuracy of 98.26%, 99.02% and 81.84% respectively. The proposed method achieved inference speed with an Frame-per-second (FPS) of 125.01. The results demonstrated that our suggested model effectively segmented polyps better than several well-known and latest models. In addition, the proposed method has advantages in trade-off between inference speed and accuracy, and it will be of great significance to real-time colonoscopic polyp segmentation. The code is available at https://github.com/promisedong/NAFormer.

Accurate crowd counting for intelligent video surveillance systems

The paper presents a novel deep learning approach for crowd counting in intelligent video surveillance systems, addressing the growing need for accurate monitoring of public spaces in urban environments. The demand for precise crowd estimation arises from challenges related to security, public safety, and efficiency in urban areas, particularly during large public events. Existing crowd counting techniques, including feature-based object detection and regression-based methods, face limitations in high-density environments due to occlusions, lighting variations, and diverse human figures. To overcome these challenges, the authors propose a new deep encoder-decoder architecture based on VGG16, which incorporates hierarchical feature extraction with spatial and channel attention mechanisms. This architecture enhances the model’s ability to manage variations in crowd density, leveraging adaptive pooling and dilated convolutions to extract meaningful features from dense crowds. The model’s decoder is further refined to handle sparse and crowded scenes through separate density maps, improving its adaptability and accuracy. Evaluations of the proposed model on benchmark datasets, including Shanghai Tech and UCF CC 50, demonstrate superior performance over state-of-the-art methods, with significant improvements in mean absolute error and mean squared error metrics. The paper emphasizes the importance of addressing environmental variability and scale differences in crowded environments and shows that the proposed model is effective in both sparse and dense crowd conditions. This research contributes to the advancement of intelligent video surveillance systems by providing a more accurate and efficient method for crowd counting, with potential applications in public safety, transportation management, and urban planning.

Samba: Semantic segmentation of remotely sensed images with state space model

High-resolution remotely sensed images pose challenges to traditional semantic segmentation networks, such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). CNN-based methods struggle to handle high-resolution images due to their limited receptive field, while ViT-based methods, despite having a global receptive field, face challenges when processing long sequences. Inspired by the Mamba network, which is based on a state space model (SSM) to efficiently capture global semantic information, we propose a semantic segmentation framework for high-resolution remotely sensed imagery, named Samba. Samba utilizes an encoder-decoder architecture, with multiple Samba blocks serving as the encoder to efficiently extract multi-level semantic information, and UperNet functioning as the decoder. We evaluate Samba on the LoveDA, ISPRS Vaihingen, and ISPRS Potsdam datasets using the mIoU and mF1 metrics, and compare it with top-performing CNN-based and ViT-based methods. The results demonstrate that Samba achieves unparalleled performance on commonly used remotely sensed datasets for semantic segmentation. Samba is the first to demonstrate the effectiveness of SSM in segmenting remotely sensed imagery, setting a new performance benchmark for Mamba-based techniques in this domain of semantic segmentation. The source code and baseline implementations are available at https://github.com/zhuqinfeng1999/Samba.

Exploring high-order correlation for hyperspectral image denoising with hypergraph convolutional network

High-order correlation is an important property of hyperspectral images (HSIs) and has been widely investigated in model-based HSI denoising. However, the existing deep learning-based HSI denoising approaches have not fully utilized the high-order correlation. Hypergraph convolutional networks have shown great potential in capturing the high-order correlation. Therefore, in this paper, we propose a novel HSI denoising method by employing hypergraph convolution to characterize the high-order correlation at the patch level. Specifically, our framework is a symmetrically skip-connected 3D encoder–decoder architecture, which enhances the extraction and utilization of local features. Furthermore, to integrate competently the hypergraph convolutional modules into the 3D framework, we devise a dimensional transformation module that facilitates the fusion of 3D convolution and hypergraph convolution. Notably, in the hypergraph convolution operation, we use a data-driven technique to acquire the incidence matrix of a hypergraph, efficiently constructing the HSI into a high-order structure. Our proposed method excels in HSI denoising performance compared to state-of-the-art approaches, evidenced by extensive experiments on synthetic and real-world noisy HSIs.

HEDN: multi-oriented hierarchical extraction and dual-frequency decoupling network for 3D medical image segmentation.

Previous 3D encoder-decoder segmentation architectures struggled with fine-grained feature decomposition, resulting in unclear feature hierarchies when fused across layers. Furthermore, the blurred nature of contour boundaries in medical imaging limits the focus on high-frequency contour features. To address these challenges, we propose a Multi-oriented Hierarchical Extraction and Dual-frequency Decoupling Network (HEDN), which consists of three modules: Encoder-Decoder Module (E-DM), Multi-oriented Hierarchical Extraction Module (Multi-HEM), and Dual-frequency Decoupling Module (Dual-DM). The E-DM performs the basic encoding and decoding tasks, while Multi-HEM decomposes and fuses spatial and slice-level features in 3D, enriching the feature hierarchy by weighting them through 3D fusion. Dual-DM separates high-frequency features from the reconstructed network using self-supervision. Finally, the self-supervised high-frequency features separated by Dual-DM are inserted into the process following Multi-HEM, enhancing interactions and complementarities between contour features and hierarchical features, thereby mutually reinforcing both aspects. On the Synapse dataset, HEDN outperforms existing methods, boosting Dice Similarity Score (DSC) by 1.38% and decreasing 95% Hausdorff Distance (HD95) by 1.03 mm. Likewise, on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset, HEDN achieves 0.5% performance gains across all categories.

Deep diffusion learning of mutual-reflective structured light patterns for multi-body three-dimensional imaging

Simultaneous structured light imaging of multiple objects has become more demanding and widely in many scenarios involving robot operations in intelligent manufacturing. However, it is challenged by pattern aliasing caused by mutual reflection between high-reflective objects. To this end, we propose to learn clear fringe patterns from aliased mutual-reflective observations by diffusion models for achieving high-fidelity multi-body reconstruction in line with typical phase-shift algorithms. Regarding mutual reflection imaging as a formation of adding significant noise, we build a supervised generative learning framework based on diffusion models and then train a self-attention-based deep network with a U-Net-like skip-connected encoder-decoder architecture. We demonstrate the generalization capability of the trained model in fringe pattern recovery and its performance in phase and three-dimensional (3D) shape reconstruction. Both experimental results show that the proposed method has the expected feasibility and accuracy, heralding a promising solution for addressing the current challenge in various multi-body mutual-reflective 3D reconstruction tasks.

Multi-part kinematic constraint prediction for automatic generation of CAD model assemblies using graph convolutional networks

This paper presents a machine learning-based approach to predict kinematic constraints between CAD models that have potentially never been assembled together before. During the learning phase, the algorithm is trained to predict the next-possible-constraints between a set of parts candidate to the assembly. Assemblies are represented in a new graph-based formalism that is capable of capturing features associated with parts, interfaces between parts and constraints between them. Using such a multi-level feature extraction strategy coupled to a state-by-state graph decomposition, the approach does not need to be trained on a large database. This formalism is used to model both the network input and output where the next-possible-constraints appear after evaluation. The core of the approach relies on a series of networks based on a link-prediction encoder–decoder architecture, integrating the capabilities of several convolutional networks trained in an end-to-end manner. A decision-making algorithm is added to post-process the output and drive the prediction process in finding one among the set of next-possible-constraints. This process is repeated until no more constraints can be added. The experimental results show that the proposed approach outperforms state-of-the-art methods on such assembly tasks. Although the state-by-state assembly algorithm is iterative, it still takes into account the whole set of parts as well as the whole set of constraints already predicted, and this makes it possible to handle constraint cycles, which is generally not possible when not considering multiple parts as input.

SA-SatMVS: Slope Feature-Aware and Across-Scale Information Integration for Large-Scale Earth Terrain Multi-View Stereo

Satellite multi-view stereo (MVS) is a fundamental task in large-scale Earth surface reconstruction. Recently, learning-based multi-view stereo methods have shown promising results in this field. However, these methods are mainly developed by transferring the general learning-based MVS framework to satellite imagery, which lacks consideration of the specific terrain features of the Earth’s surface and results in inadequate accuracy. In addition, mainstream learning-based methods mainly use equal height interval partition, which insufficiently utilizes the height hypothesis surface, resulting in inaccurate height estimation. To address these challenges, we propose an end-to-end terrain feature-aware height estimation network named SA-SatMVS for large-scale Earth surface multi-view stereo, which integrates information across different scales. Firstly, we transform the Sobel operator into slope feature-aware kernels to extract terrain features, and a dual encoder–decoder architecture with residual blocks is applied to incorporate slope information and geometric structural characteristics to guide the reconstruction process. Secondly, we introduce a pixel-wise unequal interval partition method using a Laplacian distribution based on the probability volume obtained from other scales, resulting in more accurate height hypotheses for height estimation. Thirdly, we apply an adaptive spatial feature extraction network to search for the optimal fusion method for feature maps at different scales. Extensive experiments on the WHU-TLC dataset also demonstrate that our proposed model achieves the best MAE metric of 1.875 and an RMSE metric of 3.785, which constitutes a state-of-the-art performance.

A variable fidelity approach for predicting aerodynamic wall quantities of hypersonic vehicles using the ConvNeXt encoder-decoder framework

Computational fluid dynamics (CFD) simulations for obtaining three-dimensional hypersonic vehicle aerodynamic characteristics are resource-intensive. Deep learning offers a promising alternative for predicting aerodynamic wall quantities but typically requires a large dataset, which conflicts with the intention to reduce CFD costs. To address this, we propose a novel prediction model leveraging variable fidelity data to alleviate computational demands. The model utilizes an encoder-decoder architecture with ConvNeXt blocks as operators, processing variable fidelity data as inputs and outputs. We also developed a U-Net with residual blocks (ResUNet) for performance comparison. Transformation and topology patching techniques were applied to tackle the challenges posed by large grid volumes and complex topologies in three-dimensional vehicles. Results demonstrate that the ConvNeXt Encoder-Decoder predicts peak regions more accurately than ResUNet and aligns closely with CFD in low-value areas. The ConvNeXt Encoder-Decoder maintains maximum heat flux errors below 1 % and viscous drag errors below 3.81 % across varying angles of attack. It exhibits superior fitting performance with minimal deviation from CFD compared to ResUNet. Prediction accuracy decreases under multiple inflow changes compared to the angle of attack variations alone. Heat flux predictions show high consistency with CFD, with relative errors below 6.38 %, whereas friction predictions exhibit higher errors with 10.28 % maximum error and 0.28 % minimum error. The model accurately predicts friction variation trends in peak regions at the blunt edge's central plane but performs poorly in low-value areas. In summary, the ConvNeXt Encoder-Decoder accurately predicts the wall quantities under varying multiple inflow conditions.

TBConvL-Net: A hybrid deep learning architecture for robust medical image segmentation

Deep learning has shown great potential for automated medical image segmentation to improve the precision and speed of disease diagnostics. However, the task presents significant difficulties due to variations in the scale, shape, texture, and contrast of the pathologies. Traditional convolutional neural network (CNN) models have certain limitations when it comes to effectively modelling multiscale context information and facilitating information interaction between skip connections across levels. To overcome these limitations, a novel deep learning architecture is introduced for medical image segmentation, which takes advantage of CNNs and vision transformers. Our proposed model, named TBConvL-Net, involves a hybrid network that combines the local features of a CNN encoder–decoder architecture with long-range and temporal dependencies using biconvolutional long-short-term memory (LSTM) networks and vision transformers (ViT). This enables the model to capture contextual channel relationships in the data and account for the uncertainty of the segmentation over time. Additionally, we introduce a novel composite loss function that considers both segmentation robustness and boundary agreement of the predicted output with the gold standard. Our proposed model shows consistent improvement over the state of the art on ten publicly available datasets of seven different medical imaging modalities.

Auto-segmentation of hemi-diaphragms in free-breathing dynamic MRI of pediatric subjects with thoracic insufficiency syndrome.

In respiratory disorders such as thoracic insufficiency syndrome (TIS), the quantitative study of the regional motion of the left hemi-diaphragm (LHD) and right hemi-diaphragm (RHD) can give detailed insights into the distribution and severity of the abnormalities in individual patients. Dynamic magnetic resonance imaging (dMRI) is a preferred imaging modality for capturing dynamic images of respiration since dMRI does not involve ionizing radiation and can be obtained under free-breathing conditions. Using 4D images constructed from dMRI of sagittal locations, diaphragm segmentation is an evident step for the said quantitative analysis of LHD and RHD in these 4D images. In this paper, we segment the LHD and RHD in three steps: recognition of diaphragm, delineation of diaphragm, and separation of diaphragm along the mid-sagittal plane into LHD and RHD. The challenges involved in dMRI images are low resolution, motion blur, suboptimal contrast resolution, inconsistent meaning of gray-level intensities for the same object across multiple scans, and low signal-to-noise ratio. We have utilized deep learning (DL) concepts such as Path Aggregation Network and Dual Attention Network for the recognition step, Dense-Net and Residual-Net in an enhanced encoder-decoder architecture for the delineation step, and a combination of GoogleNet and Recurrent Neural Network for the identification of the mid-sagittal plane in the separation step. Due to the challenging images of TIS patients attributed to their highly distorted and variable anatomy of the thorax, in such images we localize the diaphragm using the auto-segmentations of the lungs and the thoraco-abdominal skin. We achieved an average±SD mean-Hausdorff distance of ∼3±3 mm for the delineation step and a positional error of ∼3±3 mm in recognizing the mid-sagittal plane in 100 3D test images of TIS patients with a different set of ∼430 3D images of TIS patients utilized for building the models for delineation, and separation. We showed that auto-segmentations of the diaphragm are indistinguishable from segmentations by experts, in images of near-normal subjects. In addition, the algorithmic identification of the mid-sagittal plane is indistinguishable from its identification by experts in images of near-normal subjects. Motivated by applications in surgical planning for disorders such as TIS, we have shown an auto-segmentation set-up for the diaphragm in dMRI images of TIS pediatric subjects. The results are promising, showing that our system can handle the aforesaid challenges. We intend to use the auto-segmentations of the diaphragm to create the initial ground truth (GT) for newly acquired data and then refining them, to expedite the process of creating GT for diaphragm motion analysis, and to test the efficacy of our proposed method to optimize pre-treatment planning and post-operative assessment of patients with TIS and other disorders.

Learning to enhance areal video captioning with visual question answering

ABSTRACT The utilization of Unmanned Aerial Vehicles (UAV) in remote sensing (RS) has witnessed a significant surge, offering valuable insights into Earth dynamics and human activities. However, this has led to a substantial increase in the volume of video data, rendering manual screening and analysis impractical. Consequently, there is a pressing need for the development of automated interpretation models for these aerial videos. In this paper, we propose a novel approach that leverages visual dialogue to enhance aerial video captioning. Our model adopts an encoder-decoder architecture, integrating a Visual Question Answering (VQA) task before the captioning task. The VQA task aims to enrich the captioning process by soliciting additional information about the image content. Specifically, our video encoder utilizes ViT-L/16, while the decoder employs Generative Pre-trained Transformer-2 (Distill-GPT-2). To validate our model, we introduce a novel benchmark dataset named CapERA-VQA, comprising videos accompanied by sets of questions, answers, and captions. Through experimental validation, we demonstrate the effectiveness of our proposed approach in enhancing the automated captioning of aerial videos.

Reconstructing hyperspectral images of textiles from a single RGB image utilizing the multihead self-attention mechanism

Hyperspectral images possess abundant information and play a pivotal role in enhancing the accuracy of color difference detection in textiles. However, traditional hyperspectral imaging methods necessitate costly equipment and intricate operational procedures. A novel deep learning model based on a multihead attention mechanism was proposed in this article to facilitate the extensive application of hyperspectral imaging technology in textile quality inspection. This model enabled the reconstruction of the hyperspectral information of plain weave textiles from a single RGB image. In this model, encoder-decoder architecture and pyramid pooling convolutional operations were employed to integrate multiscale features of plain weave cotton-linen textiles. This could capture details and contextual information in textile images more precisely, enhancing the accuracy of hyperspectral image reconstruction. Simultaneously, an attention mechanism was introduced to increase the model’s receptive field and improve its focus on key regions in the input image and feature maps. This resulted in a reduced weighting of redundant information during network learning, leading to an improved feature extraction capability of the network. Through these methods, successful reconstructions of plain weave textiles hyperspectral information from a single RGB image was achieved. Quantitative and qualitative tests were conducted on two datasets, namely, the NTIRE 2020 dataset and a self-made textile dataset, to evaluate the performance of the proposed method. The approach proposed in this article exhibited promising results on both datasets. Specifically, the reconstructed textile hyperspectral images achieved a root mean square error of 0.0344, a peak signal-to-noise ratio of 29.945, a spectral angle mapper of 3.753, and a structural similarity index measure of 0.955 on the textile dataset. In the reconstructed hyperspectral colorimetric experiment, the maximum value of average color difference was 2.641. These results demonstrate that the method can meet the requirements for textile color measurement applications.

AFC-Unet: Attention-fused full-scale CNN-transformer unet for medical image segmentation

In the field of medical image segmentation, although U-Net has achieved significant achievements, it still exposes some inherent disadvantages when dealing with complex anatomical structures and small targets, such as inaccurate target localization, blurry edges, and insufficient integration of contextual information. To address these challenges, this study proposes the Attention-Fused Full-Scale CNN-Transformer Unet (AFC-Unet), aiming to effectively overcome the limitations of traditional U-Net through multi-scale feature fusion, attention mechanisms, and CNN-Transformer hybrid modules. Specifically, we adopt an encoder–decoder architecture, incorporating full-scale feature block fusion and pyramid sampling modules to enhance the model’s ability to recognize fine to overall structural features by integrating cross-level multi-scale features. We propose the Multi-feature Fusion Attention Gates (MFAG) module, which focuses on and highlights discriminative information of potential lesions and key anatomical boundaries, effectively suppressing irrelevant background interference. We design a module Convolutional Hybrid Attention Transformer (CHAT) that integrates CNN and Transformer to address the shortcomings of traditional single models in handling long-range dependencies and global context understanding. Experimental results on three datasets of different scales demonstrate that the model’s segmentation performance for medical images surpasses state-of-the-art models, showcasing high generalization ability.

Removing nonrigid refractive distortions for underwater images using an attention-based deep neural network

Refractive distortions in underwater images usually occur when these images are captured through a dynamic refractive water surface, such as unmanned aerial vehicles capturing shallow underwater scenes from the surface of water or autonomous underwater vehicles observing floating platforms in the air. We propose an end-to-end deep neural network for learning to restore real scene images for removing refractive distortions. This network adopts an encoder-decoder architecture with a specially designed attention module. The use of the attention image and the distortion field generated by the proposed deep neural network can restore the exact distorted areas in more detail. Qualitative and quantitative experimental results show that the proposed framework effectively eliminates refractive distortions and refines image details. We also test the proposed framework in practical applications by embedding it into an NVIDIA JETSON TX2 platform, and the results demonstrate the practical value of the proposed framework.

Recurrent and Concurrent Prediction of Longitudinal Progression of Stargardt Atrophy and Geographic Atrophy towards Comparative Performance on Optical Coherence Tomography as on Fundus Autofluorescence

Stargardt atrophy and geographic atrophy (GA) represent pivotal endpoints in FDA-approved clinical trials. Predicting atrophy progression is crucial for evaluating drug efficacy. Fundus autofluorescence (FAF), the standard 2D imaging modality in these trials, has limitations in patient comfort. In contrast, spectral-domain optical coherence tomography (SD-OCT), a 3D imaging modality, is more patient friendly but suffers from lower image quality. This study has two primary objectives: (1) develop an efficient predictive modeling for the generation of future FAF images and prediction of future Stargardt atrophic (as well as GA) regions and (2) develop an efficient predictive modeling with advanced 3D OCT features at ellipsoid zone (EZ) for the comparative performance in the generation of future enface EZ maps and prediction of future Stargardt atrophic regions on OCT as on FAF. To achieve these goals, we propose two deep neural networks (termed ReConNet and ReConNet-Ensemble) with recurrent learning units (long short-term memory, LSTM) integrating with a convolutional neural network (CNN) encoder–decoder architecture and concurrent learning units integrated by ensemble/multiple recurrent learning channels. The ReConNet, which incorporates LSTM connections with CNN, is developed for the first goal on longitudinal FAF. The ReConNet-Ensemble, which incorporates multiple recurrent learning channels based on enhanced EZ enface maps to capture higher-order inherent OCT EZ features, is developed for the second goal on longitudinal OCT. Using FAF images at months 0, 6, and 12 to predict atrophy at month 18, the ReConNet achieved mean (±standard deviation, SD) and median Dice coefficients of 0.895 (±0.086) and 0.922 for Stargardt atrophy and 0.864 (±0.113) and 0.893 for GA. Using SD-OCT images at months 0 and 6 to predict atrophy at month 12, the ReConNet-Ensemble achieved mean and median Dice coefficients of 0.882 (±0.101) and 0.906 for Stargardt atrophy. The prediction performance on OCT images is comparably good to that on FAF. These results underscore the potential of SD-OCT for efficient and practical assessment of atrophy progression in clinical trials and retina clinics, complementing or surpassing the widely used FAF imaging technique.

SFA-Net: Semantic Feature Adjustment Network for Remote Sensing Image Segmentation

Advances in deep learning and computer vision techniques have made impacts in the field of remote sensing, enabling efficient data analysis for applications such as land cover classification and change detection. Convolutional neural networks (CNNs) and transformer architectures have been utilized in visual perception algorithms due to their effectiveness in analyzing local features and global context. In this paper, we propose a hybrid transformer architecture that consists of a CNN-based encoder and transformer-based decoder. We propose a feature adjustment module that refines the multiscale feature maps extracted from an EfficientNet backbone network. The adjusted feature maps are integrated into the transformer-based decoder to perform the semantic segmentation of the remote sensing images. This paper refers to the proposed encoder–decoder architecture as a semantic feature adjustment network (SFA-Net). To demonstrate the effectiveness of the SFA-Net, experiments were thoroughly conducted with four public benchmark datasets, including the UAVid, ISPRS Potsdam, ISPRS Vaihingen, and LoveDA datasets. The proposed model achieved state-of-the-art accuracy on the UAVid, ISPRS Vaihingen, and LoveDA datasets for the segmentation of the remote sensing images. On the ISPRS Potsdam dataset, our method achieved comparable accuracy to the latest model while reducing the number of trainable parameters from 113.8 M to 10.7 M.