U-shaped Convolutional Neural Network Research Articles

RAGE-Net: Enhanced retinal vessel segmentation U-shaped network using Gabor convolution

Extracting vessel morphology from fundus images is pivotal in acquiring pathological insights and enabling early diagnosis of retinal disorders. Manual segmentation of retinal vessels requires a high degree of expertise and is notably time-intensive. Although existing deep learning techniques for retinal vessel segmentation predominantly hinge on U-shaped convolutional neural networks, significant headway has been made, complexities persist in delineating faint, low-contrast vessels amidst noisy backgrounds. To confront these challenges, we propose an innovative U-shaped convolutional neural network fortified with oriented priors, labeled as the Receptive Field Aggregating Gabor Enhance Network (RAGE-Net). We revamp the conventional U-shaped convolutional network with a foundation in Gabor wavelet and Gabor convolutional network, introducing a Gabor Matching Enhance Architecture (GMEA) amalgamated into the U-shaped convolutional network. This architecture comprises two distinct modules. Initially, a Dual-scale Gabor Enhance Module (DGEB) is introduced to bolster vessel continuity and effectively fortify delicate vessels by integrating oriented feature enhancement through Gabor convolution. Subsequently, a Receptive Field Pyramid Module (RPM) is proposed to supplant the escalation of scale count in the Gabor filter bank for vessel alignment, also serving as feature fusion to enhance the network's comprehensive vessel discernment. In comparison to U-Net, our model boasts fewer parameters and surpasses in terms of sensitivity, accuracy, and F1 score on the DRIVE dataset by 3.58%, 0.34%, and 2.29%, respectively. Our model showcases stellar performance across three public datasets: DRIVE, STARE, and CHASE_DB1, with sensitivity values of 0.8172, 0.8126, and 0.8540 and accuracy figures of 0.9708, 0.9725, and 0.9757, respectively. The analysis on three datasets reveals that our model demonstrates distinct strengths in AUC and Se as a result of integrating GMEA, which comprises RPM and DGEB, into U-shaped networks. However, it does not reduce overall accuracy to improve the model's ability to perceive weak contrast and small blood vessels. While maintaining superior Acc, our model also demonstrates advancements in Sp and F1 score, indicating a balanced progression in multiple evaluation metrics.

A Parallel Convolutional Network Based on Spiking Neural Systems.

Deep convolutional neural networks have shown advanced performance in accurately segmenting images. In this paper, an SNP-like convolutional neuron structure is introduced, abstracted from the nonlinear mechanism in nonlinear spiking neural P (NSNP) systems. Then, a U-shaped convolutional neural network named SNP-like parallel-convolutional network, or SPC-Net, is constructed for segmentation tasks. The dual-convolution concatenate (DCC) and dual-convolution addition (DCA) network blocks are designed, respectively, in the encoder and decoder stages. The two blocks employ parallel convolution with different kernel sizes to improve feature representation ability and make full use of spatial detail information. Meanwhile, different feature fusion strategies are used to fuse their features to achieve feature complementarity and augmentation. Furthermore, a dual-scale pooling (DSP) module in the bottleneck is designed to improve the feature extraction capability, which can extract multi-scale contextual information and reduce information loss while extracting salient features. The SPC-Net is applied in medical image segmentation tasks and is compared with several recent segmentation methods on the GlaS and CRAG datasets. The proposed SPC-Net achieves 90.77% DICE coefficient, 83.76% IoU score and 83.93% F1 score, 86.33% ObjDice coefficient, 135.60 Obj-Hausdorff distance, respectively. The experimental results show that the proposed model can achieve good segmentation performance.

DBNet-SI: Dual branch network of shift window attention and inception structure for skin lesion segmentation

The U-shaped convolutional neural network (CNN) has attained remarkable achievements in the segmentation of skin lesion. However, given the inherent locality of convolution, this architecture cannot capture long-range pixel dependencies and multiscale global contextual information effectively. Moreover, repeated convolutions and downsampling operations can readily result in the omission of intricate local fine-grained details. In this paper, we proposed a U-shaped network (DBNet-SI) equipped with a dual-branch module that combines shift window attention and inception structures. First, we proposed a dual-branch module that combines shift window attention and inception structures (MSI) to better capture multiscale global contextual information and long-range pixel dependencies. Specifically, we have devised a cross-branch bidirectional interaction module within the MSI module to enable information complementarity between the two branches in the channel and spatial dimensions. Therefore, MSI is capable of extracting distinguishing and comprehensive features to accurately identify the skin lesion boundaries. Second, we have devised a progressive feature enhancement and information compensation module (PFEIC), which progressively compensates for fine-grained features through reconstructed skip connections and integrated global context attention modules. The results of the experiment show the superior segmentation performance of DBNet-SI compared with other deep learning models for skin lesion segmentation in the ISIC2017 and ISIC2018 datasets. Ablation studies demonstrate that our model can effectively extract rich multiscale global contextual information and compensate for the loss of local details.

A modified U-Net convolutional neural network for segmenting periprostatic adipose tissue based on contour feature learning

ObjectiveThis study trains a U-shaped fully convolutional neural network (U-Net) model based on peripheral contour measures to achieve rapid, accurate, automated identification and segmentation of periprostatic adipose tissue (PPAT). MethodsCurrently, no studies are using deep learning methods to discriminate and segment periprostatic adipose tissue. This paper proposes a novel and modified, U-shaped convolutional neural network contour control points on a small number of datasets of MRI T2W images of PPAT combined with its gradient images as a feature learning method to reduce feature ambiguity caused by the differences in PPAT contours of different patients. This paper adopts a supervised learning method on the labeled dataset, combining the probability and spatial distribution of control points, and proposes a weighted loss function to optimize the neural network's convergence speed and detection performance. Based on high-precision detection of control points, this paper uses a convex curve fitting to obtain the final PPAT contour. The imaging segmentation results were compared with those of a fully convolutional network (FCN), U-Net, and semantic segmentation convolutional network (SegNet) on three evaluation metrics: Dice similarity coefficient (DSC), Hausdorff distance (HD), and intersection over union ratio (IoU). ResultsCropped images with a 270 × 270-pixel matrix had DSC, HD, and IoU values of 70.1%, 27 mm, and 56.1%, respectively; downscaled images with a 256 × 256-pixel matrix had 68.7%, 26.7 mm, and 54.1%. A U-Net network based on peripheral contour characteristics predicted the complete periprostatic adipose tissue contours on T2W images at different levels. FCN, U-Net, and SegNet could not completely predict them. ConclusionThis U-Net convolutional neural network based on peripheral contour features can identify and segment periprostatic adipose tissue quite well. Cropped images with a 270 × 270-pixel matrix are more appropriate for use with the U-Net convolutional neural network based on contour features; reducing the resolution of the original image will lower the accuracy of the U-Net convolutional neural network. FCN and SegNet are not appropriate for identifying PPAT on T2 sequence MR images. Our method can automatically segment PPAT rapidly and accurately, laying a foundation for PPAT image analysis.

CMPF-UNet: a ConvNeXt multi-scale pyramid fusion U-shaped network for multi-category segmentation of remote sensing images

Most U-shaped convolutional neural network (CNN) methods have the problems of insufficient feature extraction and fail to fully utilize global/multi-scale context information, which makes it difficult to distinguish similar objects and shadow occluded objects in remote sensing images. This article proposes a ConvNeXt multi-scale pyramid fusion U-shaped network (CMPF-UNet). In this work, we first propose a novel backbone network based on ConvNeXt to enhance image feature extraction, and use ConvNeXt bottleneck blocks to reconstruct the decoder. Furthermore, a scale aware pyramid fusion (SAPF) module and Residual Atrous Spatial Pyramid Pooling (RASPP) module are proposed to dynamically fuse the rich multi-scale context information in advanced features. Finally, multiple Global Pyramid Guidance (GPG) modules are embedded in the network, aiming to provide different levels of global context information for the decoder by reconstructing skip-connections. Experiments on the Vaihingen and Potsdam datasets indicate that the proposed CMPF-UNet segmentation achieves more accurate results.

Tracking and Localization Method of Pipeline Pigs Based on a Noise Reduction Autoencoder

In the pigging operation of an oil pipeline, the traditional pigging localization method can only achieve fixed-point detection and cannot track continuously. To ensure the normal operation of the pigging operation and track and locate the pig in a timely manner, a tracking and localization method of an oil pipeline pig based on a noise reduction autoencoder is proposed. First, using the advantages of U-Net (U-shaped Convolutional Neural Network) multi-scale feature extraction, combined with coding and residual-dense blocks, U-Net combined with Residual Dense Block U-shaped Network (RDBU) was proposed for noise reduction, and then real-time localization was calculated out based on the negative pressure wave localization formula of pigs. The experimental results show that, compared with the traditional method, the SNR of the RDBU is increased by 0.9dB and the root mean square error is reduced by 14.92%. The denoising algorithm in this paper can effectively eliminate the noise of the negative pressure wave signal of pigs.

Sparse Coding Inspired LSTM and Self-Attention Integration for Medical Image Segmentation.

Accurate and automatic segmentation of medical images plays an essential role in clinical diagnosis and analysis. It has been established that integrating contextual relationships substantially enhances the representational ability of neural networks. Conventionally, Long Short-Term Memory (LSTM) and Self-Attention (SA) mechanisms have been recognized for their proficiency in capturing global dependencies within data. However, these mechanisms have typically been viewed as distinct modules without a direct linkage. This paper presents the integration of LSTM design with SA sparse coding as a key innovation. It uses linear combinations of LSTM states for SA's query, key, and value (QKV) matrices to leverage LSTM's capability for state compression and historical data retention. This approach aims to rectify the shortcomings of conventional sparse coding methods that overlook temporal information, thereby enhancing SA's ability to do sparse coding and capture global dependencies. Building upon this premise, we introduce two innovative modules that weave the SA matrix into the LSTM state design in distinct manners, enabling LSTM to more adeptly model global dependencies and meld seamlessly with SA without accruing extra computational demands. Both modules are separately embedded into the U-shaped convolutional neural network architecture for handling both 2D and 3D medical images. Experimental evaluations on downstream medical image segmentation tasks reveal that our proposed modules not only excel on four extensively utilized datasets across various baselines but also enhance prediction accuracy, even on baselines that have already incorporated contextual modules. Code is available at https://github.com/yeshunlong/SALSTM.

CPFTransformer: transformer fusion context pyramid medical image segmentation network.

The application of U-shaped convolutional neural network (CNN) methods in medical image segmentation tasks has yielded impressive results. However, this structure's single-level context information extraction capability can lead to problems such as boundary blurring, so it needs to be improved. Additionally, the convolution operation's inherent locality restricts its ability to capture global and long-distance semantic information interactions effectively. Conversely, the transformer model excels at capturing global information. Given these considerations, this paper presents a transformer fusion context pyramid medical image segmentation network (CPFTransformer). The CPFTransformer utilizes the Swin Transformer to integrate edge perception for segmentation edges. To effectively fuse global and multi-scale context information, we introduce an Edge-Aware module based on a context pyramid, which specifically emphasizes local features like edges and corners. Our approach employs a layered Swin Transformer with a shifted window mechanism as an encoder to extract contextual features. A decoder based on a symmetric Swin Transformer is employed for upsampling operations, thereby restoring the resolution of feature maps. The encoder and decoder are connected by an Edge-Aware module for the extraction of local features such as edges and corners. Experimental evaluations on the Synapse multi-organ segmentation task and the ACDC dataset demonstrate the effectiveness of our method, yielding a segmentation accuracy of 79.87% (DSC) and 20.83% (HD) in the Synapse multi-organ segmentation task. The method proposed in this paper, which combines the context pyramid mechanism and Transformer, enables fast and accurate automatic segmentation of medical images, thereby significantly enhancing the precision and reliability of medical diagnosis. Furthermore, the approach presented in this study can potentially be extended to image segmentation of other organs in the future.

A novel convolutional neural network for identification of retinal layers using sliced optical coherence tomography images

Retinal imaging is crucial for observing the retina and accurately diagnosing pathological problems. Optical Coherence Tomography (OCT) has been a transformative breakthrough for developing high-resolution cross-sectional images. It is imperative to delineate the multiple layers of the retina for a proper diagnosis. A novel segmentation-based approach is introduced in this study to identify seven distinct layers of the retina using OCT images. The developed approach presents SliceOCTNet, a customized U-shaped Convolutional Neural Network (CNN) that introduces group normalization and intricate skip connections. Paired alongside a hybrid loss function, the SliceOCTNet outperformed most state-of-the-art approaches. The introduction of Group Normalization in SliceOCTNet stabilized the model and improved layer identification even when working with small datasets. The use of skip connections also contributed to an improvement in the spatial outlook of the model. Implementing a hybrid loss function addresses the class imbalance problem in the dataset. Duke University’s spectral-domain optical coherence tomography (SD-OCT) B-scan dataset of healthy and Diabetic Macular Edema (DME) afflicted patients was utilized to train and evaluate the SliceOCTNet. The model accurately recognizes the seven layers of the retina. It can achieve a high dice coefficient value of 0.941 and refine the segmentation process to a higher level of precision.

A three-stage pavement image crack detection framework with positive sample augmentation

Most pavement crack detection methods based on deep learning rely too much on pixel-wise labels, and are facing with sample imbalance problem. This paper proposes a three-stage pavement crack detection framework with positive sample augmentation by an autoencoder-deep convolutional generative adversarial network (E-DCGAN). Positive samples are the images labeled as crack in the training datasets. Firstly, crack regions are located using graph based on visual saliency (GBVS). Secondly, a convolutional neural network (CNN) model is designed to recognize crack pixels in the located crack regions. The E-DCGAN is explored to generate crack images, which are then added as augmented positive samples in the training dataset of the CNN model. The trained CNN model takes the located crack regions as inputs, outputs the recognized crack pixels. Finally, fine crack detection is realized through region growing. Crack contours are completely depicted pixel-wise based on the recognized crack pixels and the Breadth-first search. Experiments are carried out on two benchmark datasets and in practical applications. The results showed that the framework could detect cracks at the pixel level. Its performance is improved compared with standard fully convolutional network (FCN)-based and U-shaped CNN (U-Net)-based crack detection methods. In addition, the standard FCN-based and U-Net-based methods require accurate pixel-wise annotations, while the proposed framework only requires image-wise annotations. It can be concluded that the proposed framework not only effectively avoids the dependence on pixel-level annotations, improves the sample imbalance issues, but also achieves competitive detection performance over the FCN and U-Net based methods.

Neural Networks Push the Limits of Luminescence Lifetime Nanosensing.

Luminescence lifetime-based sensing is ideally suited to monitor biological systems due to its minimal invasiveness and remote working principle. Yet, its applicability is limited in conditions of low signal-to-noise ratio (SNR) induced by, e.g., short exposure times and presence of opaque tissues. Herein this limitation is overcome by applying a U-shaped convolutional neural network (U-NET) to improve luminescence lifetime estimation under conditions of extremely low SNR. Specifically, the prowess of the U-NET is showcased in the context of luminescence lifetime thermometry, achieving more precise thermal readouts using Ag2 S nanothermometers. Compared to traditional analysis methods of decay curve fitting and integration, the U-NET can extract average lifetimes more precisely and consistently regardless of the SNR value. The improvement achieved in the sensing performance using the U-NET is demonstrated with two experiments characterized by extreme measurement conditions: thermal monitoring of free-falling droplets, and monitoring of thermal transients in suspended droplets through an opaque medium. These results broaden the applicability of luminescence lifetime-based sensing in fields including in vivo experimentation and microfluidics, while, hopefully, spurring further research on the implementation of machine learning (ML) in luminescence sensing.

Multi-Layer Preprocessing and U-Net with Residual Attention Block for Retinal Blood Vessel Segmentation.

Retinal blood vessel segmentation is a valuable tool for clinicians to diagnose conditions such as atherosclerosis, glaucoma, and age-related macular degeneration. This paper presents a new framework for segmenting blood vessels in retinal images. The framework has two stages: a multi-layer preprocessing stage and a subsequent segmentation stage employing a U-Net with a multi-residual attention block. The multi-layer preprocessing stage has three steps. The first step is noise reduction, employing a U-shaped convolutional neural network with matrix factorization (CNN with MF) and detailed U-shaped U-Net (D_U-Net) to minimize image noise, culminating in the selection of the most suitable image based on the PSNR and SSIM values. The second step is dynamic data imputation, utilizing multiple models for the purpose of filling in missing data. The third step is data augmentation through the utilization of a latent diffusion model (LDM) to expand the training dataset size. The second stage of the framework is segmentation, where the U-Nets with a multi-residual attention block are used to segment the retinal images after they have been preprocessed and noise has been removed. The experiments show that the framework is effective at segmenting retinal blood vessels. It achieved Dice scores of 95.32, accuracy of 93.56, precision of 95.68, and recall of 95.45. It also achieved efficient results in removing noise using CNN with matrix factorization (MF) and D-U-NET according to values of PSNR and SSIM for (0.1, 0.25, 0.5, and 0.75) levels of noise. The LDM achieved an inception score of 13.6 and an FID of 46.2 in the augmentation step.

Segmentation of backscattered electron images of cement-based materials using lightweight U-Net with attention mechanism (LWAU-Net)

Backscattered electron (BSE) image segmentation is used to characterise the microstructure of cement-based materials to deepen the understanding of macroscopic properties. However, processing bottlenecks have been encountered when using conventional segregation methods for cement-based materials with complex imaging conditions, and it is often necessary to manually adjust the parameters in image processing. This paper used a lightweight U-shaped convolutional neural network with an attention mechanism (LWAU-Net) to segment the backscattered electron images of cement-based materials, and the results of different attention modules were investigated. An augmentation method combining jigsaw transform and piecewise affine is also proposed. The presented segmentation method can more completely segment the hydrated cement, unhydrated cement, and pore regions under different sampling conditions. It achieves high-precision recognition by using lightweight models. The paper finally quantifies the degree of cement hydration and proves the feasibility of the proposed method.

An automatic cephalometric landmark detection method based on heatmap regression and Monte Carlo dropout.

Cephalometric analysis plays an important role in orthodontic diagnosis and treatment planning. It depends on the detection of multiple landmarks, while the process is time-consuming and tedious. Although some deep learning-based automatic landmark detection algorithms have achieved excellent performance, most of them adopt multi-stage models increasing the complexity and detection time. Meanwhile, few studies focused on the uncertainty of detection results, thereby ignoring its significant clinical value. In this paper, we propose a novel approach based on heatmap regression for landmark detection, which can achieve competitive accuracy and good robustness with only one step. Furthermore, by applying Monte Carlo dropout to a U-shaped convolutional neural network, we can obtain not only the coordinate of each landmark but also the corresponding simple uncertainty, so that doctors can pay more attention to those landmarks with higher uncertainty. The evaluation results showed the mean radial error is 1.39±1.06mm and the average successful detection rate is 79.65%, 97.22% within 2mm, 4mm for the IEEE ISBI2015 Test Dataset 1, the indicators for the IEEE ISBI2015 Test Dataset 2 are 1.33±0.93mm, 80.05% and 97.53%, respectively. Our method has the potential to become an assistant tool in clinical practice. Automatic and accurate detection with uncertainty analysis is expected to help guide the doctor's judgment.

A new fluid flow approximation method using a vision transformer and a U-shaped convolutional neural network

Numerical simulation of fluids is important in modeling a variety of physical phenomena, such as weather, climate, aerodynamics, and plasma physics. The Navier–Stokes equations are commonly used to describe fluids, but solving them at a large scale can be computationally expensive, particularly when it comes to resolving small spatiotemporal features. This trade-off between accuracy and tractability can be challenging. In this paper, we propose a novel artificial intelligence-based method for improving fluid flow approximations in computational fluid dynamics (CFD) using deep learning (DL). Our method, called CFDformer, is a surrogate model that can handle both local and global features of CFD input data. It is also able to adjust boundary conditions and incorporate additional flow conditions, such as velocity and pressure. Importantly, CFDformer performs well under different velocities and pressures outside of the flows it was trained on. Through comprehensive experiments and comparisons, we demonstrate that CFDformer outperforms other baseline DL models, including U-shaped convolutional neural network (U-Net) and TransUNet models.

Registration of 3D medical images based on unsupervised cooperative cascade of deep networks

In this paper, a deformable registration network (DR-Net) and a multi-scale cascading strategy are designed for the registration of largely deformed 3D medical images. Our DR-Net appears as a U-shaped convolutional neural network with a pyramidal input module (PIM), a light weighted sequential Inception module and an SCAM convolutional attention module. Our multi-scale cooperative cascading strategy integrates the deformation field information within and between sub-networks at different scales to synthesize the cascaded deformation fields. To cooperatively train the cascaded network, not only the output of the final network layer but also the multi-scale outputs from different layers of the decoder in the last cascaded sub-network are used to calculate loss function. As compared with the VoxelMorph and IVTN, the average dice similarity coefficients (Dice) achieved with our DR-Net are 2.4% and 2.5% higher on the Sliver dataset and are 2.5% and 2.4% higher on the LiTS dataset. The average Dice coefficients achieved with our multi-scale cascading strategy of three DR-Nets are 1.6% and 1.9% higher than those of the VM-CR3 and are 1.5% and 1.7% higher than those of the IVTN-CR3 on these two datasets, respectively. These results show that not only our proposed DR-Net itself but also the cascade of them outperform the state-of-the-art methods and their cascades in registration accuracy.

JAUNet: A U-Shape Network with Jump Attention for Semantic Segmentation of Road Scenes

The task of complex scene semantic segmentation is to classify and label the scene image pixel by pixel. For the complex image information in autonomous driving scenes, its characteristics such as many kinds of targets and various scene changes make the segmentation task more difficult, making various kinds of FCN-based networks unable to restore the image information well. In contrast, the encoder–decoder network structure represented by SegNet and UNet uses jump connections and other methods to restore image information. Still, its extraction of shallow details is simple and unfocused. In this paper, we propose a U-shaped convolutional neural network with a jump attention mechanism, which is an improved encoder plus decoder structure to achieve semantic segmentation by four times of convolutional downsampling and four transposed convolutional upsamplings while adding a jump attention module in the upsampling process to realize selective extraction of contextual information from high-dimensional features to guide low-dimensional features, improve the fusion of deep and shallow features, and ensure the consistency of the same type of pixel prediction. The CamVid and Cityscapes datasets are sampled for the experiments, and the model ground mIoU evaluation metrics can reach 66.3% and 69.1%. Compared with other mainstream semantic segmentation algorithms, this method is competitive in terms of segmentation performance and model size.

Denoising Method for Seismic Co-Band Noise Based on a U-Net Network Combined with a Residual Dense Block

To address the problem of waveform distortion in the existing seismic signal denoising method when removing co-band noise, further improving the signal-to-noise ratio (SNR) of seismic signals and enhancing their quality, this paper designs a seismic co-band denoising model Atrous Residual Dense Block U-Net (ARDU), which uses a U-shaped convolutional neural network (U-Net) as a basic framework and combines atrous convolution and the residual dense block (RDB). In the ARDU model, atrous convolution is connected with residual dense blocks to form the feature extraction unit of the model encoder. Among them, the residual dense blocks can deepen the network’s depth and enhance the feature extraction ability of the network on the premise of mitigating the gradient-vanishing and gradient-exploding problem. Atrous convolution can enlarge receptive fields, reduce waveform distortion, and protect effective signals without increasing network parameters. To test the denoising performance of the ARDU model, the Stanford Global Seismic dataset was used to construct a training set and a test set and the model was trained and tested on it. The experimental results of the ARDU model for different types of seismic co-band noise showed that this model can effectively remove seismic co-band noise, protect effective signals, improve the SNR of seismic signals, and enhance the quality of seismic signals. To further verify the denoising effect of the model, this model was compared with the wavelet threshold denoising U-Net model and the denoising residual dense block (DnRDB) model, and the results showed that the ARDU model has the best SNR, r (correlation coefficient), and root-mean-square error (RMSE) and the least distortion of the seismic signal waveform.

Denoising image by matrix factorization in U-shaped convolutional neural network

Image denoising requires both spatial details and global contextualized information to recover a clean version from the deteriorative one. Previous deep convolution networks usually focus on modeling the local feature and stacked convolution blocks to expand the receptive field, which can catch the long-distance dependencies. However, contrary to the expectation, the extracted local feature incapacity recovers the global details by traditional convolution while the stacked blocks hinder the information flow. To tackle these issues, we introduce the Matrix Factorization Denoising Module (MD) to model the interrelationship between the global context aggregating process and the reconstructed process to attain the context details. Besides, we redesign a new basic block to ease the information flow and maintain the network performance. In addition, we conceive the Feature Fusion Module (FFU) to fuse the information from the different sources. Inspired by the multi-stage progressive restoration architecture, we adopt two-stage convolution branches progressively reconstructing the denoised image. In this paper, we propose an original and efficient neural convolution network dubbed MFU. Experimental results on various image denoising datasets: SIDD, DND, and synthetic Gaussian noise datasets show that our MFU can produce comparable visual quality and accuracy results with state-of-the-art methods.

TMCrack-Net: A U-Shaped Network with a Feature Pyramid and Transformer for Mural Crack Segmentation

The detection of crack information is very important in mural conservation. In practice, the number of ancient murals is scarce, and the difficulty of collecting digital information about murals leads to minimal data being collected. Crack information appears in pictures of paintings, which resembles painting traces and is easy to misidentify. However, the current mainstream semantic segmentation networks directly use the features of the backbone network for prediction, which do not fully use the features at different scales and ignore the differences between the decoder and encoder features. This paper proposes a new U-shaped convolutional neural network with feature pyramids and a transformer called TMCrack-net. Instead of U-Net’s jump-join, an AG-BiFPN network is used, which consists of two modules: a channel cross-fusion (CCT) module with a transformer and a bidirectional feature pyramid network. While fully using the information in different network dimensions, the channel cross-fusion module optimizes the final features of each layer to reduce the confounding effect caused by the fusion of features. For the fusion of multi-scale channel information with decoder features, we designed a fusion module based on a channel attention (called FCA) to guide the fusion of enhanced encoder features with decoder features and reduce the ambiguity between the two feature sets. To demonstrate the effectiveness and generalization of the model, TMCrack-Net was evaluated on the Tang Dynasty tomb chamber mural dataset and Crack500. MIou values of 0.7731 and 0.7944 were achieved, respectively, which are better than those of other advanced crack detection methods. The method yields accurate segmentation performance and is advantageous for mural painting crack segmentation tasks.