Deformable Convolutional Neural Network Research Articles

TriConvUNeXt: A Pure CNN-Based Lightweight Symmetrical Network for Biomedical Image Segmentation.

Biomedical image segmentation is essential in clinical practices, offering critical insights for accurate diagnosis and strategic treatment approaches. Nowadays, self-attention-based networks have achieved competitive performance in both natural language processing and computer vision, but the computational cost has reduced their popularity in practical applications. The recent study of Convolutional Neural Network (CNN) explores linear functions within modified CNN layer demonstrating pure CNN-based networks can still achieve competitive results against Vision Transformer (ViT) in biomedical image segmentation, with fewer parameters. The modified CNN, i.e., Depthwise CNN, however, leaves the features cleaved off in the channel dimension and prevents the extraction of features in the perspective of channel interaction. To effectively further explore the feature learning power of modified CNN with biomedical image segmentation, we design a lightweight multi-convolutional multi-scale convolutional network block (MSConvNeXt) for U-shape symmetrical network. Specifically, a network block consisting of a depthwise CNN, a deformable CNN, and a dilated CNN is proposed to capture semantic feature information effectively while with low computational cost. Furthermore, channel shuffling operation is proposed to dynamically promote an efficient feature fusion among the feature maps. The network block hereby is properly deployed within U-shape symmetrical encoder-decoder style network, named TriConvUNeXt. The proposed network is validated on a public benchmark dataset with a comprehensive evaluation in both computational cost and segmentation performance against 13 baseline methods. Specifically, TriConvUNeXt achieves 1% higher than UNet and TransUNet in Dice-Coefficient while 81% and 97% lower in computational cost. The implementation of TriConvUNeXt is made publicly accessible via https://github.com/ziyangwang007/TriConvUNeXt .

Analysis of interactive impact and student behavior of teaching online courses based on gamified learning

The potential of gamified learning as an emerging force for educational change has been widely recognized, to address the design deficiencies in the automatic recognition methods in traditional gamified learning models and the low accuracy of learning behavior recognition. Based on the improved gamified learning model, this paper jointly proposes an optimal design and recognition model of learning mode based on a deformable convolutional type network. Based on the gamified machine learning theory and interactive feedback theory, it comprehensively analyzes the procedural improvement of the gamified interactive intelligence model to provide a theoretical and practical basis for carrying out the analysis of the interactive impact and student behavior of teaching online courses based on gamified learning. The research results show that: (1) with the increase of the number of gamified learning samples, the value of iterative loss function decreases, the accuracy of recognition improves, and gradually converges and tends to be stable when the number of training times is 500; the 800 training samples setting can fully meet the network training requirements. (2) The deformable convolutional neural learning network constructed with the proposed joint algorithm reduces the network error during the training process and introduces the radical change function to improve the data processing capability. This improved gamified learning model is not only superior in recognition accuracy but also shows great advantages in recognition time.

Motion recognition of football players based on deformable convolutional neural networks

Motion recognition of football players based on deformable convolutional neural networks

Recurrent Residual Deformable Conv Unit and Multi-Head with Channel Self-Attention Based on U-Net for Building Extraction from Remote Sensing Images

Considering the challenges associated with accurately identifying building shape features and distinguishing between building and non-building features during the extraction of buildings from remote sensing images using deep learning, we propose a novel method for building extraction based on U-Net, incorporating a recurrent residual deformable convolution unit (RDCU) module and augmented multi-head self-attention (AMSA). By replacing conventional convolution modules with an RDCU, which adopts a deformable convolutional neural network within a residual network structure, the proposed method enhances the module’s capacity to learn intricate details such as building shapes. Furthermore, AMSA is introduced into the skip connection function to enhance feature expression and positions through content–position enhancement operations and content–content enhancement operations. Moreover, AMSA integrates an additional fusion channel attention mechanism to aid in identifying cross-channel feature expression Intersection over Union (IoU) score differences. For the Massachusetts dataset, the proposed method achieves an Intersection over Union (IoU) score of 89.99%, PA (Pixel Accuracy) score of 93.62%, and Recall score of 89.22%. For the WHU Satellite dataset I, the proposed method achieves an IoU score of 86.47%, PA score of 92.45%, and Recall score of 91.62%, For the INRIA dataset, the proposed method achieves an IoU score of 80.47%, PA score of 90.15%, and Recall score of 85.42%.

Robust spatial temporal imputation based on spatio-temporal generative adversarial nets

Spatial temporal imputation is an important part of intelligent transportation system, and spatial temporal data missing also largely affects the performance of its downstream tasks. Spatial temporal data are affected by spatial, temporal, and unexpected conditions, and has a complex change pattern. Existing researches mostly focus on time series and often ignore the modeling of spatial dependency. Moreover, the imputation performance is poor under high missing rate or block missing scenarios, needing to enhance the model robustness. Therefore, we proposed a spatio-temporal generative adversarial imputation network (named STGAIN) to handle with the complex spatial temporal dependency and the diversity of missing conditions on spatio-temporal data. It includes a spatio-temporal generator and a spatio-temporal discriminator, which uses a graph convolutional neural network as a spatial aggregator and a one-dimensional deformable convolutional neural network as a temporal extractor. The generator observes the unmissing part data and the spatial relationships to impute the missing parts. The discriminator then takes the completed data as input and attempts to determine which components were actually observed and which were imputed. The spatio-temporal modeling ability was improved in the competition with each other. In order to make the generator in GAN distinguish the observed part from the missing part, a fusion loss function (including reconstruction loss and BCE loss) was designed for the generator training. Finally, experiments were conducted on two real highway datasets for different missing rates. And we test the model robustness under different size of dataset and noise interference. The results show that STGAIN significantly outperforms the existing methods.

Replay attack detection based on deformable convolutional neural network and temporal-frequency attention model

Abstract As an important identity authentication method, speaker verification (SV) has been widely used in many domains, e.g., mobile financials. At the same time, the existing SV systems are insecure under replay spoofing attacks. Toward a more secure and stable SV system, this article proposes a replay attack detection system based on deformable convolutional neural networks (DCNNs) and a time–frequency double-channel attention model. In DCNN, the positions of elements in the convolutional kernel are not fixed. Instead, they are modified by some trainable variable to help the model extract more useful local information from input spectrograms. Meanwhile, a time–frequency domino double-channel attention model is adopted to extract more effective distinctive features to collect valuable information for distinguishing genuine and replay speeches. Experimental results on ASVspoof 2019 dataset show that the proposed model can detect replay attacks accurately.

BCE-Net: Reliable building footprints change extraction based on historical map and up-to-date images using contrastive learning

Automatic and periodic recompiling of building databases with up-to-date high-resolution images has become a critical requirement for rapidly developing urban environments. However, the architecture of most existing approaches for change extraction attempts to learn features related to changes but ignores objectives related to buildings. This inevitably leads to the generation of significant pseudo-changes, due to factors such as seasonal changes in images and the inclination of building façades. To alleviate the above-mentioned problems, we developed a contrastive learning approach by validating historical building footprints against single up-to-date remotely sensed images. This contrastive learning strategy allowed us to inject the semantics of buildings into a pipeline for the detection of changes, which is achieved by increasing the distinguishability of features of buildings from those of non-buildings. In addition, to reduce the effects of inconsistencies between historical building polygons and buildings in up-to-date images, we employed a deformable convolutional neural network to learn offsets intuitively. In summary, we formulated a multi-branch building extraction method that identifies newly constructed and removed buildings, respectively. To validate our method, we conducted comparative experiments using the public Wuhan University building change detection dataset and a more practical dataset named SI-BU that we established. Our method achieved F1 scores of 93.99% and 70.74% on the above datasets, respectively. Moreover, when the data of the public dataset were divided in the same manner as in previous related studies, our method achieved an F1 score of 94.63%, which surpasses that of the state-of-the-art method. Code and datasets are available at https://vrlab.org.cn/~hanhu/projects/bcenet.

A Novel Deep Learning Network with Deformable Convolution and Attention Mechanisms for Complex Scenes Ship Detection in SAR Images

Synthetic aperture radar (SAR) can detect objects in various climate and weather conditions. Therefore, SAR images are widely used for maritime object detection in applications such as maritime transportation safety and fishery law enforcement. However, nearshore ship targets in SAR images are often affected by background clutter, resulting in a low detection rate, high false alarm rate, and high missed detection rate, especially for small-scale ship targets. To address this problem, in this paper, we propose a novel deep learning network with deformable convolution and attention mechanisms to improve the Feature Pyramid Network (FPN) model for nearshore ship target detection in SAR images with complex backgrounds. The proposed model uses a deformable convolutional neural network in the feature extraction network to adapt the convolution position to the target sampling point, enhancing the feature extraction ability of the target, and improving the detection rate of the ship target against the complex background. Moreover, this model uses a channel attention mechanism to capture the feature dependencies between different channel graphs in the feature extraction network and reduce the false detection rate. The designed experiments on a public SAR image ship dataset show that our model achieves 87.9% detection accuracy for complex scenes and 95.1% detection accuracy for small-scale ship targets. A quantitative comparison of the proposed model with several classical and recently developed deep learning models on the same SAR images dataset demonstrated the superior performance of the proposed method over other models.

Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF

The semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote sensing image segmentation tasks. Traditional convolutional-neural-network-based models often ignore spatial information in the feature extraction stage and pay less attention to global context information. However, spatial context information is important in complex remote sensing images, which means that the segmentation effect of traditional models needs to be improved. In addition, neural networks with a superior segmentation performance often suffer from the problem of high computational resource consumption. To address the above issues, this paper proposes a combination model of a modified multiscale deformable convolutional neural network (mmsDCNN) and dense conditional random field (DenseCRF). Firstly, we designed a lightweight multiscale deformable convolutional network (mmsDCNN) with a large receptive field to generate a preliminary prediction probability map at each pixel. The output of the mmsDCNN model is a coarse segmentation result map, which has the same size as the input image. In addition, the preliminary segmentation result map contains rich multiscale features. Then, the multi-level DenseCRF model based on the superpixel level and the pixel level is proposed, which can make full use of the context information of the image at different levels and further optimize the rough segmentation result of mmsDCNN. To be specific, we converted the pixel-level preliminary probability map into a superpixel-level predicted probability map according to the simple linear iterative clustering (SILC) algorithm and defined the potential function of the DenseCRF model based on this. Furthermore, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider the features of various scales and prevent poor superpixel segmentation results from affecting the final result. To restore the contour of the object more clearly, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused them into the potential function of the DenseCRF model. Finally, extensive experiments on the Potsdam and Vaihingen datasets demonstrated that the proposed model exhibited significant advantages compared to the current state-of-the-art models.

A deformable CNN architecture for predicting clinical acceptability of ECG signal

The degraded quality of the electrocardiogram (ECG) signals is the main source of false alarms in critical care units. Therefore, a preliminary analysis of the ECG signal is required to decide its clinical acceptability. In conventional techniques, different handcrafted features are extracted from the ECG signal based on signal quality indices (SQIs) to predict clinical acceptability. A one-dimensional deformable convolutional neural network (1D-DCNN) is proposed in this work to extract features automatically, without manual interference, to detect the clinical acceptability of ECG signals efficiently. In order to create DCNN, the deformable convolution and pooling layers are merged into the regular convolutional neural network (CNN) architecture. In DCNN, the equidistant sampling locations of a regular CNN are replaced with adaptive sampling locations, which improves the network’s ability to learn based on the input. Deformable convolution layers concentrate more on significant segments of the ECG signals rather than giving equal attention to all segments. The proposed method is able to detect acceptable and unacceptable ECG signals with an accuracy of 99.50%, recall of 99.78%, specificity of 99.60%, precision of 99.47%, and F-score of 0.999. Experimental results show that the proposed method performs better than earlier state-of-the-art techniques.

A comprehensive scheme for tattoo text detection

Tattoo text detection provides a vital clue for person and crime identification. Due to the freestyle and unconstrained nature of handwritten tattoo text over skin regions, accurate tattoo text detection is very challenging. This paper proposes a comprehensive scheme for tattoo text detection which comprises (a) adaptive Deformable Convolutional Neural Network (DCNN) for skin region detection to reduce text detection complexity (b) a Decoupled Gradient Text Detector (DGTD) for tattoo text detection from skin region (c) a Deep Q-Network (DQN) to refine the bounding boxes detected by DGTD, and (d) a Term-Frequency-Inverse-Document-Frequency (TF-IDF) model to group the words into text lines based on semantic information to fix the bounding box for the line. To test the effectiveness, the proposed method is evaluated on different datasets, namely, (i) a newly developed tattoo text dataset, (ii) benchmark bib number dataset of the marathon, and (iii) person re-identification dataset. The proposed method achieves 91.2, 87.5, and 88.8 F-scores from these three respective datasets. To demonstrate its superior performance, the text detection module (without skin detection) is also compared with state-of-the-art scene text detection methods on benchmark datasets, namely, ICDAR 2019 ArT, Total-Text, and DAST1500 and the proposed method achieves 90.3, 88.5 and 89.8 F-score from these respective datasets.

Forecasting large-scale circulation regimes using deformable convolutional neural networks and global spatiotemporal climate data

Classifying the state of the atmosphere into a finite number of large-scale circulation regimes is a popular way of investigating teleconnections, the predictability of severe weather events, and climate change. Here, we investigate a supervised machine learning approach based on deformable convolutional neural networks (deCNNs) and transfer learning to forecast the North Atlantic-European weather regimes during extended boreal winter for 1–15 days into the future. We apply state-of-the-art interpretation techniques from the machine learning literature to attribute particular regions of interest or potential teleconnections relevant for any given weather cluster prediction or regime transition. We demonstrate superior forecasting performance relative to several classical meteorological benchmarks, as well as logistic regression and random forests. Due to its wider field of view, we also observe deCNN achieving considerably better performance than regular convolutional neural networks at lead times beyond 5–6 days. Finally, we find transfer learning to be of paramount importance, similar to previous data-driven atmospheric forecasting studies.

Cerebral Arterial Stenosis Detection Based on a Retained Two-Stage Detection Algorithm

Stroke is one of the fatal diseases worldwide, and its primary mechanism is produced by cerebrovascular stenosis, blockages, or embolisms. Computer-aided diagnosis can assist clinical practitioners in identifying cerebrovascular anomalies, elucidating the precise lesions’ location in the patients, and providing guidance for clinical therapy. Due to different portions of the cerebrovascular possessing diverse morphological properties and the limited narrow area, the detection effect is unsatisfactory. A retrained two-stage algorithm for detecting cerebral arterial stenosis in CTA images is proposed to solve these problems by further fusing image features and improving the quality of regions of interest. In Faster R-CNN and Libra R-CNN, the backbone network was Resnet50, with deformable convolutional and nonlocal neural networks introduced in the third, fourth, and fifth stages of the backbone network. Deformable convolutional networks learned offsets to extract morphological features of blood vessels in different tomographic planes. Nonlocal neural networks fused global information and extracted global features from location information of feature maps. A cascade detector refined object classification and bounding box regression before prediction. The experimental results show that the retained algorithm increases mAP by 7.3% and 7.5%, respectively, compared with Faster R-CNN and Libra R-CNN. Deformable convolutional networks, nonlocal neural networks, and cascade detectors are incorporated into further feature fusion; thus, semantic information about the cerebrovascular structure is learned, demonstrating more accurate stenotic region detection and demonstrating generalizability across different two-stage algorithms.

DCU-net: a deformable convolutional neural network based on cascade U-net for retinal vessel segmentation

DCU-net: a deformable convolutional neural network based on cascade U-net for retinal vessel segmentation

Earthquake Crack Detection From Aerial Images Using a Deformable Convolutional Neural Network

Detecting the terrain surface cracks caused by earthquakes, which are termed coseismic ruptures, has important significance for discovering concealed faults, monitoring their movements, and forecasting possible follow-on earthquakes. On May 22, 2021, Maduo County in Qinghai province, China, suffered an earthquake with a magnitude of 7.4, which created densely distributed cracks. In this study, we designed an automatic crack detection framework based on remote sensing technology. With the use of low-altitude unmanned aerial vehicles (UAVs), we obtained very high-resolution aerial images of the area affected by the earthquake, which were further processed by photogrammetric software to produce digital orthophoto maps (DOMs). We then designed a novel terrain surface crack detection neural network, which differs from the previous methods that focus on detecting cracks in man-made object surfaces such as flat roads. We investigated the spatial property of the sinuous linear cracks and handled this by introducing adaptive deformable convolutions with a context-channel-space boosted mechanism. The feature extraction stage, feature optimization stage, and upsampling stage were embedded with the deformable convolutions to form a compact and powerful crack detector, named Crack-CADNet (the Context-chAnnel-space boosted Deformable convolutional neural network for crack detection). The postprocessing included filtering out the nontectonic cracks, aided by annotations from experts, and grouping and vectorizing the generated binary segmentation map as crack polygons, which were evaluated at the instance level. In addition to the first in-depth investigation of detecting earthquake cracks with aerial remote sensing and a deep learning based process, the crack detection network we propose outperformed the recent convolutional neural network (CNN)-based methods designed for general semantic segmentation and crack detection. Source code and the Maduo earthquake crack dataset will be available at http://gpcv.whu.edu.cn/data/.

A deformable CNN-based triplet model for fine-grained sketch-based image retrieval

With the popularity of electronic touch-screen and pressure sensing devices, fine-grained sketch based image retrieval (FG-SBIR) has become a research hotspot. In this paper, we stress the core problems of FG-SBIR: a. how to reduce the difference between the non-homogenous of heterogeneous media, and b. how to improve the distinguishability of sketch features. Specifically, a sketch generation model is first proposed to replace the conventional pre-processing of roughly extracting image edges, moreover, this model can alleviate the dilemma of sketch data scarcity. We then construct a novel FG-SBIR model which takes advantage of deformable convolutional neural network while taking into consideration of semantic attributes together. In addition, we build a fine-grained clothing sketch-image dataset, which has rich attribute annotations, for the first time. Extensive experiments exhibit that our proposed model achieves a better performance in improving the retrieval accuracy over the state-of-the-art baselines.

A Deformable Convolutional Neural Network with Spatial-Channel Attention for Remote Sensing Scene Classification

Remote sensing scene classification converts remote sensing images into classification information to support high-level applications, so it is a fundamental problem in the field of remote sensing. In recent years, many convolutional neural network (CNN)-based methods have achieved impressive results in remote sensing scene classification, but they have two problems in extracting remote sensing scene features: (1) fixed-shape convolutional kernels cannot effectively extract features from remote sensing scenes with complex shapes and diverse distributions; (2) the features extracted by CNN contain a large number of redundant and invalid information. To solve these problems, this paper constructs a deformable convolutional neural network to adapt the convolutional sampling positions to the shape of objects in the remote sensing scene. Meanwhile, the spatial and channel attention mechanisms are used to focus on the effective features while suppressing the invalid ones. The experimental results indicate that the proposed method is competitive to the state-of-the-art methods on three remote sensing scene classification datasets (UCM, NWPU, and AID).

A Performance Comparison and Enhancement of Animal Species Detection in Images with Various R-CNN Models

Object detection is one of the vital and challenging tasks of computer vision. It supports a wide range of applications in real life, such as surveillance, shipping, and medical diagnostics. Object detection techniques aim to detect objects of certain target classes in a given image and assign each object to a corresponding class label. These techniques proceed differently in network architecture, training strategy and optimization function. In this paper, we focus on animal species detection as an initial step to mitigate the negative impacts of wildlife–human and wildlife–vehicle encounters in remote wilderness regions and on highways. Our goal is to provide a summary of object detection techniques based on R-CNN models, and to enhance the performance of detecting animal species in accuracy and speed, by using four different R-CNN models and a deformable convolutional neural network. Each model is applied on three wildlife datasets, results are compared and analyzed by using four evaluation metrics. Based on the evaluation, an animal species detection system is proposed.

Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks

Scene classification of high-resolution remote sensing imagery based on deep transfer deformable convolutional neural networks

An End-to-End 12-Leading Electrocardiogram Diagnosis System Based on Deformable Convolutional Neural Network With Good Antinoise Ability

Electrocardiogram (ECG) is a tool to help judge heart activity. In recent years, the convolutional neural network (CNN) and various deep learning algorithms have been widely used in ECG diagnosis. CNN only considers the local feature. However, the ECG signal is susceptible to noise, and the waveform is complex, making it difficult for existing methods to get a good result. This article presents a novel neural network architecture for ECG diagnosis based on deformable CNN (Deform-CNN). The architecture makes good use of the feature-learning capability of deformable convolution to learn the time-domain and lead characteristics of multilead ECG signals. The proposed end-to-end method can achieve an overall diagnostic accuracy of 86.3% in the 12-lead ECG data of CPSC-2018, with good antinoise ability, which makes the method have a more competitive performance than other deep learning algorithms. The source code is publicly available at https://github.com/HeartbeatAI/Deform-CNN .