Local Receptive Fields Research Articles

ScribFormer: Transformer Makes CNN Work Better for Scribble-Based Medical Image Segmentation.

Most recent scribble-supervised segmentation methods commonly adopt a CNN framework with an encoder-decoder architecture. Despite its multiple benefits, this framework generally can only capture small-range feature dependency for the convolutional layer with the local receptive field, which makes it difficult to learn global shape information from the limited information provided by scribble annotations. To address this issue, this paper proposes a new CNN-Transformer hybrid solution for scribble-supervised medical image segmentation called ScribFormer. The proposed ScribFormer model has a triple-branch structure, i.e., the hybrid of a CNN branch, a Transformer branch, and an attention-guided class activation map (ACAM) branch. Specifically, the CNN branch collaborates with the Transformer branch to fuse the local features learned from CNN with the global representations obtained from Transformer, which can effectively overcome limitations of existing scribble-supervised segmentation methods. Furthermore, the ACAM branch assists in unifying the shallow convolution features and the deep convolution features to improve model's performance further. Extensive experiments on two public datasets and one private dataset show that our ScribFormer has superior performance over the state-of-the-art scribble-supervised segmentation methods, and achieves even better results than the fully-supervised segmentation methods. The code is released at https://github.com/HUANGLIZI/ScribFormer.

Transformer-CNN hybrid network for improving PET time of flight prediction

Objective. In positron emission tomography (PET) reconstruction, the integration of time-of-flight (TOF) information, known as TOF-PET, has been a major research focus. Compared to traditional reconstruction methods, the introduction of TOF enhances the signal-to-noise ratio of images. Precision in TOF is measured by full width at half maximum (FWHM) and the offset from ground truth, referred to as coincidence time resolution (CTR) and bias. Approach. This study proposes a network combining transformer and convolutional neural network (CNN) to utilize TOF information from detector waveforms, using event waveform pairs as inputs. This approach integrates the global self-attention mechanism of Transformer, which focuses on temporal relationships, with the local receptive field of CNN. The combination of global and local information allows the network to assign greater weight to the rising edges of waveforms, thereby extracting valuable temporal information for precise TOF predictions. Experiments were conducted using lutetium yttrium oxyorthosilicate (LYSO) scintillators and silicon photomultiplier (SiPM) detectors. The network was trained and tested using the waveform datasets after cropping. Main results. Compared to the constant fraction discriminator (CFD), CNN, CNN with attention, long short-term memory (LSTM) and Transformer, our network achieved an average CTR of 189 ps, reducing it by 82 ps (more than 30%), 13 ps (6.4%), 12 ps (6.0%), 16 ps (7.8%) and 9 ps (4.6%), respectively. Additionally, a reduction of 10.3, 8.7, 6.7 and 4 ps in average bias was achieved compared to CNN, CNN with attention, LSTM and Transformer. Significance. This work demonstrates the potential of applying the Transformer for PET TOF estimation using real experimental data. Through the integration of both CNN and Transformer with local and global attention, it achieves optimal performance, thereby presenting a novel direction for future research in this field.

Internet of Things intrusion detection: Research and practice of NSENet and LSTM fusion models

To address the problems of complex environment, limited device computational resources and limited memory resources in the existing IoT, SELSTM, an intrusion detection system composed of NSENet and LSTM fusion based on SENet, is investigated. The NSENet part of the SELSTM system is based on the squeeze-and-excitation network (SENet). The lightweight computational modules NonLocal, SKConv and inverted residuals are fused into SE blocks, and self-attention of Nonlocal is used to improve the local receptive field of feature extraction. The channel attention and spatial attention of each part of the data are strengthened by the use of SKConv To enhance the adaptive convolution ability of the model and ensure the completeness of the information, the properties of the inverted residual structure are used to ensure that the gradient of the model decreases steadily without gradient explosion or disappearance. For the problem of data imbalance, the dataset is randomly resampled using the weight resampling technique to improve the balance of the dataset to ensure that the final detection effect of the model is more effective and generalized, while the data flow is divided into two parts for processing, and the model parameters are optimized using the model gradient optimizer consisting of the optimizer Lion and the optimization function Lookahead. The model extracts the spatial and temporal features of the data through multidimensional extraction to ensure the completeness of the data feature information in multiple dimensions, thus obtaining better detection results. The results of the experiments comparing the SELSTM model with other models on the intrusion dataset show that the intrusion detection model has a higher detection precision and accuracy than the traditional deep learning intrusion detection model, which indicates that the SELSTM has better detection performance properties and better practicality and effectiveness on IoT devices.

A transformer-guided cross-modality adaptive feature fusion framework for esophageal gross tumor volume segmentation

Background and ObjectiveAccurate segmentation of esophageal gross tumor volume (GTV) indirectly enhances the efficacy of radiotherapy for patients with esophagus cancer. In this domain, learning-based methods have been employed to fuse cross-modality positron emission tomography (PET) and computed tomography (CT) images, aiming to improve segmentation accuracy. This fusion is essential as it combines functional metabolic information from PET with anatomical information from CT, providing complementary information. While the existing three-dimensional (3D) segmentation method has achieved state-of-the-art (SOTA) performance, it typically relies on pure-convolution architectures, limiting its ability to capture long-range spatial dependencies due to convolution's confinement to a local receptive field. To address this limitation and further enhance esophageal GTV segmentation performance, this work proposes a transformer-guided cross-modality adaptive feature fusion network, referred to as TransAttPSNN, which is based on cross-modality PET/CT scans. MethodsSpecifically, we establish an attention progressive semantically-nested network (AttPSNN) by incorporating the convolutional attention mechanism into the progressive semantically-nested network (PSNN). Subsequently, we devise a plug-and-play transformer-guided cross-modality adaptive feature fusion model, which is inserted between the multi-scale feature counterparts of a two-stream AttPSNN backbone (one for the PET modality flow and another for the CT modality flow), resulting in the proposed TransAttPSNN architecture. ResultsThrough extensive four-fold cross-validation experiments on the clinical PET/CT cohort. The proposed approach acquires a Dice similarity coefficient (DSC) of 0.76 ± 0.13, a Hausdorff distance (HD) of 9.38 ± 8.76 mm, and a Mean surface distance (MSD) of 1.13 ± 0.94 mm, outperforming the SOTA competing methods. The qualitative results show a satisfying consistency with the lesion areas. ConclusionsThe devised transformer-guided cross-modality adaptive feature fusion module integrates the strengths of PET and CT, effectively enhancing the segmentation performance of esophageal GTV. The proposed TransAttPSNN has further advanced the research of esophageal GTV segmentation.

Large-Scale Data-Driven Optimization in Deep Modeling With an Intelligent Decision-Making Mechanism.

This study focuses on building an intelligent decision-making attention mechanism in which the channel relationship and conduct feature maps among specific deep Dense ConvNet blocks are connected to each other. Thus, develop a novel freezing network with a pyramid spatial channel attention mechanism (FPSC-Net) in deep modeling. This model studies how specific design choices in the large-scale data-driven optimization and creation process affect the balance between the accuracy and effectiveness of the designed deep intelligent model. To this end, this study presents a novel architecture unit, which is termed as the "Activate-and-Freeze" block on popular and highly competitive datasets. In order to extract informative features by fusing spatial and channel-wise information together within local receptive fields and boost the representation power, this study constructs a Dense-attention module (pyramid spatial channel (PSC) attention) to perform feature recalibration, and through the PSC attention to model the interdependence among convolution feature channels. We join the PSC attention module in the activating and back-freezing strategy to search for one of the most important parts of the network for extraction and optimization. Experiments on various large-scale datasets demonstrate that the proposed method can achieve substantially better performance for improving the ConvNets representation power than the other state-of-the-art deep models.

EdiTor: Edge-guided Transformer for Ghost-free High Dynamic Range Imaging

Synthesizing the high dynamic range (HDR) image from multi-exposure images has been extensively studied by exploiting convolutional neural networks (CNNs) recently. Despite the remarkable progress, existing CNN-based methods have the intrinsic limitation of local receptive field, which hinders the model’s capability of capturing long-range correspondence and large motions across under/over-exposure images, resulting in ghosting artifacts of dynamic scenes. To address the above challenge, we propose a novel Ed ge-gu i ded T ransf or mer framework (EdiTor) customized for ghost-free HDR reconstruction, where the long-range motions across different exposures can be delicately modeled by incorporating the edge prior. Specifically, EdiTor calculates patch-wise correlation maps on both image and edge domains, enabling the network to effectively model the global movements and the fine-grained shifts across multiple exposures. Based on this framework, we further propose an exposure-masked loss to adaptively compensate for the severely distorted regions ( e.g. , highlights and shadows). Experiments demonstrate that EdiTor outperforms state-of-the-art methods both quantitatively and qualitatively, achieving appealing HDR visualization with unified textures and colors.

Brain tumor segmentation using neuro-technology enabled intelligence-cascaded U-Net model.

According to experts in neurology, brain tumours pose a serious risk to human health. The clinical identification and treatment of brain tumours rely heavily on accurate segmentation. The varied sizes, forms, and locations of brain tumours make accurate automated segmentation a formidable obstacle in the field of neuroscience. U-Net, with its computational intelligence and concise design, has lately been the go-to model for fixing medical picture segmentation issues. Problems with restricted local receptive fields, lost spatial information, and inadequate contextual information are still plaguing artificial intelligence. A convolutional neural network (CNN) and a Mel-spectrogram are the basis of this cough recognition technique. First, we combine the voice in a variety of intricate settings and improve the audio data. After that, we preprocess the data to make sure its length is consistent and create a Mel-spectrogram out of it. A novel model for brain tumor segmentation (BTS), Intelligence Cascade U-Net (ICU-Net), is proposed to address these issues. It is built on dynamic convolution and uses a non-local attention mechanism. In order to reconstruct more detailed spatial information on brain tumours, the principal design is a two-stage cascade of 3DU-Net. The paper's objective is to identify the best learnable parameters that will maximize the likelihood of the data. After the network's ability to gather long-distance dependencies for AI, Expectation-Maximization is applied to the cascade network's lateral connections, enabling it to leverage contextual data more effectively. Lastly, to enhance the network's ability to capture local characteristics, dynamic convolutions with local adaptive capabilities are used in place of the cascade network's standard convolutions. We compared our results to those of other typical methods and ran extensive testing utilising the publicly available BraTS 2019/2020 datasets. The suggested method performs well on tasks involving BTS, according to the experimental data. The Dice scores for tumor core (TC), complete tumor, and enhanced tumor segmentation BraTS 2019/2020 validation sets are 0.897/0.903, 0.826/0.828, and 0.781/0.786, respectively, indicating high performance in BTS.

CSDFormer: A cloud and shadow detection method for landsat images based on transformer

Cloud and shadow (CS) detection is crucial prerequisite for application of remote sensing images. Current deep learning-based detection algorithms mainly employ Convolutional Neural Networks (CNNs). However, the local receptive field in CNNs cannot effectively capture global contextual information, which hinders accurate characterization of the dependency between clouds and shadows. In vision Transformers, self-attention mechanisms can effectively capture the long-distance dependencies between different regions in an image. Inspired by this, this paper proposed a new CS Detection algorithm based on a Transformer, called CSDFormer. Specifically, we exclusively employed a hierarchical Transformer structure in the encoder stage to extract features of CS. Each Transformer layer contains several multi-head self-attention mechanisms for calculating pixel-wise long-distance connectivity. The designed structure enables the Transformer to better extract global context information, which helps to strengthen the comprehension of the semantic relationships between clouds and shadows. Benefiting from the global feature extraction capability of the encoder stage, we employed several simple multilayer perceptron layers for multi-scale feature map fusion and pixel classification in the decoder stage. The proposed CSDFormer was validated using 898 Landsat 8 Biome images with 512 × 512 pixels, producing an overall accuracy of 95.28 % and a mean intersection over union of 84.08 %, outperforming three state-of-the-art CNN-based algorithms. CSDFormer is consistently more accurate in detection of both clouds and shadows. Owing to the parallel computing capability of the self-attention mechanism, CSDFormer is computationally more efficient than the three CNN-based benchmark methods. For the input spectral bands, the performance of CSDFormer produced can be further enhanced with additional thermal infrared bands.

DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract)

With the help of Vision transformers (ViTs), medical image segmentation was able to achieve outstanding performance. In particular, they overcome the limitation of convolutional neural networks (CNNs) which rely on local receptive fields. ViTs use self-attention mechanisms to consider relationships between all image pixels or patches simultaneously. However, they require large datasets for training and did not perform well on capturing low-level features. To that end, we propose DDViT, a novel ViT model that unites a CNN to alleviate data-hunger for medical image segmentation with two multi-scale feature representations. Significantly, our approach incorporates a ViT with a plug-in domain adapter (DA) with Double-Level Fusion (DLF) technique, complemented by a mutual knowledge distillation paradigm, facilitating the seamless exchange of knowledge between a universal network and specialized domain-specific network branches. The DLF framework plays a pivotal role in our encoder-decoder architecture, combining the innovation of the TransFuse module with a robust CNN-based encoder. Extensive experimentation across diverse medical image segmentation datasets underscores the remarkable efficacy of DDViT when compared to alternative approaches based on CNNs and Transformer-based models.

Column Row Convolutional Neural Network: Reducing Parameters for Efficient Image Processing.

Recent advancements in deep learning have achieved significant progress by increasing the number of parameters in a given model. However, this comes at the cost of computing resources, prompting researchers to explore model compression techniques that reduce the number of parameters while maintaining or even improving performance. Convolutional neural networks (CNN) have been recognized as more efficient and effective than fully connected (FC) networks. We propose a column row convolutional neural network (CRCNN) in this letter that applies 1D convolution to image data, significantly reducing the number of learning parameters and operational steps. The CRCNN uses column and row local receptive fields to perform data abstraction, concatenating each direction's feature before connecting it to an FC layer. Experimental results demonstrate that the CRCNN maintains comparable accuracy while reducing the number of parameters and compared to prior work. Moreover, the CRCNN is employed for one-class anomaly detection, demonstrating its feasibility for various applications.

CTF-former: A novel simplified multi-task learning strategy for simultaneous multivariate chaotic time series prediction

Multivariate chaotic time series prediction is a challenging task, especially when multiple variables are predicted simultaneously. For multiple related prediction tasks typically require multiple models, however, multiple models are difficult to keep synchronization, making immediate communication between predicted values challenging. Although multi-task learning can be applied to this problem, the principles of allocation and layout options between shared and specific representations are ambiguous. To address this issue, a novel simplified multi-task learning method was proposed for the precise implementation of simultaneous multiple chaotic time series prediction tasks. The scheme proposed consists of a cross-convolution operator designed to capture variable correlations and sequence correlations, and an attention module proposed to capture the information embedded in the sequence structure. In the attention module, a non-linear transformation was implemented with convolution, and its local receptive field and the global dependency of the attention mechanism achieve complementarity. In addition, an attention weight calculation was devised that takes into account not only the synergy of time and frequency domain features, but also the fusion of series and channel information. Notably the scheme proposed a purely simplified design principle of multi-task learning by reducing the specific network to single neuron. The precision of the proposed solution and its potential for engineering applications were verified with the Lorenz system and power consumption. The mean absolute error of the proposed method was reduced by an average of 82.9% in the Lorenz system and 19.83% in power consumption compared to the Gated Recurrent Unit.

OASL: Orientation-aware adaptive sampling learning for arbitrary oriented object detection

Arbitrary oriented object detection (AOOD) is a fundamental task in aeiral image interpretation, which is commonly implemented by optimizing three subtasks: classification, localization, and orientation. The consistency of classification, localization, and orientation is crucial for achieving high performance of AOOD detectors. However, independent prediction between subtasks and differences in quality metrics for task-specific samples reduces the accuracy of high-precision detection. In this paper, we propose a novel dense anchor-free detector Orientation-aware Adaptive Sampling Learning (OASL) that includes two core modules: Orientation-aware Structured Information Extraction Module (O-SIEM) and Weights adaptive Sample Information Richness Metric (W-SIRM). Specifically, O-SIEM applies the global and local receptive fields of objects to explicitly extract spatial contextual features of oriented and densely distributed objects, which helps to strike a balance between capturing task-interactive and task-specific features, ultimately leading to learning common patterns between subtasks. W-SIRM integrates information from three subtasks and achieves dynamic evaluation of sample quality and adaptive adjustment of sample weights based on the information richness of samples. Extensive experiments demonstrate that the proposed OASL method can effectively handle the consistency problem and enhance the validity of the sample quality metric.

MRFormer: Multiscale retractable transformer for medical image progressive denoising via noise level estimation

Clear medical images are important for auxiliary diagnoses, but the images generated by various medical devices inevitably contain considerable noise. Although various models have been proposed for denoising, these methods ignore the fact that different types of medical images have different noise levels, which leads to unsatisfactory test results. In addition, collecting many medical images for training denoising models consumes many material resources. To address these issues, we formulate a progressive denoising architecture that contains preliminary and profound denoising. First, we construct a noise level estimation network to estimate the noise level via self-supervised learning and perform preliminary denoising with a dilated blind-spot network. Second, with the learned noise distribution, we synthesize noisy natural images to construct clean-noisy natural image pairs. Finally, we design a novel medical image denoising model for profound denoising by training these pairs. The proposed three-stage learning scheme and progressive denoising architecture not only solve the problem that the denoising model only adapts to a single noise level but also alleviate the lack of medical image pairs. Moreover, we integrate dense attention and sparse attention to constitute the retractable transformer module in the profound denoising model, which reconciles a wider receptive field and enhances the representation ability of the transformer, s allowing the denoising model to obtain retractable attention on the input feature and capture more local and global receptive fields simultaneously. The results of qualitative and quantitative experiments demonstrate the effectiveness of our method in removing noise at various levels.

FAM: Improving columnar vision transformer with feature attention mechanism

Vision Transformer has garnered outstanding performance in visual tasks due to its capability for global modeling of image information. However, during the self-attention computation of image tokens, a common issue of attention map homogenization arises, impacting the final performance of the model as attention maps propagate through feature maps layer by layer. In this research, we propose a token-based approach to adjust the output of attention sub-layer, focusing on the feature dimensions, to address the homogenization problem. Furthermore, different network architectures exhibit variations in their approaches to modeling image features. Specifically, Vision Transformers excel at modeling long-range relationships, while convolutional neural networks possess local receptive fields. Therefore, this paper introduces a plug-and-play convolutional operator-based component, integrated into the Vision Transformer, to validate the impact of structural enhancements on model performance. Experimental results on image recognition and adversarial attack tasks respectively demonstrate the effectiveness and robustness of the two proposed methods. Additionally, the analysis of information entropy on the feature maps of the model’s final layer indicates that the improved model exhibits higher information richness, making it more conducive to the classifier’s discriminative capabilities.

Hierarchical damage correlations for old photo restoration

Restoring old photographs can preserve cherished memories. Previous methods handled diverse damages within the same network structure, which proved impractical. In addition, these methods cannot exploit correlations among artifacts, especially in scratches versus patch-misses issues. Hence, a tailored network is particularly crucial. In light of this, we propose a unified framework consisting of two key components: ScratchNet and PatchNet. In detail, ScratchNet employs the parallel Multi-scale Partial Convolution Module to effectively repair scratches, learning from multi-scale local receptive fields. In contrast, the patch-misses necessitate the network to emphasize global information. To this end, we incorporate a transformer-based encoder and decoder architecture. In the encoder phase, we introduce a Non-local Inpainting Attention Module, replacing the multi-head attention, to facilitate holistic context inpainting. In the decoder phase, the Mask-aware Instance Norm Module replaces the Layer Normalization, ensuring style consistency between foreground and background. Finally, the outcomes of ScratchNet are integrated into the PatchNet pipeline to supplement contextual information hierarchically. Mining damage correlations assists in training the network in an easy-to-hard manner. Extensive experiments demonstrate the superiority of our method over state-of-the-art approaches. The code is available at https://github.com/cwyyt/Hierarchical-Damage-Correlations-for-OldPhoto-Restoration.

Real time Semantic Segmentation for Human-Labeled Data: A comparative Study Between CNN and Transformer

Convolutional Neural Networks have rapidly developed in the field of computer vision, notably in image classification, semantic segmentation, and object detection. These networks efficiently extract image features through local receptive fields and shared weights. Despite their effectiveness in various applications, CNNs face limitations, such as challenges in managing large-scale parameters and a tendency to overfit, especially in complex scenarios requiring contextual understanding. On the other hand, Transformer-based models, originally designed for natural language processing, have recently gained prominence in computer vision. They are particularly adept at capturing long-range dependencies, a critical aspect for interpreting complex visual scenes. Their scalability and adaptability open up new avenues for innovation. However, these models also come with drawbacks, including a need for extensive training data and higher computational costs. Their complex structures make optimization particularly challenging in resource-constrained environments. In this paper, our focus is on real-time model comparisons between CNN and Transformer architectures. We represent CNNs with the STDC model and Transformer-based models with the SegFormer. Our analysis revealed that the STDC model significantly outperforms in inference speed, achieving around 97 frames per second (fps), which is notably faster than the 50 fps of the SegFormer B0. However, when it comes to accuracy, the SegFormer B0 demonstrates superiority with a mean Intersection over Union (mIoU) of 86.78, which is more favorable compared to the 82.9 mIoU of the STDC. This study underscores the efficiency and accuracy trade-offs between these two architectures, highlighting the strengths of both CNNs and Transformer-based models in real-time applications.

ST-GAN: A Swin Transformer-Based Generative Adversarial Network for Unsupervised Domain Adaptation of Cross-Modality Cardiac Segmentation.

Unsupervised domain adaptation (UDA) methods have shown great potential in cross-modality medical image segmentation tasks, where target domain labels are unavailable. However, the domain shift among different image modalities remains challenging, because the conventional UDA methods are based on convolutional neural networks (CNNs), which tend to focus on the texture of images and cannot establish the global semantic relevance of features due to the locality of CNNs. This paper proposes a novel end-to-end Swin Transformer-based generative adversarial network (ST-GAN) for cross-modality cardiac segmentation. In the generator of ST-GAN, we utilize the local receptive fields of CNNs to capture spatial information and introduce the Swin Transformer to extract global semantic information, which enables the generator to better extract the domain-invariant features in UDA tasks. In addition, we design a multi-scale feature fuser to sufficiently fuse the features acquired at different stages and improve the robustness of the UDA network. We extensively evaluated our method with two cross-modality cardiac segmentation tasks on the MS-CMR 2019 dataset and the M&Ms dataset. The results of two different tasks show the validity of ST-GAN compared with the state-of-the-art cross-modality cardiac image segmentation methods.

Base on temporal convolution and spatial convolution transformer for fluid prediction through well logging data

Fluid prediction is important in exploration work, helping to determine the location of exploration targets and the reserve potential of the estimated area. Machine learning methods can better adapt to different data distributions and nonlinear relationships through model training, resulting in better learning of these complex relationships. We started by using the convolution operation to process the log data, which includes temporal convolution and spatial convolution. Temporal convolution is specifically designed to capture time series relationships in time series data. In well log data, time information is often critical for understanding fluid changes and other important details. Temporal convolution learns trends and cyclical changes in the data. The spatial convolution operation makes the model more sensitive to the local features in the logging data through the design of the local receptive field and improves the sensitivity to fluid changes. Spatial convolution helps capture spatial correlations at different depths or locations. This can help the model understand the change of fluid in the vertical direction and identify the spatial relationship between different fluids. Then, we use the transformer module to predict the fluid. The transformer module uses a self-attention mechanism that allows the model to focus on information with different weights at different locations in the sequence. In the well log data, this helps the model to better capture the formation characteristics at different depths or time points and improves the modeling ability of time series information. The fully connected structure in the transformer module enables each position to interact directly with other locations in the sequence. By applying it to the data of Tarim Oilfield, the experimental results show that the convolutional transformer model proposed in this paper has better results than other machine learning models. This study provides a new idea in the field of logging fluid prediction.

Deepfake Detection Using Spatiotemporal Transformer

Recent advances in generative models and the availability of large-scale benchmarks have made deepfake video generation and manipulation easier. Nowadays, the number of new hyper-realistic deepfake videos used for negative purposes is dramatically increasing, thus creating the need for effective deepfake detection methods. Although many existing deepfake detection approaches, particularly CNN-based methods, show promising results, they suffer from several drawbacks. In general, poor generalization results have been obtained under unseen/new deepfake generation methods. The crucial reason for the above defect is that CNN-based methods focus on the local spatial artifacts, which are unique for every manipulation method. Therefore, it is hard to learn the general forgery traces of different manipulation methods without considering the dependencies that extend beyond the local receptive field. To address this problem, this paper proposes a framework that combines aper proposes a framework that combines with Vision Transformer (ViT) to improve detection accuracy and enhance generalizability. Our method, named HCiT , exploits the advantages of CNNs to extract meaningful local features, as well as the VIT’s self-attention mechanism to learn discriminative global contextual dependencies in a frame-level image explicitly. In this hybrid architecture, the high-level feature maps extracted from the CNN are fed into the ViT model that determines whether a specific video is fake or real. Experiments were performed on Faceforensics++, DeepFake Detection Challenge preview, Celeb datasets, and the results show that the proposed method significantly outperforms the state-of-the-art methods. In addition, the HCiT method shows a great capacity for generalization on datasets covering various techniques of deepfake generation. The source code is available at: https://github.com/KADDAR-Bachir/HCiT

Refined composite multiscale dynamic causality diagram: a novel time-series analysis approach and its application in fault diagnosis of rotating machinery

Entropy-based feature extraction methods have been widely used in the fault diagnosis of rotating machinery, but the entropy-based methods still have the defects of poor noise robustness, weak feature extraction, and low computational efficiency. To solve this problem, this article proposes a fault diagnosis method based on refined composite multiscale dynamic causal diagram (RCMSDCD) and local receptive field extreme learning machine (LRFELM). First, a novel feature extraction method, named dynamic causal diagram (DCD), is proposed to comprehensively quantify static and dynamic complexity. DCD is obtained by combining generalized inverse fractional order entropy with complexity–entropy causal plane. Then, combined with the coarse-graining process, DCD is extended to a multiscale analysis called RCMSDCD to complement the feature description at cross-time scales. Third, RCMSDCD features are input into LRFELM classifier for fault recognition of rotating machinery. The effectiveness of the proposed RCMSDCD-LRFELM method is verified by the Paderborn University bearing test and real wind turbine gear signals. The results show that this method has the highest classification accuracy of 100% with high computational efficiency, good stability, and strong generalization ability.