Depthwise Separable Convolution Research Articles

Automatic assessment of freeze-thaw damage in concrete structures using piezoelectric-based active sensing approach and deep learning technique

Concrete structures in cold climates are susceptible to freeze-thaw (f-t) damage, which could lead to structural deterioration and stability issues. Accurate assessment of such damage is crucial for effective maintenance and repair strategies. However, current methods for evaluating concrete f-t damage rely on tedious testing procedures that lack automation and real-time capabilities, potentially delaying the repair process. This study proposes an automatic method to assess f-t damage in concrete by combining piezoelectric-based active sensing and deep learning (DL). Firstly, concrete beams with two different force states were prepared and subjected to different f-t cycles, during which measurements including surface characteristics, mass loss, dynamic elastic modulus and stress wave were performed. The relationship between concrete f-t damage evolution and stress wave signal variation was investigated. The collected signals were then converted into time-frequency maps using continuous wavelet transform (CWT), thereby establishing a CWT image-based dataset. More importantly, an innovative DL model (DSC-ACGRU) that integrates depth-wise separable convolution (DSC), convolutional gated recurrent unit (ConvGRU) and attention mechanism (AM) was developed to automatically extract damage-sensitive feature from CWT images and ultimately predict the degree of concrete f-t damage. Finally, the performances of the proposed DSC-ACGRU model were experimentally evaluated and compared with five machine/deep learning methods. The results show that the proposed model can rapidly and accurately assess the f-t damage degree in concrete and outperforms other learning algorithms, which is significant for identifying potential hazards and repairing damaged concrete.

NVS-GAN: Benefit of generative adversarial network on novel view synthesis

The methodology to generate new views for an object from provided input object view is called Novel View Synthesis (NVS). Humans imagine novel views through prior knowledge gathered through their lifetime. NVS-GAN predicts the novel views through computation. Literature survey reveals that there are limited NVS models with low Trainable Parameter Count (TPC) and low model size. Also, a study on the effect of different loss functions on NVS models was lacking. Lowering the TPC indicates less computational steps for the model to predict the output, therefore desirable. Combined with a low model size, the proposed model will become more suitable for deployment in diverse devices having limited resources for computation. Application of right combination of loss functions yield better accuracy. To address these research gaps, NVS-GAN is proposed. NVS-GAN is a Generative Adversarial Network (GAN) approach which yields NVS-Generator which performs NVS. NVS-Generator incorporates identity skip connections, bilinear sampling module, Depthwise Separable Convolution (DSC) as design features and results in low TPC, model size. In addition to discriminator loss, NVS-GAN is trained with different combinations of loss functions i.e. Mean Absolute Error (MAE) loss, Structural Similarity Index Measure (SSIM) loss, Huber loss on chair and car objects of ShapeNet dataset. The performance of NVS-Generator on test set measured in terms of MAE and SSIM is tabulated and analysed. The performance is compared with existing NVS models. The proposed NVS-GAN experiment recorded reduction in NVS-Generator TPC in 37 %–54.6 % range and reduction in model size between 37.2 % and 47.6 % range. NVS-Generator reduced MAE upto 55 % and improved SSIM upto 4 % than existing models. Summarily, NVS-GAN increased model performance and made the model “lightweight”.

Large Model for Rotating Machine Fault Diagnosis Based on a Dense Connection Network With Depthwise Separable Convolution

Large Model for Rotating Machine Fault Diagnosis Based on a Dense Connection Network With Depthwise Separable Convolution

Improving Video Vision Transformer for Deepfake Video Detection Using Facial Landmark, Depthwise Separable Convolution and Self Attention

Improving Video Vision Transformer for Deepfake Video Detection Using Facial Landmark, Depthwise Separable Convolution and Self Attention

EISATC-Fusion: Inception Self-Attention Temporal Convolutional Network Fusion for Motor Imagery EEG Decoding.

The motor imagery brain-computer interface (MI-BCI) based on electroencephalography (EEG) is a widely used human-machine interface paradigm. However, due to the non-stationarity and individual differences among subjects in EEG signals, the decoding accuracy is limited, affecting the application of the MI-BCI. In this paper, we propose the EISATC-Fusion model for MI EEG decoding, consisting of inception block, multi-head self-attention (MSA), temporal convolutional network (TCN), and layer fusion. Specifically, we design a DS Inception block to extract multi-scale frequency band information. And design a new cnnCosMSA module based on CNN and cos attention to solve the attention collapse and improve the interpretability of the model. The TCN module is improved by the depthwise separable convolution to reduces the parameters of the model. The layer fusion consists of feature fusion and decision fusion, fully utilizing the features output by the model and enhances the robustness of the model. We improve the two-stage training strategy for model training. Early stopping is used to prevent model overfitting, and the accuracy and loss of the validation set are used as indicators for early stopping. The proposed model achieves within-subject classification accuracies of 84.57% and 87.58% on BCI Competition IV Datasets 2a and 2b, respectively. And the model achieves cross-subject classification accuracies of 67.42% and 71.23% (by transfer learning) when training the model with two sessions and one session of Dataset 2a, respectively. The interpretability of the model is demonstrated through weight visualization method.

LHAR: Lightweight Human Activity Recognition on Knowledge Distillation.

Sensor-based Human Activity Recognition (HAR) is widely used in daily life and is the basic-level bridge to virtual healthcare in the metaverse. The current challenge is the low recognition accuracy for personalized users on smart wearable devices. The limited resource cannot support large deep learning models updated locally. Besides, integrating and transmitting sensor data to the cloud would reduce the efficiency. Considering the tradeoff between performance and complexity, we propose a Lightweight Human Activity Recognition (LHAR) framework. In LHAR, we combine the cross-people HAR task with the lightweight model task. LHAR framework is designed on the teacher-student architecture and the student network consists of multiple depthwise separable convolution layers to achieve fewer parameters. The dark knowledge distilled from the complex teacher model enhances the generalization ability of LHAR. To achieve effective knowledge distillation, we propose two optimization methods. Firstly, we train the teacher model by ensemble learning to promote teacher performance. Secondly, a multi-channel data augmentation method is proposed for the diversity of the dataset, which is a plug-in operation for the ensemble teacher model. In the experiments, we compare LHAR with state-of-art models in comparison evaluation, ablation study and the hyperparameter analysis, which proves the better performance of LHAR in efficiency and effectiveness.

FPGA based Flexible Implementation of Light Weight Inference on Deep Convolutional Neural Networks

Standard Convolution (StdConv) is the main technique used in the state of the art Deep Convolutional Neural Networks (DCNNs). Fewer computations are achieved if Depthwise Separable Convolution technique (SepConv) is used as an alternative. A crucial issue in many applications like smart cameras and autonomous vehicles where low latency is essential stems from deploying a lightweight and low cost inference models. An acceptable accuracy should be kept with tolerable computations and memory access load. A flexible architecture for different DCNN convolution types and models is proposed. The flexibility comes from the sharing of one memory access unit with different types of layers regardless of the selected kernel size, by multiplying each weight vector by local operators with variant aperture. Moreover, one depthwise computation unit can be used for both standard and pointwise layers. The learnable parameters are quantized to 8-bits fixed point representation and that gives very limited reduction of accuracy and a considerable reduction of the Field-Programmable Gate Array (FPGA) resources. To reduce processing time, inter layer parallel computations are performed. The experiment is conducted by using grey scale ORL database with shallow Convolutional Neural Network (CNN) and the colored Canadian Institute for Advanced Research 10 classes (CIFAR-10) database with DCNN, and a comparable accuracies of 93% and 85.7% are achieved respectively using very low cost of Spartan 3E and moderate cost of zynq FPGA platforms

A lightweight rolling bearing fault diagnosis method based on multiscale Depth-wise Separable Convolutions and network pruning

A lightweight rolling bearing fault diagnosis method based on multiscale Depth-wise Separable Convolutions and network pruning

An FPGA-Based Approach for Compressing and Accelerating Depthwise Separable Convolution

An FPGA-Based Approach for Compressing and Accelerating Depthwise Separable Convolution

Multiscale Spatial-Temporal Feature Fusion Neural Network for Motor Imagery Brain-Computer Interfaces.

Motor imagery, one of the main brain-computer interface (BCI) paradigms, has been extensively utilized in numerous BCI applications, such as the interaction between disabled people and external devices. Precise decoding, one of the most significant aspects of realizing efficient and stable interaction, has received a great deal of intensive research. However, the current decoding methods based on deep learning are still dominated by single-scale serial convolution, which leads to insufficient extraction of abundant information from motor imagery signals. To overcome such challenges, we propose a new end-to-end convolutional neural network based on multiscale spatial-temporal feature fusion (MSTFNet) for EEG classification of motor imagery. The architecture of MSTFNet consists of four distinct modules: feature enhancement module, multiscale temporal feature extraction module, spatial feature extraction module and feature fusion module, with the latter being further divided into the depthwise separable convolution block and efficient channel attention block. Moreover, we implement a straightforward yet potent data augmentation strategy to bolster the performance of MSTFNet significantly. To validate the performance of MSTFNet, we conduct cross-session experiments and leave-one-subject-out experiments. The cross-session experiment is conducted across two public datasets and one laboratory dataset. On the public datasets of BCI Competition IV 2a and BCI Competition IV 2b, MSTFNet achieves classification accuracies of 83.62% and 89.26%, respectively. On the laboratory dataset, MSTFNet achieves 86.68% classification accuracy. Besides, the leave-one-subject-out experiment is performed on the BCI Competition IV 2a dataset, and MSTFNet achieves 66.31% classification accuracy. These experimental results outperform several state-of-the-art methodologies, indicate the proposed MSTFNet's robust capability in decoding EEG signals associated with motor imagery.

Autonomous Identification of Bridge Concrete Cracks Using Unmanned Aircraft Images and Improved Lightweight Deep Convolutional Networks

The identification of the development of structural defects is an important part of bridge structure damage diagnosis, and cracks are considered the most typical and highly dangerous structural disease. However, existing deep learning‐based methods are mostly aimed at the scene of concrete cracks, while they rarely focus on designing network architectures to improve the vision‐based model performance from the perspective of unmanned aircraft system (UAS) inspection, which leads to a lack of specificity. Because of this, this study proposes a novel lightweight deep convolutional neural network‐based crack pixel‐level segmentation network for UAS‐based inspection scenes. Firstly, the classical encoder‐decoder architecture UNET is utilized as the base model for bridge structural crack identification, and the hourglass‐shaped depthwise separable convolution is introduced to replace the traditional convolutional operation in the UNET model to reduce model parameters. Then, a kind of lightweight and efficient channel attention module is used to improve model feature fuzzy ability and segmentation accuracy. We conducted a series of experiments on bridge structural crack detection tasks by utilizing a long‐span bridge as the research item. The experimental results show that the constructed method achieves an effective balance between reasoning accuracy and efficiency with the value of 97.62% precision, 97.23% recall, 97.42% accuracy, and 93.25% IOU on the bridge concrete crack datasets, which are significantly higher than those of other state‐of‐the‐art baseline methods. It can be inferred that the application of hourglass‐shaped depth‐separable volumes can actively reduce basic model parameters. Moreover, the lightweight and efficient attention modules can achieve local cross‐channel interaction without dimensionality reduction and improve the network segmentation performance.

TinyAD: Memory-Efficient Anomaly Detection for Time-Series Data in Industrial IoT

Monitoring and detecting abnormal events in cyber-physical systems is crucial to industrial production. With the prevalent deployment of the Industrial Internet of Things (IIoT), an enormous amount of time series data is collected to facilitate machine learning models for anomaly detection, and it is of the utmost importance to directly deploy the trained models on the IIoT devices. However, it is most challenging to deploy complex deep learning models such as Convolutional Neural Networks (CNNs) on these memory-constrained IIoT devices embedded with microcontrollers (MCUs). To alleviate the memory constraints of MCUs, we propose a novel framework named Tiny Anomaly Detection (TinyAD) to efficiently facilitate onboard inference of CNNs for real-time anomaly detection. First, we conduct a comprehensive analysis of depthwise separable CNNs and regular CNNs for anomaly detection and find that the depthwise separable convolution operation can reduce the model size by 50-90% compared with the traditional CNNs. Then, to reduce the peak memory consumption of CNNs, we explore two complementary strategies, in-place, and patch-by-patch memory rescheduling, and integrate them into a unified framework. The in-place method decreases the peak memory of the depthwise convolution by sparing a temporary buffer to transfer the activation results, while the patch-by-patch method further reduces the peak memory of layer-wise execution by slicing the input data into corresponding receptive fields and executing in order. Furthermore, by adjusting the dimension of convolution filters, these strategies apply to both univariate time series and multidomain time series features. Extensive experiments on real-world industrial datasets show that our framework can reduce peak memory consumption by 2-5x with negligible computation overhead.

Lightweight image super-resolution for IoT devices using deep residual feature distillation network

The 5th industrial revolution is characterized by an extensive interconnection of embedded devices, which offer a range of services, including the monitoring of their environments. Images captured from remote cameras require enhancements for effective analysis. Despite recent progress in single-image super-resolution techniques by yielding impressive results through deep convolutional neural networks, the complexity of these advanced models renders them impractical for use on miniaturized Internet of Things (IoT) devices, primarily due to their limited computational capabilities and memory constraints. Furthermore, the rapid evolution of IoT devices necessitates efficient image super-resolution techniques, while existing advanced methods, based on deep convolutional neural networks, are too resource-intensive for these devices, and this gap highlights the need for a more suitable solution. In this study, we introduce a lightweight, efficient super-resolution model specifically designed for IoT devices. This model incorporates a novel deep residual feature distillation block (DRFDB), which leverages a depthwise-separable convolution block (DCB) for effective feature extraction. The focus is on reducing computational and memory demands without compromising on image quality. The proposed DCB extracts coarse features from given input features as calculation units, using two operations, depthwise and pointwise convolutions. These two operations are able to significantly reduce the number of parameters and floating-point operations while maintaining a PSNR value higher than the 90% threshold. We modify the proposed DCB and introduce a multi-kernel depthwise-separable convolution block (MKDCB) to fine-tune the model. The experiments, conduct on various standard datasets such as DIV2K, Set5, Set14, Urban100, and Manga109 by demonstrating that our model significantly outperforms existing methods in terms of both image quality and computational efficiency. The model shows improved performance metrics like PSNR, while requiring fewer parameters and less memory usage, making it highly suitable for IoT applications. This study presents a breakthrough in super-resolution for IoT devices, balancing high-quality image reconstruction with the limited resources of these devices.

A hybrid lightweight breast cancer classification framework using the histopathological images

A crucial element in the diagnosis of breast cancer is the utilization of a classification method that is efficient, lightweight, and precise. Convolutional neural networks (CNNs) have garnered attention as a viable approach for classifying histopathological images. However, deeper and wider models tend to rely on first-order statistics, demanding substantial computational resources and struggling with fixed kernel dimensions that limit encompassing diverse resolution data, thereby degrading the model’s performance during testing. This study introduces BCHI-CovNet, a novel lightweight artificial intelligence (AI) model for histopathological breast image classification. Firstly, a novel multiscale depth-wise separable convolution is proposed. It is introduced to split input tensors into distinct tensor fragments, each subject to unique kernel sizes integrating various kernel sizes within one depth-wise convolution to capture both low- and high-resolution patterns. Secondly, an additional pooling module is introduced to capture extensive second-order statistical information across the channels and spatial dimensions. This module works in tandem with an innovative multi-head self-attention mechanism to capture the long-range pixels contributing significantly to the learning process, yielding distinctive and discriminative features that further enrich representation and introduce pixel diversity during training. These novel designs substantially reduce computational complexities regarding model parameters and FLOPs, which is crucial for resource-constrained medical devices. The outcomes achieved by employing the suggested model on two openly accessible datasets for breast cancer histopathological images reveal noteworthy performance. Specifically, the proposed approach attains high levels of accuracy: 99.15 % at 40× magnification, 99.08 % at 100× magnification, 99.22 % at 200× magnification, and 98.87 % at 400× magnification on the BreaKHis dataset. Additionally, it achieves an accuracy of 99.38 % on the BACH dataset. These results highlight the exceptional effectiveness and practical promise of BCHI-CovNet for the classification of breast cancer histopathological images.

A lightweight YOLOv7 insulator defect detection algorithm based on DSC-SE.

As the UAV(Unmanned Aerial Vehicle) carrying target detection algorithm in transmission line insulator inspection, we propose a lightweight YOLOv7 insulator defect detection algorithm for the problems of inferior insulator defect detection speed and high model complexity. Firstly, a lightweight DSC-SE module is designed using a DSC(Depthwise Separable Convolution) fused SE channel attention mechanism to substitute the SC(Standard Convolution) of the YOLOv7 backbone extraction network to decrease the number of parameters in the network as well as to strengthen the shallow network's ability to obtain information about target features. Then, in the feature fusion part, GSConv(Grid Sensitive Convolution) is used instead of standard convolution to further lessen the number of parameters and the computational effort of the network. EIoU-loss(Efficient-IoU) is performed in the prediction head part to make the model converge faster. According to the experimental results, the recognition accuracy rate of the improved model is 95.2%, with a model size of 7.9M. Compared with YOLOv7, the GFLOPs are reduced by 54.5%, the model size is compressed by 37.8%, and the accuracy is improved by 4.9%. The single image detection time on the Jetson Nano is 105ms and the capture rate is 13FPS. With guaranteed accuracy and detection speed, it meets the demands of real-time detection.

DCDS-Net: Deep transfer network based on depth-wise separable convolution with residual connection for diagnosing gastrointestinal diseases

Gastrointestinal (GI) diseases are the most common in the human digestive system and has a significantly higher mortality rate. Accurate evaluation of endoscopic images plays an important role in decision making regarding patient treatment. Recently, convolutional neural networks (CNNs) have been introduced for the diagnosis of GI diseases. However, achieving high accuracy is still a challenging task. To overcome these limitations, we propose the “Densely Connected Depth-wise Separable Convolution-Based Network” (DCDS-Net) model, utilizing depth-wise separable convolution (DWSC) with residual connections and densely connected blocks (DCB), to effectively diagnose various endoscopic images of GI diseases. In addition, we incorporate global average pooling (GAP), batch normalization, dropout and dense layers in DCB to learn rich discriminative features and improve the performance of the model. We explored the feasibility of block-wise fine-tuning using transfer learning on the proposed model to reduce overfitting, and experimentally explore the optimal level of fine-tuning, since transfer learning is well suited to medical data where labeled data is scarce. The proposed method has been evaluated on 6000 labeled endoscopic images containing 4 classes of GI diseases. In addition, data augmentation has been incorporated into the training pipeline to improve the performance of the model. Furthermore, a critical study was conducted to evaluate the generalizability of the proposed model on smaller training samples (e.g., 60 %, 70 %, 80 %, and 90 %). The study employed Grad-CAM to generate heatmaps that identify the regions in the GI tract that are indicative of the presence of different diseases. The results of extensive experiments show that the proposed model shows significant improvements and achieves the highest classification accuracy of 99.33 %, precision of 99.37 %, recall of 99.32 % and outperforms all pre-trained and existing models for the detection of GI diseases. In conclusion, DCDS-Net exhibits high classification performance and can help endoscopists in automatic GI disease diagnosis.

Hyperspectral Image Classification Network Based on 3D Octave Convolution and Multiscale Depthwise Separable Convolution

Hyperspectral images (HSIs) are pivotal in various fields due to their rich spectral–spatial information. While convolutional neural networks (CNNs) have notably enhanced HSI classification, they often generate redundant spatial features. To address this, we introduce a novel HSI classification method, OMDSC, employing 3D Octave convolution combined with multiscale depthwise separable convolutional networks. This method initially utilizes 3D Octave convolution for efficient spectral–spatial feature extraction from HSIs, thereby reducing spatial redundancy. Subsequently, multiscale depthwise separable convolution is used to further improve the extraction of spatial features. Finally, the HSI classification results are output by softmax classifier. This work compares the method with other methods on three publicly available datasets in order to confirm its efficacy. The outcomes show that the method performs better in terms of classification.

A novel feature enhancement and semantic segmentation scheme for identifying low-contrast ocean oil spills

The oil spill accidents on the sea surface pose a severe threat to the marine environment and human health. This paper proposes a novel Semantic Segmentation Network (SSN) for processing oil spill images so that low-contrast oil spills on the sea surface can be accurately identified. After the detection accuracy and real-time performance of the current SSNs are compared, the basic network architecture of DeeplabV3+ based target detection is analyzed. The standard convolution is replaced by the Omni-dimensional Dynamic Convolution (ODConv) in the Ghost Module Depth-Wise separable Convolution (DWConv) to further enhance the feature extraction ability of the network. Furthermore, a new DeeplabV3+ based network with ODGhostNetV2 is constructed as the main feature extraction module, and an Adaptive Triplet Attention (ATA) module is deployed in the encoder and decoder at the same time. This not only improves the richness of semantic features but also increases the following receptive fields of the network model. ATA integrates the Adaptively Spatial Feature Fusion (ASFF) module to optimize the weight assignment problem in the feature map fusion process. The ablation experiments are conducted to verify the proposed network which show high accuracy and good real-time performance for the oil spill detection.

Improvement of response speed and precision of distributed Brillouin optical fiber sensors using neural networks

Distributed Brillouin optical fiber sensors can detect temperature and strain over long distances in standard single-mode fibers from the measurement of their Brillouin gain spectrum (BGS). However, the performance of these sensors is restricted by parameters such as the frequency scanning interval, average times of data acquisition, and Brillouin frequency shift (BFS) extraction time. Therefore, achieving high-precision real-time monitoring is challenging. In this study, we introduce a nonlinear interpolation deep neural network (NIDNN) to interpolate and denoise the BGS. Additionally, we propose a BFS extraction convolutional neural network (BFSECNN) based on depthwise separable convolution (DSC) to shorten the BFS extraction and training time. Both the simulation and experimental results show that the NIDNN and BFSECNN contributed to higher accuracy in BFS extraction and shorter data processing time. The sweep time of the distributed Brillouin optical fiber sensor with a 28.5-km length after using the NIDNN is 1/6 of the hardware sweep time of the 1-MHz interval, and the BFSECNN extraction time is 3.7 s. Moreover, combining the NIDNN with the BFSECNN reduced the BFS uncertainty by 5.3 MHz at the end of the optical fiber. Used individually or together, the proposed NIDNN and BFSECNN could improve the performance of existing distributed Brillouin optical fiber sensors without any hardware modifications.

Spatiotemporal Pyramidal CNN with Depth-Wise Separable Convolution for Eye Blinking Detection in the Wild

Eye blinking detection in the wild plays an essential role in deception detection, driving fatigue detection, etc. Despite the fact that numerous attempts have already been made, the majority of them have encountered difficulties, such as the derived eye images having different resolutions as the distance between the face and the camera changes; or the requirement of a lightweight detection model to obtain a short inference time in order to perform in realtime. In this research, two problems are addressed: how the eye blinking detection model can learn efficiently from different resolutions of eye pictures in diverse conditions; and how to reduce the size of the detection model for faster inference time. We propose to utilize upsampling and downsampling the input eye images to the same resolution as one potential solution for the first problem, then find out which interpolation method can result in the highest performance of the detection model. For the second problem, although a recent spatiotemporal convolutional neural network used for eye blinking detection has a strong capacity to extract both spatial and temporal characteristics, it remains having a high number of network parameters, leading to high inference time. Therefore, using Depth-wise Separable Convolution rather than conventional convolution layers inside each branch is considered in this paper as a feasible solution.