Fusion Layer Research Articles

Dual-Modal Fusion PRI-SWT Model for Eddy Current Detection of Cracks, Delamination, and Impact Damage in Carbon Fiber-Reinforced Plastic Materials

Carbon fiber-reinforced plastic (CFRP) composites are prone to damage during both manufacturing and operational phases, making the classification and identification of defects critical for maintaining structural integrity. This paper presents a novel dual-modal feature classification approach for the eddy current detection of CFRP defects, utilizing a Parallel Real–Imaginary/Swin Transformer (PRI-SWT) model. Built using the Transformer architecture, the PRI-SWT model effectively integrates the real and imaginary components of sinusoidal voltage signals, demonstrating a significant performance improvement over traditional classification methods such as Support Vector Machine (SVM) and Vision Transformer (ViT). The proposed model achieved a classification accuracy exceeding 95%, highlighting its superior capability in terms of addressing the complexities of defect detection. Furthermore, the influence of key factors—including the real–imaginary fusion layer, the number of layers, the window shift size, and the model’s scale—on the classification performance of the PRI-SWT model was systematically evaluated.

Text Command Intelligent Understanding for Cybersecurity Testing

Research on named entity recognition (NER) and command-line generation for network security evaluation tools is relatively scarce, and no mature models for recognition or generation have been developed thus far. Therefore, in this study, the aim is to build a specialized corpus for network security evaluation tools by combining knowledge graphs and information entropy for automatic entity annotation. Additionally, a novel NER approach based on the KG-BERT-BiLSTM-CRF model is proposed. Compared to the traditional BERT-BiLSTM model, the KG-BERT-BiLSTM-CRF model demonstrates superior performance when applied to the specialized corpus of network security evaluation tools. The graph attention network (GAT) component effectively extracts relevant sequential content from datasets in the network security evaluation domain. The fusion layer then concatenates the feature sequences from the GAT and BiLSTM layers, enhancing the training process. Upon successful NER execution, in this study, the identified entities are mapped to pre-established command-line data for network security evaluation tools, achieving automatic conversion from textual content to evaluation commands. This process not only improves the efficiency and accuracy of command generation but also provides practical value for the development and optimization of network security evaluation tools. This approach enables the more precise automatic generation of evaluation commands tailored to specific security threats, thereby enhancing the timeliness and effectiveness of cybersecurity defenses.

A Mechanical Fault Identification Method for On-Load Tap Changers Based on Hybrid Time—Frequency Graphs of Vibration Signals and DSCNN-SVM with Small Sample Sizes

In engineering applications, the accuracy of on-load tap changer (OLTC) mechanical fault identification methods based on vibration signals is constrained by the quantity and quality of the samples. Therefore, a novel small-sample-size OLTC mechanical fault identification method incorporating short-time Fourier transform (STFT), synchrosqueezed wavelet transform (SWT), a dual-stream convolutional neural network (DSCNN), and support vector machine (SVM) is proposed. Firstly, the one-dimensional time-series vibration signals are transformed using STFT and SWT to obtain time–frequency graphs. STFT time–frequency graphs capture the global features of the OLTC vibration signals, while SWT time–frequency graphs capture the local features of the OLTC vibration signals. Secondly, these time–frequency graphs are input into the CNN to extract key features. In the fusion layer, the feature vectors from the STFT and SWT graphs are combined to form a fusion vector that encompasses both global and local time–frequency features. Finally, the softmax classifier of the traditional CNN is replaced with an SVM classifier, and the fusion vector is input into this classifier. Compared to the traditional fault identification methods, the proposed method demonstrates higher identification accuracy and stronger generalization ability under the conditions of small sample sizes and noise interference.

Visual detection of drilling robot position for rockburst prevention in mining processing by a new image dehazing method

The positioning of drill pipe based on visual detection is a crucial aspect in achieving unmanned operation of drilling robot for rockburst prevention. However, the images directly collected from mechanized mining face are always polluted. In order to eliminate the interference of dust and haze in the image, an image dehazing method based on improved CycleGAN model is proposed in this paper. Firstly, a pipe image dehazing dataset for the rockburst prevention drilling robot is collected and established. Moreover, a generator architecture with a multi-scale U-shaped network structure is designed to improve the quality and accuracy of image recovery. A new reconstruction block is designed and an SK fusion layer is introduced to improve the feature extraction capability of the model, and a MU-CycleGAN network structure is constructed. Finally, an experimental platform for drill pipe image dehazing of the drilling robot for rockburst prevention is set up in the intelligent mining equipment laboratory. Experimental results showed that the PSNR and SSIM of the image dehazing model achieved 27.04 and 0.946, and the success rate of drill pipe grabbing has increased by 13.75%. Experimental results reveal that the proposed framework achieves superior image enhancement performance than the comparison algorithms.

A Lightweight Blind Obstacle Detection Network for Mobile Side

Abstract. China has the longest and widest distribution of blind corridors in the world, but in many cities they are virtually non-existent, with all kinds of obstacles affecting the movement of the blind. Thus, guaranteeing safe traveling for the blind has become an increasingly important topic. Some existing assistive traveling devices for the blind have problems such as poor portability and low-cost performance. With the rise of the mobility era and the rapid development of deep learning technology, the target detection of blind obstacles on cell phones has become feasible. In this paper, we take the target detection model YOLOv8n as the base network, redesign the neck of the network, use GS convolution as the basis, adopt one-time aggregation and cross-channel branching to build a lightweight module, and use the CARAFE operator as the up-sampling method, and stack the lightweight module with CARAFE operator to build a new feature fusion layer, forming a lightweight feature-aware enhanced target detection network. The results show that the accuracy still reaches 99.5% of the original network under the premise that the model size is reduced by 1.2%, and the improved network achieves a balance between accuracy and lightweight.

Multi-modal classification of breast cancer lesions in Digital Mammography and contrast enhanced spectral mammography images

Breast cancer ranks as the second most prevalent cancer in women, recognized as one of the most dangerous types of cancer, and is on the rise globally. Regular screenings are essential for early-stage treatment. Digital mammography (DM) is the most recognized and widely used technique for breast cancer screening. Contrast-Enhanced Spectral Mammography (CESM or CM) is used in conjunction with DM to detect and identify hidden abnormalities, particularly in dense breast tissue where DM alone might not be as effective. In this work, we explore the effectiveness of each modality (CM, DM, or both) in detecting breast cancer lesions using deep learning methods. We introduce an architecture for detecting and classifying breast cancer lesions in DM and CM images in Craniocaudal (CC) and Mediolateral Oblique (MLO) views. The proposed architecture (JointNet) consists of a convolution module for extracting local features, a transformer module for extracting long-range features, and a feature fusion layer to fuse the local features, global features, and global features weighted based on the local ones. This significantly enhances the accuracy of classifying DM and CM images into normal or abnormal categories and lesion classification into benign or malignant. Using our architecture as a backbone, three lesion classification pipelines are introduced that utilize attention mechanisms focused on lesion shape, texture, and overall breast texture, examining the critical features for effective lesion classification. The results demonstrate that our proposed methods outperform their components in classifying images as normal or abnormal and mitigate the limitations of independently using the transformer module or the convolution module. An ensemble model is also introduced to explore the effect of each modality and each view to increase our baseline architecture's accuracy. The results demonstrate superior performance compared with other similar works. The best performance on DM images was achieved with the semi-automatic AOL Lesion Classification Pipeline, yielding an accuracy of 98.85 %, AUROC of 0.9965, F1-score of 98.85 %, precision of 98.85 %, and specificity of 98.85 %. For CM images, the highest results were obtained using the automatic AOL Lesion Classification Pipeline, with an accuracy of 97.47 %, AUROC of 0.9771, F1-score of 97.34 %, precision of 94.45 %, and specificity of 97.23 %. The semi-automatic ensemble AOL Classification Pipeline provided the best overall performance when using both DM and CM images, with an accuracy of 94.74 %, F1-score of 97.67 %, specificity of 93.75 %, and sensitivity of 95.45 %. Furthermore, we explore the comparative effectiveness of CM and DM images in deep learning models, indicating that while CM images offer clearer insights to the human eye, our model trained on DM images yields better results using Attention on Lesion (AOL) techniques. The research also suggests a multimodal approach using both DM and CM images and ensemble learning could provide more robust classification outcomes.

Alignable kernel network

To enhance the adaptability and performance of Convolutional Neural Networks (CNN), we present an adaptable mechanism called Alignable Kernel (AliK) unit, which dynamically adjusts the receptive field (RF) dimensions of a model in response to varying stimuli. The branches of AliK unit are integrated through a novel align transformation softmax attention, incorporating prior knowledge through rank ordering constraints. The attention weightings across the branches establish the effective RF scales, leveraged by neurons in the fusion layer. This mechanism is inspired by neuroscientific observations indicating that the RF dimensions of neurons in the visual cortex vary with the stimulus, a feature often overlooked in CNN architectures. By aggregating successive AliK ensembles, we develop a deep network architecture named the Alignable Kernel Network (AliKNet). AliKNet with interdisciplinary design improves the network’s performance and interpretability by taking direct inspiration from the structure and function of human neural systems, especially the visual cortex. Empirical evaluations in the domains of image classification and semantic segmentation have demonstrated that AliKNet excels over numerous state-of-the-art architectures, achieving this without increasing model complexity. Furthermore, we demonstrate that AliKNet can identify target objects across various scales, confirming their ability to dynamically adapt their RF sizes in response to the input data.

Medical Visual Question‐Answering Model Based on Knowledge Enhancement and Multi‐Modal Fusion

ABSTRACTThis paper aims to utilize a knowledge graph for importing external knowledge. It combines multi‐modal fusion mechanisms and confidence detection mechanisms to explore the correlation between clinical problems and medical images, enhancing their effectiveness in medical visual question‐answering tasks. The proposed medical visual question answering model comprises a text knowledge enhancement layer, an image embedding layer, a multimodal fusion layer, a confidence detection layer, and a prediction layer. The experimental results demonstrate that the medical vision question‐answering model, based on knowledge enhancement and multi‐modal fusion, achieves an optimal accuracy of 59.3% and 16.2% in open‐domain question‐answering tasks on the VQA‐RAD and PathVQA datasets, respectively, thus validating the effectiveness of the proposed model.

Efficient audio–visual information fusion using encoding pace synchronization for Audio–Visual Speech Separation

Contemporary audio–visual speech separation (AVSS) models typically use encoders that merge audio and visual representations by concatenating them at a specific layer. This approach assumes that both modalities progress at the same pace and that information is adequately encoded at the chosen fusion layer. However, this assumption is often flawed due to inherent differences between the audio and visual modalities. In particular, the audio modality, being more directly tied to the final output (i.e., denoised speech), tends to converge faster than the visual modality. This discrepancy creates a persistent challenge in selecting the appropriate layer for fusion. To address this, we propose the Encoding Pace Synchronization Network (EPS-Net) for AVSS. EPS-Net allows for the independent encoding of the two modalities, enabling each to be processed at its own pace. At the same time, it establishes communication between the audio and visual modalities at corresponding encoding layers, progressively synchronizing their encoding speeds. This approach facilitates the gradual fusion of information while preserving the unique characteristics of each modality. The effectiveness of the proposed method has been validated through extensive experiments on the LRS2, LRS3, and VoxCeleb2 datasets, demonstrating superior performance over state-of-the-art methods.

Heuristic Heterogeneous Graph Reasoning Networks for Fact Verification.

Existing studies on table-based fact verification generally capture linguistic evidence from claim-table subgraphs or logical evidence from program-table subgraphs independently. However, there is insufficient association interaction between the two types of evidence, which makes it difficult to obtain valuable consistency features between them. In this work, we propose heuristic heterogeneous graph reasoning networks (H2GRN) to capture the shared consistent evidence by strengthening associations between linguistic and logical evidence from two perspectives of graph construction and reasoning mechanism. Specifically, 1) to enhance the close connectivity of the two subgraphs, rather than simply connecting two subgraphs by the nodes with the same content (the constructed graph in this way has severe sparsity), we construct a heuristic heterogeneous graph, which relies on claim semantics as heuristic knowledge to guide the connections of the program-table subgraph, and in turn expands the connectivity of the claim-table subgraph through logical information of programs as heuristic knowledge; and 2) to establish adequate association interaction between linguistic evidence and logical evidence, we design multiview reasoning networks. In detail, we propose local-view multihop knowledge reasoning (MKR) networks to enable the current node to establish association not only with one-hop neighbors, but also with multihop neighbors, to capture context-richer evidence information. We execute MKR on heuristic claim-table and program-table subgraphs to learn context-richer linguistic evidence and logical evidence, respectively. Meanwhile, we develop global-view graph dual-attention networks (DAN) that execute on the entire heuristic heterogeneous graph, reinforcing global-level significant consistency evidence. Finally, the consistency fusion layer is devised to weaken the disagreement between the three types of evidence to assist in capturing consistent shared evidence for verifying claims. Experiments on TABFACT and FEVEROUS demonstrate the effectiveness of H2GRN.

MLFGCN: short-term residential load forecasting via graph attention temporal convolution network.

Residential load forecasting is a challenging task due to the random fluctuations caused by complex correlations and individual differences. The existing short-term load forecasting models usually introduce external influencing factors such as climate and date. However, these additional information not only bring computational burden to the model, but also have uncertainty. To address these issues, we propose a novel multi-level feature fusion model based on graph attention temporal convolutional network (MLFGCN) for short-term residential load forecasting. The proposed MLFGCN model fully considers the potential long-term dependencies in a single load series and the correlations between multiple load series, and does not require any additional information to be added. Temporal convolutional network (TCN) with gating mechanism is introduced to learn potential long-term dependencies in the original load series. In addition, we design two graph attentive convolutional modules to capture potential multi-level dependencies in load data. Finally, the outputs of each module are fused through an information fusion layer to obtain the highly accurate forecasting results. We conduct validation experiments on two real-world datasets. The results show that the proposed MLFGCN model achieves 0.25, 7.58% and 0.50 for MAE, MAPE and RMSE, respectively. These values are significantly better than those of baseline models. The MLFGCN algorithm proposed in this paper can significantly improve the accuracy of short-term residential load forecasting. This is achieved through high-quality feature reconstruction, comprehensive information graph construction and spatiotemporal features capture.

Deep Learning-Based Biomimetic Identification Method for Mask Wearing Standardization.

Deep learning technology can automatically learn features from large amounts of data, with powerful feature extraction and pattern recognition capabilities, thereby improving the accuracy and efficiency of object detection. [The objective of this study]: In order to improve the accuracy and speed of mask wearing deep learning detection models in the post pandemic era, the [Problem this study aimed to resolve] was based on the fact that no research work has been reported on standardized detection models for mask wearing with detecting nose targets specially. [The topic and method of this study]: A mask wearing normalization detection model (towards the wearing style exposing the nose to outside, which is the most obvious characteristic of non-normalized style) based on improved YOLOv5s (You Only Look Once v5s is an object detection network model) was proposed. [The improved method of the proposed model]: The improvement design work of the detection model mainly includes (1) the BottleneckCSP (abbreviation of Bottleneck Cross Stage Partial) module was improved to a BottleneckCSP-MASK (abbreviation of Bottleneck Cross Stage Partial-MASK) module, which was utilized to replace the BottleneckCSP module in the backbone architecture of the original YOLOv5s model, which reduced the weight parameters' number of the YOLOv5s model while ensuring the feature extraction effect of the bonding fusion module. (2) An SE module was inserted into the proposed improved model, and the bonding fusion layer in the original YOLOv5s model was improved for better extraction of the features of mask and nose targets. [Results and validation]: The experimental results indicated that, towards different people and complex backgrounds, the proposed mask wearing normalization detection model can effectively detect whether people are wearing masks and whether they are wearing masks in a normalized manner. The overall detection accuracy was 99.3% and the average detection speed was 0.014 s/pic. Contrasted with original YOLOv5s, v5m, and v5l models, the detection results for two types of target objects on the test set indicated that the mAP of the improved model increased by 0.5%, 0.49%, and 0.52%, respectively, and the size of the proposed model compressed by 10% compared to original v5s model. The designed model can achieve precise identification for mask wearing behaviors of people, including not wearing a mask, normalized wearing, and wearing a mask non-normalized.

GOI-YOLOv8 Grouping Offset and Isolated GiraffeDet Low-Light Target Detection.

In the realm of computer vision, object detection holds significant importance and has demonstrated commendable performance across various scenarios. However, it typically requires favorable visibility conditions within the scene. Therefore, it is imperative to explore methodologies for conducting object detection under low-visibility circumstances. With its balanced combination of speed and accuracy, the state-of-the-art YOLOv8 framework has been recognized as one of the top algorithms for object detection, demonstrating outstanding performance results across a range of standard datasets. Nonetheless, current YOLO-series detection algorithms still face a significant challenge in detecting objects under low-light conditions. This is primarily due to the significant degradation in performance when detectors trained on illuminated data are applied to low-light datasets with limited visibility. To tackle this problem, we suggest a new model named Grouping Offset and Isolated GiraffeDet Target Detection-YOLO based on the YOLOv8 architecture. The proposed model demonstrates exceptional performance under low-light conditions. We employ the repGFPN feature pyramid network in the design of the feature fusion layer neck to enhance hierarchical fusion and deepen the integration of low-light information. Furthermore, we refine the repGFPN feature fusion layer by introducing a sampling map offset to address its limitations in terms of weight and efficiency, thereby better adapting it to real-time applications in low-light environments and emphasizing the potential features of such scenes. Additionally, we utilize group convolution to isolate interference information from detected object edges, resulting in improved detection performance and model efficiency. Experimental results demonstrate that our GOI-YOLO reduces the parameter count by 11% compared to YOLOv8 while decreasing computational requirements by 28%. This optimization significantly enhances real-time performance while achieving a competitive increase of 2.1% in Map50 and 0.6% in Map95 on the ExDark dataset.

Investigation on the influence of electrospark deposited 718 alloy coating on the penetration performance of 93 W rod

A ballistic experiment on a semi-infinite RHA target impacted by two thicknesses clade 93 W rods (200 μm 718 alloy and 400 μm 718 alloy) with an impact velocity of 1500 ± 50 m/s is conducted to investigate the influence of 93 W surface deposited coating on penetration performance. Moreover, the findings are compared with those of an unclad 93 W rod. This study aims to provide a comprehensive microstructural overview of the penetration process, including the solid-state penetrator flow and the exchange interaction mode of rod and target (R.T.) materials, using optical metallography and electron microscopy. Results indicate that the coating area is composed of approximately equiaxed or short columnar cellular crystals. Furthermore, a metallurgical fusion layer is formed between the substrate and the coating, ensuring excellent bonding strength of the coating. No significant difference in the rheological mode is found between the interfaces of the 200 μm 718 alloy and 93 W R.T. materials, while the 400 μm 718 alloy forms a barrier mix zone at the interface between the R.T. This phenomenon alters the exchange mode of the target material and affects the interface flow mechanism. The barrier mix zone also acts as a solid-state lubricant at the interface, reducing the resistance to the rod's penetration. The average penetration depth of the 400 μm clad rod is 6.89 % higher than that of the 93 W rod. This finding reveals the clad material enhances the rod's performance and provides self-lubricating characteristics.

An advanced hybrid deep learning model for accurate energy load prediction in smart building

In smart cities, sustainable development depends on energy load prediction since it directs utilities in effectively planning, distributing and generating energy. This work presents a novel hybrid deep learning model including components of the Improved-convolutional neural network (CNN), bidirectional long short-term memory (Bi-LSTM), Graph neural network (GNN), Transformer and Fusion Layer architectures for precise energy load forecasting. Better feature extraction results from the Improved-CNN's dilated convolution and residual block accommodation of wide receptive fields reduced the vanishing gradient problem. By capturing temporal links in both directions, Bi-LSTM networks help to better grasp complicated energy use patterns. Graph neural networks improve predictive capacities across linked systems by characterizing the spatial relationships between energy-consuming units in smart cities. Emphasizing critical trends to guarantee reliable forecasts, transformer models use attention methods to manage long-term dependencies in energy consumption data. Combining CNN, Bi-LSTM, Transformer and GNN component predictions in a Fusion Layer synthesizes numerous data representations to increase accuracy. With Root Mean Square Error of 5.7532 Wh, Mean Absolute Percentage Error of 3.5001%, Mean Absolute Error of 6.7532 Wh and R2 of 0.9701, the hybrid model fared better than other models on the ‘Electric Power Consumption’ Kaggle dataset. This work develops a realistic model that helps informed decision-making and enhances energy efficiency techniques, promoting energy load forecasting in smart cities.

Research on Detection Algorithm of Green Walnut in Complex Environment

The growth environment of green walnuts is complex. In the actual picking and identification process, interference from near-background colors, occlusion by branches and leaves, and excessive model complexity pose higher demands on the performance of walnut detection algorithms. Therefore, a lightweight walnut detection algorithm suitable for complex environments is proposed based on YOLOv5s. First, the backbone network is reconstructed using the lightweight GhostNet network, laying the foundation for a lightweight model architecture. Next, the C3 structure in the feature fusion layer is optimized by proposing a lightweight C3 structure to enhance the model’s focus on important walnut features. Finally, the loss function is improved to address the problems of target loss and gradient adaptability during training. To further reduce model complexity, the improved algorithm undergoes pruning and knowledge distillation operations, and is then deployed and tested on small edge devices. Experimental results show that compared to the original YOLOv5s model, the improved algorithm reduces the number of parameters by 72.9% and the amount of computation by 84.1%. The mAP0.5 increased by 1.1%, the precision increased by 0.7%, the recall increased by 0.3%, and the FPS is 179.6% of the original model, meeting the real-time detection needs for walnut recognition and providing a reference for walnut harvesting identification.

Spatial exchanging fusion network for RGB-T crowd counting

RGB-T crowd counting (RGB-T CC) aims to estimate the crowd population size utilizing the complementary information from visible and thermal images. Current deep models for RGB-T CC typically adopt a three-tier architecture, featuring a middle fusion layer that aggregates both RGB and thermal streams. However, we find that this dedicated fusion layer dominates the training process, causing under-optimization of both modal branches, which becomes the performance bottleneck in mainstream multi-modal counting models. To address this challenge, we propose a simple-yet-effective counting architecture, the Spatial Exchanging Fusion Network (SEFNet). It is built on a Dual Attention Guided Spatial Exchanging (DASE) mechanism, enabling direct extraction and exchange of modality-complementary features between modalities without the extra fusion branch employed in most existing works. This design ensures a more balanced gradient back-propagation over networks, attaining optimized representations in multi-modality fusion over prior models. Besides, the Modality Gradient Enhancement Module (MGEM) in SEFNet can effectively learn modality-specific crowd representations with two counting sub-tasks, dynamically achieving better gradient distribution and further enhancing optimization in both modalities. Extensive experiments demonstrate that SEFNet significantly outperforms state-of-the-art methods on mainstream benchmark datasets, and also exhibits promising generalization ability across various counting backbones and losses.

EEG-Based Seizure Prediction Using Hybrid DenseNet-ViT Network with Attention Fusion.

Epilepsy seizure prediction is vital for enhancing the quality of life for individuals with epilepsy. In this study, we introduce a novel hybrid deep learning architecture, merging DenseNet and Vision Transformer (ViT) with an attention fusion layer for seizure prediction. DenseNet captures hierarchical features and ensures efficient parameter usage, while ViT offers self-attention mechanisms and global feature representation. The attention fusion layer effectively amalgamates features from both networks, guaranteeing the most relevant information is harnessed for seizure prediction. The raw EEG signals were preprocessed using the short-time Fourier transform (STFT) to implement time-frequency analysis and convert EEG signals into time-frequency matrices. Then, they were fed into the proposed hybrid DenseNet-ViT network model to achieve end-to-end seizure prediction. The CHB-MIT dataset, including data from 24 patients, was used for evaluation and the leave-one-out cross-validation method was utilized to evaluate the performance of the proposed model. Our results demonstrate superior performance in seizure prediction, exhibiting high accuracy and low redundancy, which suggests that combining DenseNet, ViT, and the attention mechanism can significantly enhance prediction capabilities and facilitate more precise therapeutic interventions.

Real-time tilapia fillet defect segmentation on edge device for robotic trimming

The implementation of robotic tilapia fillet trimming instead of manual labor is a pivotal advancement in intelligent fish processing, offering substantial superiority in operational efficiency and product quality. The study presents an improved model called TFDS-YOLOv8n for tilapia fillet defects segmentation. The model incorporates Coordinate Attention (CA) into the feature extraction layer, thereby enhancing its ability to capture characteristics at various levels of the input. Additionally, the feature fusion layer is reconstructed as Slim-Neck to reduce the number of parameters without compromising prediction accuracy. Furthermore, the bounding box loss function is modified by MPDIoU to expedite the model convergence. The proposed model was tested employing the dataset collected from the practical tilapia processing plant. Ablation experiments demonstrate that TFDS-YOLOv8n achieves a reduction of 0.29 MB in parameters and 1 G in Floating Point Operations (FLOPs) while increasing bbox_mAP and mask_mAP by 2.8 % and 2.5 %, respectively. Eventually, the model is further accelerated by TensorRT and deployed on the edge device. Experimental results show that the quantized model possesses a faster inference speed than the untransformed model, which realizes real-time detection of tilapia fillet defects and facilitates robotization in fish processing.

FLAT: Fusing layer representations for more efficient transfer learning in NLP

Parameter efficient transfer learning (PETL) methods provide an efficient alternative for fine-tuning. However, typical PETL methods inject the same structures to all Pre-trained Language Model (PLM) layers and only use the final hidden states for downstream tasks, regardless of the knowledge diversity across PLM layers. Additionally, the backpropagation path of existing PETL methods still passes through the frozen PLM during training, which is computational and memory inefficient. In this paper, we propose FLAT, a generic PETL method that explicitly and individually combines knowledge across all PLM layers based on the tokens to perform a better transferring. FLAT considers the backbone PLM as a feature extractor and combines the features in a side-network, hence the backpropagation does not involve the PLM, which results in much less memory requirement than previous methods. The results on the GLUE benchmark show that FLAT outperforms other tuning techniques in the low-resource scenarios and achieves on-par performance in the high-resource scenarios with only 0.53% trainable parameters per task and 3.2× less GPU memory usagewith BERTbase. Besides, further ablation study is conducted to reveal that the proposed fusion layer effectively combines knowledge from PLM and helps the classifier to exploit the PLM knowledge to downstream tasks. We will release our code for better reproducibility.