Large Model Size Research Articles

Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

Speech separation, sometimes known as the “cocktail party problem”, is the process of separating individual speech signals from an audio mixture that includes ambient noises and several speakers. The goal is to extract the target speech in this complicated sound scenario and either make it easier to understand or increase its quality so that it may be used in subsequent processing. Speech separation on overlapping audio data is important for many speech-processing tasks, including natural language processing, automatic speech recognition, and intelligent personal assistants. New speech separation algorithms are often built on a deep neural network (DNN) structure, which seeks to learn the complex relationship between the speech mixture and any specific speech source of interest. DNN-based speech separation algorithms outperform conventional statistics-based methods, although they typically need a lot of processing and/or a larger model size. This study presents a new end-to-end speech separation network called ESC-MASD-Net (effective speaker separation through convolutional multi-view attention and SuDoRM-RF network), which has relatively fewer model parameters compared with the state-of-the-art speech separation architectures. The network is partly inspired by the SuDoRM-RF++ network, which uses multiple time-resolution features with downsampling and resampling for effective speech separation. ESC-MASD-Net incorporates the multi-view attention and residual conformer modules into SuDoRM-RF++. Additionally, the U-Convolutional block in ESC-MASD-Net is refined with a conformer layer. Experiments conducted on the WHAM! dataset show that ESC-MASD-Net outperforms SuDoRM-RF++ significantly in the SI-SDRi metric. Furthermore, the use of the conformer layer has also improved the performance of ESC-MASD-Net.

Aircraft Objection Detection Method of Airport Surface based on Improved YOLOv5

An aircraft object detection method on the basis of improved YOLOv5 was proposed to address the issues of large model size, high number of parameters, and inability to meet real-time monitoring requirements of aircrafts in traditional object detection. Firstly, the basic unit of ShuffleNetv2 network was optimized through replacing 3x3 convolution with 5x5 convolution and removing subsequent 1x1 convolution. Simultaneously, the original ReLU activation function was replaced with PReLU. Secondly, CBAM (Convolutional Block Attention Module) attention mechanism was developed to enhance the detection accuracy of the improved network. Finally, improved ShuffleNetv2 network was applied as the backbone structure of YOLOv5. Experimental results revealed that the parameter number of the improved YOLOv5 method introduced in this paper was decreased by 18 times, with a model size of 1.03M. Therefore, a 20.8% increase was achieved in frames per second (FPS) in GPU environments and a 234.6% increase was observed in FPS in CPU environments, while a mean average precision (mAP@0.5) of 0.99 was maintained compared with traditional YOLOv5 network. Because of the advantages of fewer parameters, faster recognition speed, higher localization accuracy, and smaller memory requirement, the developed method was found to be suitable for real-time monitoring of aircrafts in airport surface.

Lightweight Improved YOLOv5s-CGhostnet for Detection of Strawberry Maturity Levels and Counting

A lightweight strawberry detection and localization algorithm plays a crucial role in enabling the harvesting robot to effectively harvest strawberries. The YOLO model has often been used in strawberry fruit detection for its high accuracy, speed, and robustness. However, some challenges exist, such as the requirement for large model sizes, high computation operation, and undesirable detection. Therefore, the lightweight improved YOLOv5s-CGhostnet was proposed to enhance strawberry detection. In this study, YOLOv5s underwent comprehensive model compression with Ghost modules GCBS and GC3, replacing modules CBS and C3 in the backbone and neck. Furthermore, the default GIOU bounding box regressor loss function was replaced by SIOU for improved localization. Similarly, CBAM attention modules were added before SPPF and between the up-sampling and down-sampling feature fusion FPN–PAN network in the neck section. The improved model exhibited higher mAP@0.5 of 91.7% with a significant decrement in model size by 85.09% and a reduction in GFLOPS by 88.5% compared to the baseline model of YOLOv5. The model demonstrated an increment in mean average precision, a decrement in model size, and reduced computation overhead compared to the standard lightweight YOLO models.

LW-YOLO: Lightweight Deep Learning Model for Fast and Precise Defect Detection in Printed Circuit Boards

Printed circuit board (PCB) manufacturing processes are becoming increasingly complex, where even minor defects can impair product performance and yield rates. Precisely identifying PCB defects is critical but remains challenging. Traditional PCB defect detection methods, such as visual inspection and automated technologies, have limitations. While defects can be readily identified based on symmetry, the operational aspect proves to be quite challenging. Deep learning has shown promise in defect detection; however, current deep learning models for PCB defect detection still face issues like large model size, slow detection speed, and suboptimal accuracy. This paper proposes a lightweight YOLOv8 (You Only Look Once version 8)-based model called LW-YOLO (Lightweight You Only Look Once) to address these limitations. Specifically, LW-YOLO incorporates a bidirectional feature pyramid network for multiscale feature fusion, a Partial Convolution module to reduce redundant calculations, and a Minimum Point Distance Intersection over Union loss function to simplify optimization and improve accuracy. Based on the experimental data, LW-YOLO achieved an mAP0.5 of 96.4%, which is 2.2 percentage points higher than YOLOv8; the precision reached 97.1%, surpassing YOLOv8 by 1.7 percentage points; and at the same time, LW-YOLO achieved an FPS of 141.5. The proposed strategies effectively enhance efficiency and accuracy for deep-learning-based PCB defect detection.

LCSNet: Light-Weighted Convolution-Based Segmentation Method with Separable Multi-Directional Convolution Module for Concrete Crack Segmentation in Drones

Concrete cracks pose significant safety hazards to buildings, and semantic segmentation models based on deep learning have achieved state-of-the-art results in concrete crack detection. However, these models usually have a large model size which is impossible to use in drones. To solve this problem, we propose a Light-Weighted Convolution-Based Segmentation Method with a Separable Multi-Directional Convolution Module (LCSNet). In our proposed method, light-weighted convolution is used to substitute all traditional convolutions. In addition, a light-weighted structure named a Separable Multi-Directional Convolution Module (SMDCM) is used to substitute traditional parallel structures or attention modules to learn contextual or detail features. Thus, the ability to extract the contextual feature information of the model can be retained while the computational complexity is largely reduced. Through these two improvements, the model size of the proposed model can have a lower computational complexity. The experimental results show that our proposed LCSNet can achieve accuracies of 94.2%, 83.6%, 99.2%, and 83.3% on the Cracktree200, CRACK500, CFD, and RECrack datasets, respectively, which are higher than those of traditional models. However, the model size of our LCSNet is only 2M.

Efficient Lightweight Image Denoising with Triple Attention Transformer

Transformer has shown outstanding performance on image denoising, but the existing Transformer methods for image denoising are with large model sizes and high computational complexity, which is unfriendly to resource-constrained devices. In this paper, we propose a Lightweight Image Denoising Transformer method (LIDFormer) based on Triple Multi-Dconv Head Transposed Attention (TMDTA) to boost computational efficiency. LIDFormer first implements Discrete Wavelet Transform (DWT), which transforms the input image into a low-frequency space, greatly reducing the computational complexity of image denoising. However, the low-frequency image lacks fine-feature information, which degrades the denoising performance. To handle this problem, we introduce the Complementary Periodic Feature Reusing (CPFR) scheme for aggregating the shallow-layer features and the deep-layer features. Furthermore, TMDTA is proposed to integrate global context along three dimensions, thereby enhancing the ability of global feature representation. Note that our method can be applied as a pipeline for both convolutional neural networks and Transformers. Extensive experiments on several benchmarks demonstrate that the proposed LIDFormer achieves a better trade-off between high performance and low computational complexity on real-world image denoising tasks.

Fairness-Aware Structured Pruning in Transformers

The increasing size of large language models (LLMs) has introduced challenges in their training and inference. Removing model components is perceived as a solution to tackle the large model sizes, however, existing pruning methods solely focus on performance, without considering an essential aspect for the responsible use of LLMs: model fairness. It is crucial to address the fairness of LLMs towards diverse groups, such as women, Black people, LGBTQ+, Jewish communities, among others, as they are being deployed and available to a wide audience. In this work, first, we investigate how attention heads impact fairness and performance in pre-trained transformer-based language models. We then propose a novel method to prune the attention heads that negatively impact fairness while retaining the heads critical for performance, i.e. language modeling capabilities. Our approach is practical in terms of time and resources, as it does not require fine-tuning the final pruned, and fairer, model. Our findings demonstrate a reduction in gender bias by 19%, 19.5%, 39.5%, 34.7%, 23%, and 8% for DistilGPT-2, GPT-2, GPT-Neo of two different sizes, GPT-J, and Llama 2 models, respectively, in comparison to the biased model, with only a slight decrease in performance. WARNING: This work uses language that is offensive in nature.

A Lightweight Remote Sensing Aircraft Object Detection Network Based on Improved YOLOv5n

Due to the issues of remote sensing object detection algorithms based on deep learning, such as a high number of network parameters, large model size, and high computational requirements, it is challenging to deploy them on small mobile devices. This paper proposes an extremely lightweight remote sensing aircraft object detection network based on the improved YOLOv5n. This network combines Shufflenet v2 and YOLOv5n, significantly reducing the network size while ensuring high detection accuracy. It substitutes the original CIoU and convolution with EIoU and deformable convolution, optimizing for the small-scale characteristics of aircraft objects and further accelerating convergence and improving regression accuracy. Additionally, a coordinate attention (CA) mechanism is introduced at the end of the backbone to focus on orientation perception and positional information. We conducted a series of experiments, comparing our method with networks like GhostNet, PP-LCNet, MobileNetV3, and MobileNetV3s, and performed detailed ablation studies. The experimental results on the Mar20 public dataset indicate that, compared to the original YOLOv5n network, our lightweight network has only about one-fifth of its parameter count, with only a slight decrease of 2.7% in mAP@0.5. At the same time, compared with other lightweight networks of the same magnitude, our network achieves an effective balance between detection accuracy and resource consumption such as memory and computing power, providing a novel solution for the implementation and hardware deployment of lightweight remote sensing object detection networks.

A practical algorithm for the design of multiple-sized porous scaffolds with triply periodic structures

In this study, we present a practical volume-merging method for generating multiple-sized porous structures that exhibit geometries with triply periodic minimal surface (TPMS) lattice structures. The proposed method consists of three stages: (1) designing the physical models with a signed distance field, (2) performing a merging operation for the porous scaffolds, and (3) assembling different units into a composite structure. The significant advantages of the proposed algorithm can be summarized as follows: Our method is independent of the model shape; the designed structures maintain a smooth surface with a constant mean curvature, and the mathematical computational complexity is low. We can join two different-sized triply periodic minimal surface lattices in the radial direction, where the transition region is obtained by smooth interpolation between the two lattice structures with different cell sizes or types. However, constructing large-sized models is only conceptually possible due to computational cost and memory storage constraints. To overcome these limitations, we present a practical method that can efficiently assemble large-scaled models at a low computational cost. The proposed method is based on a Boolean union operation of basic units of TPMS. Thus, it is simple to generate large-scale three-dimensional multiple-sized porous volumes based on our proposed method, which can be applied to many applications in mechanical and electrical engineering. The produced multi-scale compound scaffolds have smooth surfaces without fractures, making them suitable for straightforward application in additive manufacturing. Several numerical tests are conducted to validate the efficiency of the proposed algorithm.

Proportional feature pyramid network based on weight fusion for lane detection.

Lane detection under extreme conditions presents a highly challenging task that requires capturing each crucial pixel to predict the complex topology of lane lines and differentiate the various lane types. Existing methods predominantly rely on deep feature extraction networks with substantial parameters or the fusion of multiple prediction modules, resulting in large model sizes, embedding difficulties, and slow detection speeds. This article proposes a Proportional Feature Pyramid Network (P-FPN) through fusing the weights into the FPN for lane detection. For obtaining a more accurately detecting result, the cross refinement block is introduced in the P-FPN network. The cross refinement block takes the feature maps and anchors as inputs and gradually refines the anchors from high to low level feature maps. In our method, the high-level features are explored to predict lanes coarsely while local-detailed features are leveraged to improve localization accuracy. Extensive experiments on two widely used lane detection datasets, The Chinese Urban Scene Benchmark for Lane Detection (CULane) and the TuSimple Lane Detection Challenge (TuSimple) datasets, demonstrate that the proposed method achieves competitive results compared with several state-of-the-art approaches.

Model validation of hollow embossing rolling for bipolar plate forming

Hollow embossing rolling is a promising forming technology for metallic bipolar plates because of the high achievable production rates. However, the simulation-based process optimization is impeded by the incremental forming character and modeling of fine channel structures, which leads to large model sizes and long computation times. This paper presents a shell-based finite element approach validated by experimental forming tests using a miniaturized test geometry with typical discontinuities and varying channel orientations. The rolling experiments demonstrated that implementing restraining tension effectively decreases wrinkling, allowing successful forming of the selected test geometry by hollow embossing rolling. It was found that representing the manufacturing-related decreased rolling gap combined with the rolling gap changes due to roll system elasticity in the numerical model is essential for model accuracy. An optimized model approach with spring-controlled rollers was developed, which considers the effect of load-dependent rolling gap changes. With this approach the applied model achieves sufficient model accuracy for technological process simulation and optimization.

Lightweight model-based sheep face recognition via face image recording channel.

The accurate identification of individual sheep is a crucial prerequisite for establishing digital sheep farms and precision livestock farming. Currently, deep learning technology provides an efficient and non-contact method for sheep identity recognition. In particular, convolutional neural networks can be used to learn features of sheep faces to determine their corresponding identities. However, the existing sheep face recognition models face problems such as large model size, and high computational costs, making it difficult to meet the requirements of practical applications. In response to these issues, we introduce a lightweight sheep face recognition model called YOLOv7-Sheep Face Recognition (YOLOv7-SFR). Considering the labor-intensive nature associated with manually capturing sheep face images, we developed a face image recording channel to streamline the process and improve efficiency. This study collected facial images of 50 Small-tailed Han sheep through a recording channel. The experimental sheep ranged in age from 1 to 3 yr, with an average weight of 63.1kg. Employing data augmentation methods further enhanced the original images, resulting in a total of 22,000 sheep face images. Ultimately, a sheep face dataset was established. To achieve lightweight improvement and improve the performance of the recognition model, a variety of improvement strategies were adopted. Specifically, we introduced the shuffle attention module into the backbone and fused the Dyhead module with themodel'sdetection head. By combining multiple attention mechanisms, we improved themodel'sability to learn target features. Additionally, the traditional convolutions in the backbone and neck were replaced with depthwise separable convolutions. Finally, leveraging knowledge distillation, we enhanced its performance further by employing You Only Look Once version 7 (YOLOv7) as the teacher model and YOLOv7-SFR as the student model. The training results indicate that our proposed approach achieved the best performance on the sheep face dataset, with a mean average precision@0.5 of 96.9%. The model size and average recognition time were 11.3 MB and 3.6ms, respectively. Compared to YOLOv7-tiny, YOLOv7-SFR showed a 2.1% improvement in mean average precision@0.5, along with a 5.8% reduction in model size and a 42.9% reduction in average recognition time. The research results are expected to drive the practical applications of sheep face recognition technology.

Lightweight and fast visual detection method for 3C assembly

In the context of 3C assembly scenarios, characterized by numerous semi-flexible, heterogeneous, and small slender targets, traditional target detection algorithms face significant challenges such as low accuracy, weak generalization, large model sizes, and slow inference speeds. To address these issues, this study introduces an enhanced method based on the YOLOv5 model, named YOLOv5-GTB. This method integrates Bidirectional Feature Pyramid Network (BiFPN), Ghost lightweight convolution, Vision Transformer (ViT), and adaptive activation function technologies, aiming to improve the accuracy and speed of target detection. Our approach utilizes Ghost convolution to construct a Ghost bottleneck layer, optimizing the feature extraction network while significantly reducing computational costs and enhancing the convolutional neural network’s feature extraction capabilities. Additionally, the Cross-Stage Partial Network (CSPNet) architecture is employed to effectively segment the data flow of the input feature map layer, thereby improving the efficiency of gradient processing. Furthermore, we introduce a fusion structure of CNN and Transformer, leveraging the strengths of convolutional neural networks in local feature extraction and the ViT’s capability in long-range information capture, thereby further enhancing the overall network’s feature extraction performance. Regarding feature fusion, considering the limitations of traditional top-down unidirectional information flow in effectively merging features with both location and semantic information, BiFPN is incorporated into the YOLOv5-GTB. This enhances the fusion of feature layers extracted from both the end of the backbone network and the first module’s fused feature layer, thus improving detection accuracy. Ablation and comparative experiments conducted on the 3C assembly scenario target detection dataset demonstrate the significant advantages of the YOLOv5-GTB model in terms of accuracy and speed. Ultimately, the application of this model to the 3C assembly platform successfully achieves rapid and accurate target recognition in this scenario.

GPT-4: A Review on Advancements and Opportunities in Natural Language Processing

Generative Pre-trained Transformer 4 (GPT-4) is the fourth-generation language model in the GPT series, developed by OpenAI, which promises significant advancements in the field of natural language processing (NLP). In this research article, we have discussed the features of GPT-4, its potential applications, and the challenges that it might face. We have also compared GPT-4 with its predecessor, GPT-3. GPT-4 has a larger model size (more than one trillion), better multilingual capabilities, improved contextual understanding, and reasoning capabilities than GPT-3. Some of the potential applications of GPT-4 include chatbots, personal assistants, language translation, text summarization, and question-answering. However, GPT-4 poses several challenges and limitations such as computational requirements, data requirements, and ethical concerns.

Lightweight DB-YOLO Facemask Intelligent Detection and Android Application Based on Bidirectional Weighted Feature Fusion

Conventional facemask detection algorithms face challenges of insufficient accuracy, large model size, and slow computation speed, limiting their deployment in real-world scenarios, especially on edge devices. Aiming at addressing these issues, we proposed a DB-YOLO facemask intelligent detection algorithm, which is a lightweight solution that leverages bidirectional weighted feature fusion. Our method is built on the YOLOv5 algorithm model, replacing the original YOLOv5 backbone network with the lightweight ShuffleNetv2 to reduce parameters and computational requirements. Additionally, we integrated BiFPN as the feature fusion layer, enhancing the model’s detection capability for objects of various scales. Furthermore, we employed a CARAFE lightweight upsampling factor to improve the model’s perception of details and small-sized objects and the EIOU loss function to expedite model convergence. We validated the effectiveness of our proposed method through experiments conducted on the Pascal VOC2007+2012 and Face_Mask datasets. Our experimental results demonstrate that the DB-YOLO model boasts a compact size of approximately 1.92 M. It achieves average precision values of 70.1% and 93.5% on the Pascal VOC2007+2012 and Face_Mask datasets, respectively, showcasing a 2.3% improvement in average precision compared to the original YOLOv5s. Furthermore, the model’s size is reduced by 85.8%. We also successfully deployed the model on Android devices using the NCNN framework, achieving a detection speed of up to 33 frames per second. Compared to lightweight algorithm models like YOLOv5n, YOLOv4-Tiny, and YOLOv3-Tiny, DB-YOLO not only reduces the model’s size but also effectively improves detection accuracy, exhibiting excellent practicality and promotional value on edge devices.

Model pruning enables localized and efficient federated learning for yield forecasting and data sharing

Federated Learning (FL) presents a decentralized approach to model training in the agri-food sector and offers the potential for improved machine learning performance, while ensuring the safety and privacy of individual farms or data silos. However, the conventional FL approach has two major limitations. First, the heterogeneous data on individual silos can cause the global model to perform well for some clients but not all, as the update direction on some clients may hinder others after they are aggregated. Second, it is lacking with respect to the efficiency perspective concerning communication costs during FL and large model sizes. This paper proposes a new technical solution that utilizes network pruning on client models and aggregates the pruned models. This method enables local models to be tailored to their respective data distribution and mitigate the data heterogeneity present in agri-food data. Moreover, it allows for more compact models that consume less data during transmission. We experiment with a soybean yield forecasting dataset and find that this approach can improve inference performance by 15.5% to 20% compared to FedAvg, while reducing local model sizes by up to 84% and the data volume communicated between the clients and the server by 57.1% to 64.7%. Our method demonstrates the potential to use efficient models that are more environmentally friendly to support the agri-food sector’s transition to net zero. Future enhancements of this method could further optimize distributed learning in agri-food, enhancing sustainability and applicability.

Lightweight target speaker separation network based on joint training

Target speaker separation aims to separate the speech components of the target speaker from mixed speech and remove extraneous components such as noise. In recent years, deep learning-based speech separation methods have made significant breakthroughs and have gradually become mainstream. However, these existing methods generally face problems with system latency and performance upper limits due to the large model size. To solve these problems, this paper proposes improvements in the network structure and training methods to enhance the model’s performance. A lightweight target speaker separation network based on long-short-term memory (LSTM) is proposed, which can reduce the model size and computational delay while maintaining the separation performance. Based on this, a target speaker separation method based on joint training is proposed to achieve the overall training and optimization of the target speaker separation system. Joint loss functions based on speaker registration and speaker separation are proposed for joint training of the network to further improve the system’s performance. The experimental results show that the lightweight target speaker separation network proposed in this paper has better performance while being lightweight, and joint training of the target speaker separation network with our proposed loss function can further improve the separation performance of the original model.

LayerCFL: an efficient federated learning with layer-wised clustering

Federated Learning (FL) suffers from the Non-IID problem in practice, which poses a challenge for efficient and accurate model training. To address this challenge, prior research has introduced clustered FL (CFL), which involves clustering clients and training them separately. Despite its potential benefits, CFL can be computationally and communicationally expensive when the data distribution is unknown beforehand. This is because CFL involves the entire neural networks of involved clients in computing the clusters during training, which can become increasingly time-consuming with large-sized models. To tackle this issue, this paper proposes an efficient CFL approach called LayerCFL that employs a Layer-wised clustering technique. In LayerCFL, clients are clustered based on a limited number of layers of neural networks that are pre-selected using statistical and experimental methods. Our experimental results demonstrate the effectiveness of LayerCFL in mitigating the impact of Non-IID data, improving the accuracy of clustering, and enhancing computational efficiency.

YOLO sparse training and model pruning for street view house numbers recognition

This paper proposes a YOLO (You Only Look Once) sparse training and model pruning technique for recognizing house numbers in street view images. YOLO is a popular object detection algorithm that has achieved state-of-the-art performance in various computer vision tasks. However, its large model size and computational complexity limit its deployment on resource-constrained devices such as smartphones and embedded systems. To address this issue, we use a sparse training technique that trains YOLO with L1 norm regularization to encourage the network to learn sparse representations. This results in a significant reduction in the number of parameters and computation without sacrificing accuracy. Furthermore, we apply a model pruning technique to the sparse-trained model to reduce the model size and computation. We evaluate our proposed method on the SVHN (Street View House Numbers) dataset. We show that it performs comparably to the original YOLO model while reducing the model size by 5% and the computation time by 7%. Overall, our proposed YOLO sparse training and model pruning technique provides an effective solution for deploying YOLO-based object detection models on resource-constrained devices.

Neuro-distributed cognitive adaptive optimization for training neural networks in a parallel and asynchronous manner

Distributed Machine learning has delivered considerable advances in training neural networks by leveraging parallel processing, scalability, and fault tolerance to accelerate the process and improve model performance. However, training of large-size models has exhibited numerous challenges, due to the gradient dependence that conventional approaches integrate. To improve the training efficiency of such models, gradient-free distributed methodologies have emerged fostering the gradient-independent parallel processing and efficient utilization of resources across multiple devices or nodes. However, such approaches, are usually restricted to specific applications, due to their conceptual limitations: computational and communicational requirements between partitions, limited partitioning solely into layers, limited sequential learning between the different layers, as well as training a potential model in solely synchronous mode. In this paper, we propose and evaluate, the Neuro-Distributed Cognitive Adaptive Optimization (ND-CAO) methodology, a novel gradient-free algorithm that enables the efficient distributed training of arbitrary types of neural networks, in both synchronous and asynchronous manner. Contrary to the majority of existing methodologies, ND-CAO is applicable to any possible splitting of a potential neural network, into blocks (partitions), with each of the blocks allowed to update its parameters fully asynchronously and independently of the rest of the blocks. Most importantly, no data exchange is required between the different blocks during training with the only information each block requires is the global performance of the model. Convergence of ND-CAO is mathematically established for generic neural network architectures, independently of the particular choices made, while four comprehensive experimental cases, considering different model architectures and image classification tasks, validate the algorithms’ robustness and effectiveness in both synchronous and asynchronous training modes. Moreover, by conducting a thorough comparison between synchronous and asynchronous ND-CAO training, the algorithm is identified as an efficient scheme to train neural networks in a novel gradient-independent, distributed, and asynchronous manner, delivering similar – or even improved results in Loss and Accuracy measures.