Memory Requirements Research Articles

FastPTM: Fast weights loading of pre-trained models for parallel inference service provisioning

Pre-trained models (PTMs) have demonstrated great success in a variety of NLP and CV tasks and have become a significant development in the field of deep learning. However, the large memory and high computational requirements associated with PTMs can increase the cost and time of inference, limiting their service provisioning in practical applications. To improve the Quality of Service (QoS) of PTM applications by reducing waiting and response times, we propose the FastPTM framework. This general framework aims to accelerate PTM inference services in a multi-tenant environment by reducing model loading time and switching overhead on GPUs. The framework utilizes a fast weights loading method based on weights and model separation of PTMs to efficiently accelerate parallel inference services in resource-constrained environments. Furthermore, an online scheduling algorithm is designed to reduce the inference service time. The results of the experiments indicate that FastPTM can improve the throughput of inference services by an average of 4x and up to 8.2x, while reducing the number of switches by 4.7x and the number of overtimes by 15.3x.

RadicalPy: A Tool for Spin Dynamics Simulations.

Radical pairs (electron-hole pairs, polaron pairs) are transient reaction intermediates that are found and exploited in all areas of science, from the hard realm of physics in the form of organic semiconductors, spintronics, quantum computing, and solar cells to the soft domain of chemistry and biology under the guise of chemical reactions in solution, biomimetic systems, and quantum biology. Quantitative analysis of radical pair phenomena has historically been successful by a few select groups. With this in mind, we present an intuitive open-source framework in the Python programming language that provides classical, semiclassical, and quantum simulation methodologies. A radical pair kinetic rate equation solver, Monte Carlo-based spin dephasing rate estimations, and molecule database functionalities are implemented. We introduce the kine-quantum method, a new approach that amalgamates classical rate equations, semiclassical, and quantum techniques. This method resolves the prohibitively large memory requirement issues of quantum approaches while achieving higher accuracy, and it also offers wavelength-resolved simulations, producing time- and wavelength-resolved magnetic field effect simulations. Model examples illustrate the versatility and ease of use of the software, including the new approach applied to the magnetosensitive absorption and fluorescence of flavin adenine dinucleotide photochemistry, spin-spin interaction estimation from molecular dynamics simulations on radical pairs inside reverse micelles, radical pair anisotropy inside proteins, and triplet exciton pairs in anthracene crystals. The intuitive interface also allows this software to be used as a teaching or learning aid for those interested in the field of spin chemistry. Furthermore, the software aims to be modular and extensible, with the aim to standardize how spin dynamics simulations are performed.

Benchmarking In-Sensor Machine Learning Computing: An Extension to the MLCommons-Tiny Suite

This paper proposes a new benchmark specifically designed for in-sensor digital machine learning computing to meet an ultra-low embedded memory requirement. With the exponential growth of edge devices, efficient local processing is essential to mitigate economic costs, latency, and privacy concerns associated with the centralized cloud processing. Emerging intelligent sensors equipped with computing assets to run neural network inferences and embedded in the same package, which hosts the sensing elements, present new challenges due to their limited memory resources and computational skills. This benchmark evaluates models trained with Quantization Aware Training (QAT) and compares their performance with Post-Training Quantization (PTQ) across three use cases: Human Activity Recognition (HAR) by means of the SHL dataset, Physical Activity Monitoring (PAM) by means of the PAMAP2 dataset, and superficial electromyography (sEMG) regression with the NINAPRO DB8 dataset. The results demonstrate the effectiveness of QAT over PTQ in most scenarios, highlighting the potential for deploying advanced AI models on highly resource-constrained sensors. The INT8 versions of the models always outperformed their FP32, regarding memory and latency reductions, except for the activations for CNN. The CNN model exhibited reduced memory usage and latency with respect to its Dense counterpart, allowing it to meet the stringent 8KiB data RAM and 32 KiB program RAM limits of the ISPU. The TCN model proved to be too large to fit within the memory constraints of the ISPU, primarily due to its greater capacity in terms of number of parameters, designed for processing more complex signals like EMG. This benchmark aims to guide the development of efficient AI solutions for In-Sensor Machine Learning Computing, fostering innovation in the field of Edge AI benchmarking, such as the one conducted by the MLCommons-Tiny working group.

Fast‐MedNeXt: Accelerating the MedNeXt Architecture to Improve Brain Tumour Segmentation Efficiency

ABSTRACTWith the rapid development of medical imaging technology, 3D image segmentation technology has gradually become a mainstream method, especially in brain tumour detection and diagnosis showing its unique advantages. The technique makes full use of 3D spatial information to locate and analyze tumours more accurately, thus playing an important role in improving diagnostic accuracy, optimising treatment planning and promoting research. However, it also suffers from significant computational expenditure and delayed processing pace. In this paper, we propose an innovative optimisation scheme to address this problem. We thoroughly investigate the MedNeXt network and propose Fast‐MedNeXt, which aims to increase the processing speed while maintaining accuracy. First, we introduce the partial convolution (PConv) technique, which replaces the deep convolutional layers in the network. This improvement effectively reduces computation and memory requirements while maintaining efficient feature extraction. Second, based on PConv, we propose PConv‐Down and PConv‐Up modules, which are applied to the up‐sampling and down‐sampling modules to further optimise the network structure and improve efficiency. To confirm the efficacy of the approach, we carried out a sequence of tests in the multimodal brain tumour segmentation challenge 2021 (BraTS2021). By comparing with the MedNeXt series network, the Fast‐MedNeXt reduced the latency by 22.1%, 20.5%, 15.8%, and 11.4% respectively, while the average accuracy also increased by 0.475% and 0.2% respectively. These significant performance improvements demonstrate the effectiveness of Fast‐MedNeXt in 3D medical image segmentation tasks and provide a new and more efficient solution for the field.

Using the Discrete Wavelet Transform for Lossy On‐the‐Fly Compression of GPU Fluid Simulations

ABSTRACTHigh‐performance computing in fluid dynamics frequently confronts substantial memory demands, especially in large‐scale applications. Data compression techniques can alleviate these memory constraints, but introduce new challenges. This paper introduces an innovative on‐the‐fly low‐overhead lossy compression technique tailored for GPU‐based fluid simulations, utilizing the discrete wavelet transform (DWT). The technique is applicable to any numerical scheme where the data is stored on a regular grid and the time step is computed using a stencil. Our approach significantly diminishes memory requirements, achieving up to a 10‐fold long‐term reduction on a D3Q27 simulation, while minimally impacting the simulation accuracy. The methodology is built around careful design choices to achieve a satisfactory compression ratio/speed trade‐off. It effectively maintains mass conservation and accurately preserves essential discontinuities in simulations. Extensive testing with a D3Q27 Lattice‐Boltzmann method (LBM) simulation on a single GPU has shown that large‐scale grids can be processed with minimal impact on the simulation accuracy and acceptable compression times. This compression technique demonstrates a robust capability to handle memory limitations in fluid simulations, opening the door to more complex and larger‐scale simulations.

Point-enhanced convolutional neural network: A novel deep learning method for transonic wall-bounded flows

Low order models can be used to accelerate engineering design processes. Ideally, these surrogates should meet the conflicting requirements of large design space coverage, high accuracy and fast evaluation. Within the context of aerospace applications at transonic conditions, this can be challenging due to the associated non-linearity of the flow regime. Different methods have been investigated in the past to predict the flow-field around shapes such as airfoils or cylinders. However, they usually have reduced spatial resolution, limiting the prediction capabilities within the boundary layer which is of interest for transonic wall-bounded flows. This work proposes a novel Point-Enhanced Convolutional Neural Network (PCNN) method that combines the advantages of the well-established PointNet and convolutional neural network approaches. The PCNN model has relatively low memory requirements in the training process, preserves the spatial correlation in the domain and has the same resolution as a traditional computational method. The architecture is used for the flow-field prediction of civil aero-engine nacelles in which it is demonstrated that the flow features of peak isentropic Mach number (Mis), pre-shock isentropic Mach number and shock location (X/Lnac) are captured within ΔMis = 0.02, ΔMis=0.04, ΔX/Lnac=0.007, respectively. The PCNN model successfully predicts the integral parameters of the boundary layer, in which the incompressible displacement thickness, momentum thickness and shape factor are typically within 5% of the CFD. Overall, the PCNN method is demonstrated for transonic wall-bounded flows for a range of flow physics that include shock waves and shock-induced separation.

A Supervised Ml Algorithm for Detecting and Predicting Fraud Credit Card Transactions

A Credit card fraud presents an escalating threat in today's digital economy, leading to significant financial losses for consumers, businesses, and financial institutions. Traditional detection methods are increasingly inadequate due to the rise in online transactions and the evolving complexity of fraudulent activities. This research aims to create a reliable supervised machine learning model designed to detect and predict fraudulent credit card transactions in real-time. Our approach leverages advanced algorithms and extensive datasets to enhance transaction security, thereby mitigating financial risks. Current systems, like FICO's Fraud Detection System (FDS), utilize a range of supervised learning techniques, including Decision Trees, Random Forests, and Neural Networks. However, they face limitations, including data quality issues, high false positive rates, and the need for continuous retraining to adapt to evolving fraud patterns. To address these challenges, we propose an innovative anomaly detection method based on the Isolation Forest algorithm. Unlike traditional methods that rely on distance and density measures, Isolation Forests isolate anomalies by randomly selecting features and split values, making it more efficient in identifying fraudulent transactions, especially in imbalanced datasets. The proposed system boasts low linear time complexity and minimal memory requirements, enabling the construction of high-performing models even with large datasets. The research demonstrates that the Isolation Forest approach significantly outperforms traditional models in detecting credit card fraud, offering a promising solution for enhancing security in the financial ecosystem.

Uncertainty propagation of initial conditions in thermal models

The operation of machine tools often demands a highly accurate knowledge of the tool center point’s (TCP) position. The displacement of the TCP over time can be inferred from coupled thermal models, which comprise a set of geometrically coupled heat equations. Each of these equations represents the temperature in part of the machine, and they are often formulated on complicated geometries. The accuracy of the TCP prediction depends highly on the accuracy of the model parameters, such as heat exchange parameters, and the initial temperature. Thus it is of utmost interest to determine the influence of these parameters on the TCP displacement prediction. In turn, the accuracy of the parameter estimate is essentially determined by the measurement accuracy and the sensor placement. Determining the accuracy of a given sensor configuration is a key prerequisite of optimal sensor placement. In this paper, we develop a thermal model for the discretization of realistic machine tool. On top of this model we propose two numerical algorithms to evaluate any given thermal sensor configuration with respect to its accuracy. We compute the posterior variances from the posterior covariance matrix with respect to an uncertain initial temperature field. The full matrix is dense and potentially very large, depending on the size of the discretized model. We first present a way to compute this approximation, which relies on the computation of the model sensitivities with respect to the initial values. Additionally, we present a low-rank tensor method which exploits the underlying system structure. We compare the efficiency of both algorithms with respect to runtime and memory requirements and discuss their respective advantages with regard to optimal sensor placement problems.

Analyzing One- and Two-bit Data to Reduce Memory Requirements for F -statistic-based Gravitational Wave Searches

Searches for continuous-wave gravitational radiation in data collected by modern long-baseline interferometers, such as the Laser Interferometer Gravitational-wave Observatory (LIGO), the Virgo interferometer, and the Kamioka Gravitational Wave Detector, can be memory intensive. A digitization scheme is described that reduces the 64-bit interferometer output to a one- or two-bit data stream while minimizing distortion and achieving considerable reduction in storage and input/output cost. For the representative example of the coherent, maximum-likelihood matched filter known as the F -statistic, it is found using Monte Carlo simulations that the injected signal only needs to be ≈24% stronger (for one-bit data) and ≈6.4% stronger (for two-bit data with optimal thresholds) than a 64-bit signal in order to be detected with 90% probability in Gaussian noise. The foregoing percentages do not change significantly when the signal frequency decreases secularly, or when the noise statistics are not Gaussian, as verified with LIGO Science Run 6 data.

Use of Deep-Learning-Accelerated Gradient Approximation for Reservoir Geological Parameter Estimation

The estimation of space-varying geological parameters is often not computationally affordable for high-dimensional subsurface reservoir modeling systems. The adjoint method is generally regarded as an efficient approach for obtaining analytical gradient and, thus, proceeding with the gradient-based iteration algorithm; however, the infeasible memory requirement and computational demands strictly prohibit its generic implementation, especially for high-dimensional problems. The autoregressive neural network (aNN) model, as a nonlinear surrogate approximation, has gradually received increasing popularity due to significant reduction of computational cost, but one prominent limitation is that the generic application of aNN to large-scale reservoir models inevitably poses challenges in the training procedure, which remains unresolved. To address this issue, model-order reduction could be a promising strategy, which enables us to train the neural network in a very efficient manner. A very popular projection-based linear reduction method, i.e., propel orthogonal decomposition (POD), is adopted to achieve dimensionality reduction. This paper presents an architecture of a projection-based autoregressive neural network that efficiently derives an easy-to-use adjoint model by the use of an auto-differentiation module inside the popular deep learning frameworks. This hybrid neural network proxy, referred to as POD-aNN, is capable of speeding up derivation of reduced-order adjoint models. The performance of POD-aNN is validated through a synthetic 2D subsurface transport model. The use of POD-aNN significantly reduces the computation cost while the accuracy remains. In addition, our proposed POD-aNN can easily obtain multiple posterior realizations for uncertainty evaluation. The developed POD-aNN emulator is a data-driven approach for reduced-order modeling of nonlinear dynamic systems and, thus, should be a very efficient modeling tool to address many engineering applications related to intensive simulation-based optimization.

VAN-DAMME: GPU-accelerated and symmetry-assisted quantum optimal control of multi-qubit systems

We present an open-source software package, VAN-DAMME (Versatile Approaches to Numerically Design, Accelerate, and Manipulate Magnetic Excitations), for massively-parallelized quantum optimal control (QOC) calculations of multi-qubit systems. To enable large QOC calculations, the VAN-DAMME software package utilizes symmetry-based techniques with custom GPU-enhanced algorithms. This combined approach allows for the simultaneous computation of hundreds of matrix exponential propagators that efficiently leverage the intra-GPU parallelism found in high-performance GPUs. In addition, to maximize the computational efficiency of the VAN-DAMME code, we carried out several extensive tests on data layout, computational complexity, memory requirements, and performance. These extensive analyses allowed us to develop computationally efficient approaches for evaluating complex-valued matrix exponential propagators based on Padé approximants. To assess the computational performance of our GPU-accelerated VAN-DAMME code, we carried out QOC calculations of systems containing 10 - 15 qubits, which showed that our GPU implementation is 18.4× faster than the corresponding CPU implementation. Our GPU-accelerated enhancements allow efficient calculations of multi-qubit systems, which can be used for the efficient implementation of QOC applications across multiple domains. Program summaryProgram Title: VAN-DAMMECPC Library link to program files::https://doi.org/10.17632/zcgw2n5bjf.1Licensing provisions: GNU General Public License 3Programming language: C++ and CUDANature of problem: The VAN-DAMME software package utilizes GPU-accelerated routines and new algorithmic improvements to compute optimized time-dependent magnetic fields that can drive a system from a known initial qubit configuration to a specified target state with a large (≈1) transition probability.Solution method: Quantum control, GPU acceleration, analytic gradients, matrix exponential, and gradient ascent optimization.

LDF-BNN: A Real-Time and High-Accuracy Binary Neural Network Accelerator Based on the Improved BNext.

Significant progress has been made in industrial defect detection due to the powerful feature extraction capabilities of deep neural networks (DNNs). However, the high computational cost and memory requirement of DNNs pose a great challenge to the deployment of industrial edge-side devices. Although traditional binary neural networks (BNNs) have the advantages of small storage space requirements, high parallel computing capability, and low power consumption, the problem of significant accuracy degradation cannot be ignored. To tackle these challenges, this paper constructs a BNN with layered data fusion mechanism (LDF-BNN) based on BNext. By introducing the above mechanism, it strives to minimize the bandwidth pressure while reducing the loss of accuracy. Furthermore, we have designed an efficient hardware accelerator architecture based on this mechanism, enhancing the performance of high-accuracy BNN models with complex network structures. Additionally, the introduction of multi-storage parallelism alleviates the limitations imposed by the internal transfer rate, thus improving the overall computational efficiency. The experimental results show that our proposed LDF-BNN outperforms other methods in the comprehensive comparison, achieving a high accuracy of 72.23%, an image processing rate of 72.6 frames per second (FPS), and 1826 giga operations per second (GOPs) on the ImageNet dataset. Meanwhile, LDF-BNN can also be well applied to defect detection dataset Mixed WM-38, achieving a high accuracy of 98.70%.

Elliptic curves cryptography for lightweight devices in IoT system

The Internet of Things (IoT) is revolutionizing industries by connecting billions of devices, ranging from home appliances to industrial machines, to the internet. This transformation enables unprecedented automation, data collection, and process optimization. However, the widespread adoption of IoT also introduces significant security challenges, especially for lightweight devices with limited computational power and energy resources. Ensuring secure communication and data integrity in such constrained environments is critical. This paper explores and compares the performance of implemented algorithms within IoT devices. A comparison between Rivest-Shamir-Adleman (RSA) and Elliptic Curve Cryptography (ECC) implementations is made to identify a lightweight, secure, and efficient solution for IoT environments. Our results indicate that ECC outperforms RSA in terms of memory requirements, overall energy consumption, key sizes, signature generation time, key generation and execution time, and decryption time in a resource-constrained environment. However, RSA performs better in signature verification and encryption.

Parallel Lossless Compression of Raw Bayer Images on FPGA-Based High-Speed Camera

Digital image compression is applied to reduce camera bandwidth and storage requirements, but real-time lossless compression on a high-speed high-resolution camera is a challenging task. The article presents hardware implementation of a Bayer colour filter array lossless image compression algorithm on an FPGA-based camera. The compression algorithm reduces colour and spatial redundancy and employs Golomb–Rice entropy coding. A rule limiting the maximum code length is introduced for the edge cases. The proposed algorithm is based on integer operators for efficient hardware implementation. The algorithm is first verified as a C++ model and later implemented on AMD-Xilinx Zynq UltraScale+ device using VHDL. An effective tree-like pipeline structure is proposed to concatenate codes of compressed pixel data to generate a bitstream representing data of 16 parallel pixels. The proposed parallel compression achieves up to 56% reduction in image size for high-resolution images. Pipelined implementation without any state machine ensures operating frequencies up to 320 MHz. Parallelised operation on 16 pixels effectively increases data throughput to 40 Gbit/s while keeping the total memory requirements low due to real-time processing.

Deep Learning Algorithms for Cryptocurrency Price Prediction: a Comparative Analysis

Over the past years, cryptocurrencies have experienced a surge in popularity within the financial markets. As of today, besides being considered for investment purposes, they also serve as a widely accepted form of currency for everyday transactions. Due to the intricate characteristics of financial markets and their dependence on various factors to determine the prices of stocks and assets, the ability to predict such price is crucial to make investment choices, especially in terms of cryptocurrencies. In this work, a comparative analysis on the suitability of Deep Learning (DL) algorithms (effective for time series forecasting) in predicting the price of three cryptocurrencies (namely: Bitcoin, BTC; Ethereum, ETH; and Ripple, XRP) is assessed in terms of both short-term and long-term prediction accuracy. The results, evaluated using Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and coefficient of determination (denoted as \(R^{2}\) ), reveal that: Transformer is generally more effective for short-term forecasts and also performs well for long-term predictions; Convolutional Neural Network-Recurrent Neural Network (CNN-RNN) demonstrates the lowest complexity in terms of number of Multiply and ACcumulate (MAC) operations; SimpleRNN has the fewest parameters and the smallest FLASH memory requirement. Overall, CNN-Gated Recurrent Unit (CNN-GRU) provides the best joint accuracy-complexity for predicting BTC and ETH prices, whereas CNN-RNN yields superior results for XRP price prediction.

Investigating the Performance of the Informer Model for Streamflow Forecasting

Recent studies have shown the potential of transformer-based neural networks in increasing prediction capacity. However, classical transformers present several problems such as computational time complexity and high memory requirements, which make Long Sequence Time-Series Forecasting (LSTF) challenging. The contribution to the prediction of time series of flood events using deep learning techniques is examined, with a particular focus on evaluating the performance of the Informer model (a particular implementation of transformer architecture), which attempts to address the previous issues. The predictive capabilities of the Informer model are explored and compared to statistical methods, stochastic models and traditional deep neural networks. The accuracy, efficiency as well as the limits of the approaches are demonstrated via numerical benchmarks relating to real river streamflow applications. Using daily flow data from the River Test in England as the main case study, we conduct a rigorous evaluation of the Informer efficacy in capturing the complex temporal dependencies inherent in streamflow time series. The analysis is extended to encompass diverse time series datasets from various locations (>100) in the United Kingdom, providing insights into the generalizability of the Informer. The results highlight the superiority of the Informer model over established forecasting methods, especially regarding the LSTF problem. For a forecast horizon of 168 days, the Informer model achieves an NSE of 0.8 and maintains a MAPE below 10%, while the second-best model (LSTM) only achieves −0.63 and 25%, respectively. Furthermore, it is observed that the dependence structure of time series, as expressed by the climacogram, affects the performance of the Informer network.

Pulmonary Fissure Segmentation in CT Images Using Image Filtering and Machine Learning.

Both lung lobe segmentation and lung fissure segmentation are useful in the clinical diagnosis and evaluation of lung disease. It is often of clinical interest to quantify each lobe separately because many diseases are associated with specific lobes. Fissure segmentation is important for a significant proportion of lung lobe segmentation methods, as well as for assessing fissure completeness, since there is an increasing requirement for the quantification of fissure integrity. We propose a method for the fully automatic segmentation of pulmonary fissures on lung computed tomography (CT) based on U-Net and PAN models using a Derivative of Stick (DoS) filter for data preprocessing. Model ensembling is also used to improve prediction accuracy. Our method achieved an F1 score of 0.916 for right-lung fissures and 0.933 for left-lung fissures, which are significantly higher than the standalone DoS results (0.724 and 0.666, respectively). We also performed lung lobe segmentation using fissure segmentation. The lobe segmentation algorithm shows results close to those of state-of-the-art methods, with an average Dice score of 0.989. The proposed method segments pulmonary fissures efficiently and have low memory requirements, which makes it suitable for further research in this field involving rapid experimentation.

A novel lightweight deep learning model based on SqueezeNet architecture for viral lung disease classification in X-ray and CT images

COVID-19 has affected hundreds of millions of individuals, seriously harming the global population’s health, welfare, and economy. Furthermore, health facilities are severely overburdened due to the record number of COVID-19 cases, which makes prompt and accurate diagnosis difficult. Automatically identifying infected individuals and promptly placing them under special care is a critical step in reducing the burden of such issues. Convolutional Neural Networks (CNN) and other machine learning techniques can be utilized to address this demand. Many existing Deep learning models, albeit producing the intended outcomes, were developed using millions of parameters, making them unsuitable for use on devices with constrained resources. Motivated by this fact, a novel lightweight deep learning model based on Efficient Channel Attention (ECA) module and SqueezeNet architecture, is developed in this work to identify COVID-19 patients from chest X-ray and CT images in the initial phases of the disease. After the proposed lightweight model was tested on different datasets with two, three and four classes, the results show its better performance over existing models. The outcomes shown that, in comparison to the current heavyweight models, our models reduced the cost and memory requirements for computing resources dramatically, while still achieving comparable performance. These results support the notion that proposed model can help diagnose Covid-19 in patients by being easily implemented on low-resource and low-processing devices.

Schottky barrier memory based on heterojunction bandgap engineering for high-density and low-power retention

Dynamic random-access memory (DRAM) has been scaled down to meet high-density, high-speed, and low-power memory requirements. However, conventional DRAM has limitations in achieving memory reliability, especially sufficient capacitance to distinguish memory states. While there have been attempts to enhance capacitor technology, these solutions increase manufacturing cost and complexity. Additionally, Silicon-based capacitorless memories have been reported, but they still suffer from serious difficulties regarding reliability and power consumption. Here, we propose a novel Schottky barrier memory (SBRAM), which is free of the complex capacitor structure and features a heterojunction based on bandgap engineering. SBRAM can be configured as vertical cross-point arrays, which enables high-density integration with a 4F2 footprint. In particular, the Schottky junction significantly reduces the reverse leakage current, preventing sneak current paths that cause leakage currents and readout errors during array operation. Moreover, the heterojunction physically divides the storage region into two regions, resulting in three distinct resistive states and inducing a gradual current slope to ensure sufficient holding margin. These states are determined by the holding voltage (Vhold) applied to the programmed device. When the Vhold is 1.1 V, the programmed state can be maintained with an exceptionally low current of 35.7 fA without a refresh operation.

TS‐Mixer: A lightweight text representation model based on context awareness

AbstractLarge pre‐trained models (PTMs) have shown their powerful ability in multiple natural language processing tasks. However, using them in practical application remains a challenge due to the significant computational cost and memory requirements. In order to achieve the balance of computational cost and accuracy, MLP architecture can be used as an alternative to the self‐attention module, such as pNLP‐Mixer and Hyper‐Mixer. Experiments indicate that, MLP‐based models can attain competitive performance with low cost. They maintain the balance of computation cost and accuracy successfully, yet, this is at the expense of not being able to capture short‐range dependencies. In this paper, a novel MLP‐based model, termed TS‐Mixer, is proposed which can capture local dependencies by shifting operation. Compared with other MLP‐based models, the parameters of TS‐Mixer are decoupled from the sequence length, hence it has a smaller model size in long sequence tasks. In addition, TS‐Mixer has linear computational complexity, therefore it can be used as a lightweight alternative to the self‐attention model. Experiments show that the TS‐Mixer outperforms other MLP‐based models, which achieves higher accuracy with fewer parameters in multiple downstream tasks. Notably, compared with pre‐trained models, TS‐Mixer can reach more than 90% of their accuracy with 1% or even one thousandth of the parameters (0.174 ~ 1.2 M). These results demonstrate that TS‐Mixer can achieve a better balance between the computing resources and accuracy. Code is available at: https://github.com/wyl-privacy-project/TS-Mixer.