Quantization Bits Research Articles

A memristive all-inclusive hypernetwork for parallel analog deployment of full search space architectures

In recent years, there has been a significant advancement in memristor-based neural networks, positioning them as a pivotal processing-in-memory deployment architecture for a wide array of deep learning applications. Within this realm of progress, the emerging parallel analog memristive platforms are prominent for their ability to generate multiple feature maps in a single processing cycle. However, a notable limitation is that they are specifically tailored for neural networks with fixed structures. As an orthogonal direction, recent research reveals that neural architecture should be specialized for tasks and deployment platforms. Building upon this, the neural architecture search (NAS) methods effectively explore promising architectures in a large design space. However, these NAS-based architectures are generally heterogeneous and diversified, making it challenging for deployment on current single-prototype, customized, parallel analog memristive hardware circuits. Therefore, investigating memristive analog deployment that overrides the full search space is a promising and challenging problem. Inspired by this, and beginning with the DARTS search space, we study the memristive hardware design of primitive operations and propose the memristive all-inclusive hypernetwork that covers 2×1025 network architectures. Our computational simulation results on 3 representative architectures (DARTS-V1, DARTS-V2, PDARTS) show that our memristive all-inclusive hypernetwork achieves promising results on the CIFAR10 dataset (89.2% of PDARTS with 8-bit quantization precision), and is compatible with all architectures in the DARTS full-space. The hardware performance simulation indicates that the memristive all-inclusive hypernetwork costs slightly more resource consumption (nearly the same in power, 22%∼25% increase in Latency, 1.5× in Area) relative to the individual deployment, which is reasonable and may reach a tolerable trade-off deployment scheme for industrial scenarios.

Enhancing physical layer security with reconfigurable intelligent surfaces and friendly jamming: A secrecy analysis

Despite its numerous advantages in enhancing wireless systems’ reliability and secrecy performance, the implementation of reconfigurable intelligent surfaces (RIS) is faced with several challenges, such as optimizing RIS phase-shift configuration to maximize the secrecy performance, particularly in multi-antenna and jamming-aided communication systems, and with the presence of phase quantization errors (PQEs) impairment due to the finite precision. Furthermore, it is crucial to provide an analytical evaluation for RIS-and-jamming-aided multi-antenna schemes. In this paper, the secrecy of a dual-hop wireless communication system, assisted by an RIS in each hop, is quantified. In particular, a multi-antenna relay assists the source node’s communication with a multi-antenna destination. Under the presence of PQEs and several eavesdroppers in the second hop, an RIS splitting technique into equal-size areas, along with friendly jamming, is used to maximize the received legitimate signal power at the relay and destination nodes and disrupt the eavesdroppers. The scheme’s secrecy level is evaluated by deriving novel approximate and asymptotic expressions of the secrecy outage probability (SOP) metric in terms of the main network parameters. Results show that the secrecy is significantly enhanced by increasing the jamming power, the number of reflective elements (REs), or the number of antennas at the relay. In particular, an SOP of 10−6 is reached with 2 receive antennas at the relay and destination, 25 REs per each RIS zone, and −30 dB of normalized jamming power-to-noise ratio even when the legitimate links’ average normalized signal-to-noise ratios are less than the eavesdropper’s one (i.e., strong eavesdropping). It is also shown that the number of quantization bits does not influence the secrecy when exceeding 3 bits. Lastly, the increase in the number of eavesdroppers does not yield a significant secrecy loss.

Performance Analysis of Deep Learning Model-Compression Techniques for Audio Classification on Edge Devices

Audio classification using deep learning models, which is essential for applications like voice assistants and music analysis, faces challenges when deployed on edge devices due to their limited computational resources and memory. Achieving a balance between performance, efficiency, and accuracy is a significant obstacle to optimizing these models for such constrained environments. In this investigation, we evaluate diverse deep learning architectures, including Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM), for audio classification tasks on the ESC 50, UrbanSound8k, and Audio Set datasets. Our empirical findings indicate that Mel spectrograms outperform raw audio data, attributing this enhancement to their synergistic alignment with advanced image classification algorithms and their congruence with human auditory perception. To address the constraints of model size, we apply model-compression techniques, notably magnitude pruning, Taylor pruning, and 8-bit quantization. The research demonstrates that a hybrid pruned model achieves a commendable accuracy rate of 89 percent, which, although marginally lower than the 92 percent accuracy of the uncompressed CNN, strikingly illustrates an equilibrium between efficiency and performance. Subsequently, we deploy the optimized model on the Raspberry Pi 4 and NVIDIA Jetson Nano platforms for audio classification tasks. These findings highlight the significant potential of model-compression strategies in enabling effective deep learning applications on resource-limited devices, with minimal compromise on accuracy.

A multi-channel ECG signal deep compressive sensing method using Treeshaped Autoecoder based on multiscale feature fusion

Conventionally, multi-lead Electrocardiogram (ECG) signals are recorded and stored using high sensing rates and high precision, resulting in huge data volumes and placing greater pressure on storage and transmission resources. Compressive sensing (CS) allows efficient encoding and decoding of signals through sparse representation, measurement and reconstruction for compressed storage and transmission of ECG signals. Traditional CS methods generally requires manually selecting the features or dictionaries used in sparse representations, which lacks adaptivity and flexibility, and the complexity of the reconstruction process limits the application of CS in some scenarios with high real-time requirements. In this paper, we propose a deep compressive sensing framework for processing multi-lead ECG signals by combining CS and deep learning, which is based on multi-scale feature fusion to construct a binary tree-shaped autoencoder architecture to achieve efficient compression and reconstruction of ECG signals. Experiments on the PTB diagnostic ECG database show that the proposed method achieves “very good” reconstruction quality at low sensing rates, with PRD and SNR of 1.77 % and 35.67 dB, respectively, when SR = 20 %. In addition, we investigate how the number of quantization bits affects the quality of reconstructed signals. After analysis, we found that 8-bit quantization can be used in practical applications to ensure the quality of reconstructed ECG signals, while decrease the bit rate of the data, improving the transmission efficiency, and reduce energy consumption.

Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models

As the size of large language models (LLMs) continues to grow, model compression without sacrificing accuracy has become a crucial challenge for deployment. While some quantization methods, such as GPTQ, have made progress in achieving acceptable 4-bit weight-only quantization, attempts at lower-bit quantization often result in severe performance degradation. In this paper, we introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision while being cost-efficient. Our approach is inspired by the observation that rectifying the quantized activation distribution to match its float counterpart can readily restore accuracy for LLMs. To achieve this, we carefully design a tweaking strategy that includes calibration data generation and channel-wise distance constraint to update the weights of normalization layers for better generalization. We conduct extensive experiments on various datasets using several open-sourced LLMs. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations, surpassing existing PTQ methods. On GLM-130B and OPT-66B, our method even achieves the same level of accuracy at 2-bit quantization as their float ones. Our simple and effective approach makes it more practical for real-world applications.

Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge

Large Language Models (LLMs) stand out for their impressive performance in intricate language modeling tasks. However, their demanding computational and memory needs pose obstacles for broad use on edge devices. Quantization is then introduced to boost LLMs' on-device efficiency. Recent works show that 8-bit or lower weight quantization is feasible with minimal impact on end-to-end task performance, while the activation is still not quantized. On the other hand, mainstream commodity edge devices still struggle to execute these sub-8-bit quantized networks effectively. In this paper, we propose Agile-Quant, an Activation-Guided quantization framework for faster Inference of popular Large Language Models (LLMs) on the Edge. Considering the hardware profiling and activation analysis, we first introduce a basic activation quantization strategy to balance the trade-off of task performance and real inference speed. Then we leverage the activation-aware token pruning technique to reduce the outliers and the adverse impact on attentivity. Ultimately, we utilize the SIMD-based 4-bit multiplier and our efficient TRIP matrix multiplication to implement the accelerator for LLMs on the edge. We apply our framework on different scales of LLMs including LLaMA, OPT, and BLOOM with 4-bit or 8-bit for the activation and 4-bit for the weight quantization. Experiments show that Agile-Quant achieves simultaneous quantization of model weights and activations while maintaining task performance comparable to existing weight-only quantization methods. Moreover, in the 8- and 4-bit scenario, Agile-Quant achieves an on-device speedup of up to 2.55x compared to its FP16 counterparts across multiple edge devices, marking a pioneering advancement in this domain.

What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation

Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance and efficiency is well-known, there is still much to be learned about the relationship between quantization and LLM performance. To shed light on this relationship, we propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We call this approach ``the lens of perturbation". Using this lens, we conduct experiments with various artificial perturbations to explore their impact on LLM performance. Our findings reveal several connections between the properties of perturbations and LLM performance, providing insights into the failure cases of uniform quantization and suggesting potential solutions to improve the robustness of LLM quantization. To demonstrate the significance of our findings, we implement a simple non-uniform quantization approach based on our insights. Our experiments show that this approach achieves minimal performance degradation on both 4-bit weight quantization and 8-bit quantization for weights and activations. These results validate the correctness of our approach and highlight its potential to improve the efficiency of LLMs without sacrificing performance.

Assessment of inference accuracy and memory capacity of computation-in-memory enabled neural network due to quantized weights, gradients, input and output signals, and memory non-idealities

This paper proposes an approach to enhance the efficiency of computation-in-memory (CiM) enabled neural networks. The proposed methods involve partial quantization of learning and inference processes within the neural network to increase the training and inference speed while reducing energy and memory consumption. The impact of quantization due to the usage of CiM is evaluated based on inference accuracy. The effect of non-idealities incurred due to the employment of different memories such as resistive random-access memory on the network accuracy is documented and reported. The results indicate that a certain quantization bit precision threshold is necessary for weights, input/output data, and gradients to maintain an acceptable inference accuracy level. Notably, the experiments demonstrate a modest degradation of approximately 2.8% in inference accuracy compared to the neural network trained without using computation-in-memory, this accuracy trade-off is accompanied by a substantial memory footprint improvement, with memory usage reductions of 62% and 93% during the training and inference phase respectively.

A 96 dB DR Second-Order CIFF Delta-Sigma Modulator with Rail-to-Rail Input Voltage Range

A second-order delta-sigma modulator (DSM) is proposed for readout integrated circuits of sensor applications requiring a small area and low-power consumption. The proposed second-order CIFF DSM with the architecture of cascaded-of-integrator feedforward (CIFF) basically consists of two integrators, a 3-bit quantizer, data-weighted averaging (DWA) circuit, and clock generator. The use of the 3-bit quantizer instead of the single-bit quantizer reduces the size of the feedback capacitor in the first integrator. The 3-bit quantizer is designed based on a successive approximation register analog-to-digital converter for small area and low power implementation. Furthermore, the proposed second-order CIFF DSM has a single supply without an additional reference driver while having a wide analog input voltage range with rail to rail. The proposed second-order CIFF DSM, implemented using a 130 nm 1-poly 6-metal CMOS process with a supply of 1.5 V, has an area of 0.096 mm2. It has a sampling frequency of 500 kHz for the implementation of an input bandwidth of 2 kHz and an oversampling ratio of 125. The measured peak signal-to-noise and distortion ratio is approximately 90 dB when the differential analog input signal has a frequency of 353 Hz and an amplitude of 1.2 Vpp. The measured dynamic range is approximately 96.3 dB.

A Miniatured Fully Integrated High Resolution and Accuracy Capacitive Angle Encoder

In this paper, a capacitive angle encoder monitored and sensed by ASIC with a sensitive structure fabricated by MEMS technology is proposed. The sensitive structure with a 27.5mm outer diameter is designed and optimized in a limited footprint to realize higher accuracy. After being fabricated by MEMS technology, it is integrated with an ASIC, realizing high fabrication accuracy and a small footprint at the same time. The ASIC is designed in an analog-digital hybrid architecture to achieve low power consumption as well as a small footprint. The analog part of the ASIC consists of a charge to voltage converter with capacitance cancellation array and a sigma-delta modulator (SDM) with a 3-bit quantizer. In the digital part of ASIC, the processor based on Cortex-M3 core is designed to perform angle calculation. The measurement result of the proposed angle sensor reveals that the power consumption is 250 mW from a single 5V supply, the bandwidth is over 130Hz, the resolution is 0.002° and the accuracy over the full range is 0.024°, indicating that the encoder has considerable potential for use in high-precision applications.

SC and OFDM hybrid coherent optical transmission scheme based on 1-bit bandpass delta-sigma modulation.

High-order quadrature amplitude modulation (QAM) can effectively improve the capacity and spectral efficiency of coherent optical transmission systems. However, as the modulation order increases, the signal becomes less tolerant to noise and nonlinear effects during transmission, and the implementation cost also increases. We propose a single carrier (SC) and orthogonal frequency division multiplexing (OFDM) hybrid coherent optical transmission scheme based on a 1-bit bandpass (BP) delta-sigma modulation (DSM). The driving I-channel and Q-channel signals for the optical in-phase/quadrature (I/Q) modulator carry SC-modulated and OFDM-modulated transmitter data, respectively. Optical quadrature-phase-shift-keying (QPSK) modulation is realized by the 1-bit DSM quantizer and I/Q modulator, which can effectively suppress quantization noise and reduce the complexity of digital signal processing (DSP) and the performance requirements of optoelectronic devices. In addition, the hybrid transmission of SC and OFDM can balance the advantages of both to meet the variable channel conditions and complex application scenarios. High-fidelity transmission of SC 512QAM and OFDM 512QAM hybrid signals, in the form of a 60 Gbaud optical QPSK signal, over 60 km single-mode fiber-28 (SMF-28) is verified by offline experiments, and the bit error rates (BERs) of both SC 512QAM and OFDM 512QAM are below the hard-decision forward-error correction (HD-FEC) threshold of 3.8e-3.

Eciton: Very Low-power Recurrent Neural Network Accelerator for Real-time Inference at the Edge

This article presents Eciton, a very low-power recurrent neural network accelerator for time series data within low-power edge sensor nodes, achieving real-time inference with a power consumption of 17 mW under load. Eciton reduces memory and chip resource requirements via 8-bit quantization and hard sigmoid activation, allowing the accelerator as well as the recurrent neural network model parameters to fit in a low-cost, low-power Lattice iCE40 UP5K FPGA. We evaluate Eciton on multiple, established time-series classification applications including predictive maintenance of mechanical systems, sound classification, and intrusion detection for IoT nodes. Binary and multi-class classification edge models are explored, demonstrating that Eciton can adapt to a variety of deployable environments and remote use cases. Eciton demonstrates real-time processing at a very low power consumption with minimal loss of accuracy on multiple inference scenarios with differing characteristics, while achieving competitive power efficiency against the state-of-the-art of similar scale. We show that the addition of this accelerator actually reduces the power budget of the sensor node by reducing power-hungry wireless transmission. The resulting power budget of the sensor node is small enough to be powered by a power harvester, potentially allowing it to run indefinitely without a battery or periodic maintenance.

A Comparative Study of Preprocessing and Model Compression Techniques in Deep Learning for Forest Sound Classification.

Deep-learning models play a significant role in modern software solutions, with the capabilities of handling complex tasks, improving accuracy, automating processes, and adapting to diverse domains, eventually contributing to advancements in various industries. This study provides a comparative study on deep-learning techniques that can also be deployed on resource-constrained edge devices. As a novel contribution, we analyze the performance of seven Convolutional Neural Network models in the context of data augmentation, feature extraction, and model compression using acoustic data. The results show that the best performers can achieve an optimal trade-off between model accuracy and size when compressed with weight and filter pruning followed by 8-bit quantization. In adherence to the study workflow utilizing the forest sound dataset, MobileNet-v3-small and ACDNet achieved accuracies of 87.95% and 85.64%, respectively, while maintaining compact sizes of 243 KB and 484 KB, respectively. Henceforth, this study concludes that CNNs can be optimized and compressed to be deployed in resource-constrained edge devices for classifying forest environment sounds.

Optimizing mineral identification for sustainable resource extraction through hybrid deep learning enabled FinTech model

Mineral extraction and use are vital to the world economy and industrial and energy industries. Environmental implications and the limited nature of many mineral resources have made mineral extraction sustainability a major issue. Deep learning and FinTech have provided creative answers to these problems in recent years. This presentation discusses the consequences of using deep learning-enabled FinTech solutions to identify minerals for sustainable natural resource extraction and use. Quantized deep learning for mineral identification during mining, using the widely used EfficientDet architecture, is proposed in this paper. We present a post-training quantization approach for the EfficientDet model that minimizes its size and complexity without compromising performance. We tested model efficiency with 8-bit and symmetric quantization. We also compare the quantized EfficientDet model to the floating-point model using a custom mineral dataset. The prescribed dataset gave the pre-train EfficientDet model 0.78 precision and 0.65 recall. The suggested model had 0.97 accuracy and 0.89 recall on the provided datasets. The suggested model is 99.5% accurate. Our results show that integrated FinTech technologies have enabled innovative financial models for ecologically friendly mineral extraction. By assessing environmental and social risks, FinTech platforms have helped investors assess mining project profitability. Investors may use this data to make informed judgments about sustainable development and ethical mining. Deep learning and FinTech mineral identification affects responsible natural resource development. These advancements enable environmentally and socially friendly mining. Data-driven insights and financial transparency can help the global mining business become more sustainable and responsible.

Federated Learning With Heterogeneous Quantization Bit Allocation and Aggregation for Internet of Things

Federated Learning With Heterogeneous Quantization Bit Allocation and Aggregation for Internet of Things

Quantized iterative learning control for nonlinear multi-agent systems with limited information communication and input saturation

This paper considers a quantized consensus problem for nonlinear multi-agent systems (MAS) using iterative learning control (ILC). For actual digital communication networks, agents can only transmit state information with limited bandwidth. Therefore, a Sigma-Delta (ΣΔ) quantizer with a finite number of quantized bits is used to satisfy the communication network requirements. In addition, the introduction of network issues like input saturation and time delay make the problem more practically relevant. Due to the discontinuity caused by quantization, Filippov’s non-smooth analysis theory is required to analyze the convergence performance of the MAS. The desired asymptotic consensus can be achieved with limited quantized information and possibly even a single bit between each pair of adjacent agents. Finally, numerical simulations are presented to illustrate the effectiveness of our theoretical analysis.

State Machine-Based Waveforms for Channels With 1-bit Quantization and Oversampling With Time-Instance Zero-Crossing Modulation

State Machine-Based Waveforms for Channels With 1-bit Quantization and Oversampling With Time-Instance Zero-Crossing Modulation

PSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers.

Data-free quantization can potentially address data privacy and security concerns in model compression and thus has been widely investigated. Recently, patch similarity aware data-free quantization for vision transformers (PSAQ-ViT) designs a relative value metric, patch similarity, to generate data from pretrained vision transformers (ViTs), achieving the first attempt at data-free quantization for ViTs. In this article, we propose PSAQ-ViT V2, a more accurate and general data-free quantization framework for ViTs, built on top of PSAQ-ViT. More specifically, following the patch similarity metric in PSAQ-ViT, we introduce an adaptive teacher-student strategy, which facilitates the constant cyclic evolution of the generated samples and the quantized model in a competitive and interactive fashion under the supervision of the full-precision (FP) model (teacher), thus significantly improving the accuracy of the quantized model. Moreover, without the auxiliary category guidance, we employ the task-and model-independent prior information, making the general-purpose scheme compatible with a broad range of vision tasks and models. Extensive experiments are conducted on various models on image classification, object detection, and semantic segmentation tasks, and PSAQ-ViT V2, with the naive quantization strategy and without access to real-world data, consistently achieves competitive results, showing potential as a powerful baseline on data-free quantization for ViTs. For instance, with Swin-S as the (backbone) model, 8-bit quantization reaches 82.13 top-1 accuracy on ImageNet, 50.9 box AP and 44.1 mask AP on COCO, and 47.2 mean Intersection over Union (mIoU) on ADE20K. We hope that accurate and general PSAQ-ViT V2 can serve as a potential and practice solution in real-world applications involving sensitive data. Code is released and merged at: https://github.com/zkkli/PSAQ-ViT.

Adaptive Global Power-of-Two Ternary Quantization Algorithm Based on Unfixed Boundary Thresholds.

In the field of edge computing, quantizing convolutional neural networks (CNNs) using extremely low bit widths can significantly alleviate the associated storage and computational burdens in embedded hardware, thereby improving computational efficiency. However, such quantization also presents a challenge related to substantial decreases in detection accuracy. This paper proposes an innovative method, called Adaptive Global Power-of-Two Ternary Quantization Based on Unfixed Boundary Thresholds (APTQ). APTQ achieves adaptive quantization by quantizing each filter into two binary subfilters represented as power-of-two values, thereby addressing the accuracy degradation caused by a lack of expression ability of low-bit-width weight values and the contradiction between fixed quantization boundaries and the uneven actual weight distribution. It effectively reduces the accuracy loss while at the same time presenting strong hardware-friendly characteristics because of the power-of-two quantization. This paper extends the APTQ algorithm to propose the APQ quantization algorithm, which can adapt to arbitrary quantization bit widths. Furthermore, this paper designs dedicated edge deployment convolutional computation modules for the obtained quantized models. Through quantization comparison experiments with multiple commonly used CNN models utilized on the CIFAR10, CIFAR100, and Mini-ImageNet data sets, it is verified that the APTQ and APQ algorithms possess better accuracy performance than most state-of-the-art quantization algorithms and can achieve results with very low accuracy loss in certain CNNs (e.g., the accuracy loss of the APTQ ternary ResNet-56 model on CIFAR10 is 0.13%). The dedicated convolutional computation modules enable the corresponding quantized models to occupy fewer on-chip hardware resources in edge chips, thereby effectively improving computational efficiency. This adaptive CNN quantization method, combined with the power-of-two quantization results, strikes a balance between the quantization accuracy performance and deployment efficiency in embedded hardware. As such, valuable insights for the industrial edge computing domain can be gained.