Hardware-software Co-design Research Articles

Hardware–Software Co-Design of an Audio Feature Extraction Pipeline for Machine Learning Applications

Keyword spotting is an important part of modern speech recognition pipelines. Typical contemporary keyword-spotting systems are based on Mel-Frequency Cepstral Coefficient (MFCC) audio features, which are relatively complex to compute. Considering the always-on nature of many keyword-spotting systems, it is prudent to optimize this part of the detection pipeline. We explore the simplifications of the MFCC audio features and derive a simplified version that can be more easily used in embedded applications. Additionally, we implement a hardware generator that generates an appropriate hardware pipeline for the simplified audio feature extraction. Using Chisel4ml framework, we integrate hardware generators into Python-based Keras framework, which facilitates the training process of the machine learning models using our simplified audio features.

Acceleration of Graph Neural Network-Based Prediction Models in Chemistry via Co-Design Optimization on Intelligence Processing Units.

Atomic structure prediction and associated property calculations are the bedrock of chemical physics. Since high-fidelity ab initio modeling techniques for computing the structure and properties can be prohibitively expensive, this motivates the development of machine-learning (ML) models that make these predictions more efficiently. Training graph neural networks over large atomistic databases introduces unique computational challenges, such as the need to process millions of small graphs with variable size and support communication patterns that are distinct from learning over large graphs, such as social networks. We demonstrate a novel hardware-software codesign approach to scale up the training of atomistic graph neural networks (GNN) for structure and property prediction. First, to eliminate redundant computation and memory associated with alternative padding techniques and to improve throughput via minimizing communication, we formulate the effective coalescing of the batches of variable-size atomistic graphs as the bin packing problem and introduce a hardware-agnostic algorithm to pack these batches. In addition, we propose hardware-specific optimizations, including a planner and vectorization for the gather-scatter operations targeted for Graphcore's Intelligence Processing Unit (IPU), as well as model-specific optimizations such as merged communication collectives and optimized softplus. Putting these all together, we demonstrate the effectiveness of the proposed codesign approach by providing an implementation of a well-established atomistic GNN on the Graphcore IPUs. We evaluate the training performance on multiple atomistic graph databases with varying degrees of graph counts, sizes, and sparsity. We demonstrate that such a codesign approach can reduce the training time of atomistic GNNs and can improve their performance by up to 1.5× compared to the baseline implementation of the model on the IPUs. Additionally, we compare our IPU implementation with a Nvidia GPU-based implementation and show that our atomistic GNN implementation on the IPUs can run 1.8× faster on average compared to the execution time on the GPUs.

ApHMM: Accelerating Profile Hidden Markov Models for Fast and Energy-efficient Genome Analysis

Profile hidden Markov models (pHMMs) are widely employed in various bioinformatics applications to identify similarities between biological sequences, such as DNA or protein sequences. In pHMMs, sequences are represented as graph structures, where states and edges capture modifications (i.e., insertions, deletions, and substitutions) by assigning probabilities to them. These probabilities are subsequently used to compute the similarity score between a sequence and a pHMM graph. The Baum-Welch algorithm, a prevalent and highly accurate method, utilizes these probabilities to optimize and compute similarity scores. Accurate computation of these probabilities is essential for the correct identification of sequence similarities. However, the Baum-Welch algorithm is computationally intensive, and existing solutions offer either software-only or hardware-only approaches with fixed pHMM designs. When we analyze state-of-the-art works, we identify an urgent need for a flexible, high-performance, and energy-efficient hardware-software co-design to address the major inefficiencies in the Baum-Welch algorithm for pHMMs. We introduce ApHMM , the first flexible acceleration framework designed to significantly reduce both computational and energy overheads associated with the Baum-Welch algorithm for pHMMs. ApHMM employs hardware-software co-design to tackle the major inefficiencies in the Baum-Welch algorithm by (1) designing flexible hardware to accommodate various pHMM designs, (2) exploiting predictable data dependency patterns through on-chip memory with memoization techniques, (3) rapidly filtering out unnecessary computations using a hardware-based filter, and (4) minimizing redundant computations. ApHMM achieves substantial speedups of 15.55×–260.03×, 1.83×–5.34×, and 27.97× when compared to CPU, GPU, and FPGA implementations of the Baum-Welch algorithm, respectively. ApHMM outperforms state-of-the-art CPU implementations in three key bioinformatics applications: (1) error correction, (2) protein family search, and (3) multiple sequence alignment, by 1.29×–59.94×, 1.03×–1.75×, and 1.03×–1.95×, respectively, while improving their energy efficiency by 64.24×–115.46×, 1.75×, and 1.96×.

Tailor : Altering Skip Connections for Resource-Efficient Inference

Deep neural networks use skip connections to improve training convergence. However, these skip connections are costly in hardware, requiring extra buffers and increasing on- and off-chip memory utilization and bandwidth requirements. In this article, we show that skip connections can be optimized for hardware when tackled with a hardware-software codesign approach. We argue that while a network’s skip connections are needed for the network to learn, they can later be removed or shortened to provide a more hardware-efficient implementation with minimal to no accuracy loss. We introduce Tailor , a codesign tool whose hardware-aware training algorithm gradually removes or shortens a fully trained network’s skip connections to lower the hardware cost. Tailor improves resource utilization by up to 34% for block random access memories (BRAMs), 13% for flip-flops (FFs), and 16% for look-up tables (LUTs) for on-chip, dataflow-style architectures. Tailor increases performance by 30% and reduces memory bandwidth by 45% for a two-dimensional processing element array architecture.

In-memory and in-sensor reservoir computing with memristive devices

Despite the significant progress made in deep learning on digital computers, their energy consumption and computational speed still fall short of meeting the standards for brain-like computing. To address these limitations, reservoir computing (RC) has been gaining increasing attention across communities of electronic devices, computing systems, and machine learning, notably with its in-memory or in-sensor implementation on the hardware–software co-design. Hardware regarded, in-memory or in-sensor computers leverage emerging electronic and optoelectronic devices for data processing right where the data are stored or sensed. This technology dramatically reduces the energy consumption from frequent data transfers between sensing, storage, and computational units. Software regarded, RC enables real-time edge learning thanks to its brain-inspired dynamic system with massive training complexity reduction. From this perspective, we survey recent advancements in in-memory/in-sensor RC, including algorithm designs, material and device development, and downstream applications in classification and regression problems, and discuss challenges and opportunities ahead in this emerging field.

IEEE CASS/SSCS/EDS HUST Student Branch Chapter Successfully Organizes a Hardware–Software Codesign Lecture [Chapters

IEEE CASS/SSCS/EDS HUST Student Branch Chapter Successfully Organizes a Hardware–Software Codesign Lecture [Chapters

Hopscotch: A Hardware-Software Co-Design for Efficient Cache Resizing on Multi-Core SoCs

Hopscotch: A Hardware-Software Co-Design for Efficient Cache Resizing on Multi-Core SoCs

SH-GAT: Software-hardware co-design for accelerating graph attention networks on FPGA

<abstract><p>Graph convolution networks (GCN) have demonstrated success in learning graph structures; however, they are limited in inductive tasks. Graph attention networks (GAT) were proposed to address the limitations of GCN and have shown high performance in graph-based tasks. Despite this success, GAT faces challenges in hardware acceleration, including: 1) The GAT algorithm has difficulty adapting to hardware; 2) challenges in efficiently implementing Sparse matrix multiplication (SPMM); and 3) complex addressing and pipeline stall issues due to irregular memory accesses. To this end, this paper proposed SH-GAT, an FPGA-based GAT accelerator that achieves more efficient GAT inference. The proposed approach employed several optimizations to enhance GAT performance. First, this work optimized the GAT algorithm using split weights and softmax approximation to make it more hardware-friendly. Second, a load-balanced SPMM kernel was designed to fully leverage potential parallelism and improve data throughput. Lastly, data preprocessing was performed by pre-fetching the source node and its neighbor nodes, effectively addressing pipeline stall and complexly addressing issues arising from irregular memory access. SH-GAT was evaluated on the Xilinx FPGA Alveo U280 accelerator card with three popular datasets. Compared to existing CPU, GPU, and state-of-the-art (SOTA) FPGA-based accelerators, SH-GAT can achieve speedup by up to 3283$ \times $, 13$ \times $, and 2.3$ \times $.</p></abstract>

Analysis of the Principle and Configuration of AI Chip and the State-of-art Applications

AI chips are specialized hardware components designed to accelerate AI operations, making them crucial for various industries. This study provides a comprehensive overview of artificial intelligence (AI) chips, their structure, principles, fabrication process, applications, limitations, and future outlooks. The article discusses the components commonly found in AI chip structures, including processing units, memory hierarchy, interconnects, and specialized hardware units. It delves into the principles guiding AI chip design, such as specialization, parallelism, low precision arithmetic, and hardware-software co-design. The fabrication process of AI chips involves intricate steps, including design, mask generation, deposition, lithography, and testing. Applications of AI chips span data centers, edge computing, autonomous vehicles, healthcare, NLP, and more. Despite their advancements, AI chips have limitations, such as energy efficiency and specificity, but the future holds promise with improvements in energy efficiency, specialized acceleration, hybrid architectures, and quantum AI chips. Overall, AI chips have transformed the AI landscape, setting the stage for continued innovation and impact in various sectors.

Novel neuromorphic architectures based on crossbar arrays of (Co-Fe-B)<sub>x</sub>(LiNbO<sub>3</sub>)<sub>100−x</sub> nanocomposite memristors

Memristor-based neuromorphic computing systems (NСSs) provide a fast, high computational and energy efficient approach to neural network (NN) training and solving cognitive problems (pattern recognition, big data processing, prediction, etc.) [1]. Memristors could be organized in large crossbar arrays to perform vector-matrix multiplication (VMM) in a natural one-step way by the weighted electrical current summation (according to the Ohm’s and Kirchhoff’s laws) [1]. In contrast, being the most massively parallel operation in NN learning and inference, VMM is extremely time- and energy-expensive in traditional von Neumann architectures. Owing to this difference, memristor-based NCSs are of high interest. Memristors have already been successfully implemented for diverse NCS realizations, and such schemes as multi-layer perceptron (MLP) [2], long short-term memory and others have been demonstrated. Most of these NCSs are usually trained by various types of gradient descent learning algorithm, the hardware realization of which is challenging due to unreliable cycle-to-cycle (c2c) and device-to-device (d2d) variations of memristive devices. Several approaches have been proposed to partially mitigate these problems, including reservoir computing [3] and fine feature engineering [4]. The general idea of such approaches is to reduce the number of required weights (i.e. memristors) compared with fully connected NNs. In this respect, such novel architectures as convolutional NN (CNN) and MLP-mixer are of high interest as they provide significant weight reduction without classification efficiency drop. Although CNN based on memristors was already demonstrated, different aspects of its realization (such as hybrid hardware-software co-design) have yet to be studied. MLP-mixer was realized only in software. Therefore, in this work we have studied the possibility of hardware realization of CNN and MLP-mixer networks based on crossbar arrays of memristors. For this purpose, we studied (Co-Fe-B)x(LiNbO3)100−x nanocomposite (CFB-LNO NC) memristors, which operate through a multifilamentary resistive switching (RS) mechanism, demonstrate high endurance, long retention and possess multilevel RS [5]. Crossbar array of memristors was fabricated using laser photolithography for patterning electrode buses and ion-beam sputtering on the original facility for active layer deposition (~10 nm thick LiNbO3 and ~290 nm thick CFB-LNO NC with x ≈10–25 at.%). Details of the fabrication process could be found elsewhere [5]. I-V curves of the fabricated memristors showed small c2c and d2d variations, plasticity with 16 different resistive states and endurance of more than 105 cycles. Using the nanocomposite based crossbar arrays, we implemented a hybrid CNN, consisting of a hardware feature extractor with one/two kernels and a software classifier. Additionally, we have demonstrated in simulation that the usage of the memristors under study in the accurately adapted MLP-Mixer architecture results in high classification accuracy that is resilient to memristive variations and stuck devices.

Novel hardware/software co-design approach for Connect6 game-solver

This paper introduces an enhanced Hardware-Software Co-Design implementation of Connect6 game solver compared to a prevalent implementation by using novel data communications and enhanced tree-searching algorithms. Hardware-software communication is improved by transferring the current board position from the Processing System to the Programmable Logic through a 21-bit change board instead of the complete 722-bit board. Tree-search, to find the next best move, is enhanced by using Negascout algorithm, Iterative Deepening, Node Ordering and Memoization. The proposed design demonstrates 50 % improvement in the overall performance and 92 % in communication overhead compared to the prevalent implementation. Further, it is around 4 times faster than its software implementation on the same platform. The analysis was carried on Xilinx Zedboard SOC using the GNU GCC Compiler with the Maximum Optimisation Flag (O3) activated.

IMGA: Efficient In-Memory Graph Convolution Network Aggregation With Data Flow Optimizations

Aggregating features from neighbor vertices is a fundamental operation in Graph Convolution Network (GCN). However, the sparsity in graph data creates poor spatial and temporal locality, causing dynamic and irregular memory access patterns and limiting the performance of aggregation on the Von Neumann architecture. The emerging processing-in-memory (PIM) architecture is based on emerging non-volatile memory (NVM), like Spin-orbit torque Magnetic RAM (SOT-MRAM), and demonstrates promising prospects in alleviating the Von Neumann bottleneck. However, the limited memory capacity of PIM medium still incurs non-negligible data movements between PIM architecture and external memory. To solve this challenge, we propose a SOT-MRAM based in-memory computing architecture, called IMGA, for efficient in-situ graph aggregation. Specifically, we design adaptive data flow management strategies that reuse vertex data in MRAM when processing graphs of different scales and adopt edge data as the control signal source to utilize the graph’s structural information. A reordering optimization strategy leveraging hardware-software co-design principle is proposed to further reduce the costly data movement. Experimental results demonstrate that IMGA achieves an average 2523x and 21x speedup, and 1.03E+6 and 1.04E+3 energy efficiency compared with CPU and GPU, respectively.

TSAR-ILP: Tile-Based, Synchronization-AwaRe ILP Allocating Heterogeneous Platforms for Streaming Applications

Automatic design space exploration (DSE) is key in hardware-software (HW/SW) co-design. To cope with the large design space, explorations are often heuristic-based and/or approximate yielding potentially locally optimal solutions. Without knowing the globally optimal solution, strong assertions about performance upper / lower bounds cannot be made. In contrast, integer linear programming (ILP) formulations can produce exact (optimal) solutions. Previous ILP-based formulations, however, lack support for tile-based architectures and realistic synchronization models, limiting their DSE capabilities. This work introduces a tile-based, synchronization-aware ILP (TSAR-ILP) formulation that overcomes previous limitations. With TSAR-ILP, the allocation / binding problems are introduced and formalized, attaining optimal solutions for mapping streaming applications onto template platforms. Using TSAR-ILP, this work explores a hardware acceleratorrich (HWACC-rich) platform with direct HWACC-to-HWACC communication under HW area constraints for 40 OpenVX applications. To illustrate design opportunities given by (a) the ILP formulation and (b) direct HWACC-to-HWACC communication, this paper analyzes the impact of job size. Results show that selecting smaller job sizes yields performance improvements and less area usage at the cost of slightly increased synchronization overhead. A job size reduction from 1 kB to 256 bytes gives 3.51x average performance increase across 40 applications. Finally, DSE with TSAR-ILP is shown not to be prohibitive through scalability analysis using a set of 5000 synthetic applications with varying size (10-125 nodes), with 94.3% of applications successfully achieving optimal solutions under 60 seconds.

Hardware–Software Co-Design for Real-Time Latency–Accuracy Navigation in Tiny Machine Learning Applications

Hardware–Software Co-Design for Real-Time Latency–Accuracy Navigation in Tiny Machine Learning Applications

Teaching Edge AI at the Undergraduate Level: A Hardware–Software Co-Design Approach

Teaching Edge AI at the Undergraduate Level: A Hardware–Software Co-Design Approach

RAPIDx: High-Performance ReRAM Processing In-Memory Accelerator for Sequence Alignment

Genome sequence alignment is the core of many biological applications. The advancement of sequencing technologies produces a tremendous amount of data, making sequence alignment a critical bottleneck in bioinformatics analysis. The existing hardware accelerators for alignment suffer from limited on-chip memory, costly data movement, and poorly optimized alignment algorithms. They cannot afford to concurrently process the massive amount of data generated by sequencing machines. In this paper, we propose a ReRAM-based accelerator, RAPIDx, using processing in-memory (PIM) for sequence alignment. RAPIDx achieves superior efficiency and performance via software-hardware co-design. First, we propose an adaptive banded parallelism alignment algorithm suitable for PIM architecture. Compared to the original dynamic programming-based alignment, the proposed algorithm significantly reduces the required complexity, data bit width, and memory footprint at the cost of negligible accuracy degradation. Then we propose the efficient PIM architecture that implements the proposed algorithm. The data flow in RAPIDx achieves four-level parallelism and we design an in-situ alignment computation flow in ReRAM, delivering $5.5$-$9.7\times$ efficiency and throughput improvements compared to our previous PIM design, RAPID. The proposed RAPIDx is reconfigurable to serve as a co-processor integrated into existing genome analysis pipeline to boost sequence alignment or edit distance calculation. On short-read alignment, RAPIDx delivers $131.1\times$ and $46.8\times$ throughput improvements over state-of-the-art CPU and GPU libraries, respectively. As compared to ASIC accelerators for long-read alignment, the performance of RAPIDx is $1.8$-$2.9\times$ higher.

BitSET: Bit-Serial Early Termination for Computation Reduction in Convolutional Neural Networks

Convolutional Neural Networks (CNNs) have demonstrated remarkable performance across a wide range of machine learning tasks. However, the high accuracy usually comes at the cost of substantial computation and energy consumption, making it difficult to be deployed on mobile and embedded devices. In CNNs, the compute-intensive convolutional layers are usually followed by a ReLU activation layer, which clamps negative outputs to zeros, resulting in large activation sparsity. By exploiting such sparsity in CNN models, we propose a software-hardware co-design BitSET, that aggressively saves energy during CNN inference. The bit-serial BitSET accelerator adopts a prediction-based bit-level early termination technique that terminates the ineffectual computation of negative outputs early. To assist the algorithm, we propose a novel weight encoding that allows more accurate predictions with fewer bits. BitSET leverages the bit-level computation reduction both in the predictive early termination algorithm and in the non-predictive, energy-efficient bit-serial architecture. Compared to UNPU, an energy-efficient bit-serial CNN accelerator, BitSET yields an average 1.5× speedup and 1.4× energy efficiency improvement with no accuracy loss due to a 48% reduction in bit-level computations. Relaxing the allowed accuracy loss to 1% increases the gains to an average of 1.6× speedup and 1.4× energy efficiency improvement.

Hardware–software codesign for peer-to-peer energy market resolution

Hardware–software codesign for peer-to-peer energy market resolution

Object Fingerprint Cache for Heterogeneous Memory System

Heterogeneous memory systems promise to provide both large storage capacity and high performance. Prior DRAM cache systems have metadata scalability issue and suffer from low cache hit rate, low DRAM space utilization, and significant data migration overhead. We observe that instances of an object type exhibit stable and predictable memory access patterns in its constituent cachelines and these patterns are referred to as object fingerprints. We propose the hardware-assisted cache that manages DRAM at the object type level and fetches data block at the granularity of a cacheline, by exploiting object fingerprint. To address its design challenges, we first present a software-hardware co-design to convey the software information to hardware. Second, we design multiple granularity sector caches that can be dynamically adjusted to adapt to changing behaviors and improve DRAM cache utilization. To address the challenges of large metadata storage overhead, we propose to bound possible sizes for each sector cache. Experimental results show our designs improve DRAM cache hit rate by 21.6%, boost IPC by 19.8%, and reduce data migration traffic by 51.6% on average, compared with state-of-art DRAM caches. More importantly, our online object fingerprint learning method is 2.3% inferior to the offline one in terms of IPC.

Systemization of Knowledge: Robust Deep Learning using Hardware-software co-design in Centralized and Federated Settings

Deep learning (DL) models are enabling a significant paradigm shift in a diverse range of fields, including natural language processing, computer vision, as well as the design and automation of complex integrated circuits. While the deep models – and optimizations-based on them, e.g., Deep Reinforcement Learning (RL) – demonstrate a superior performance and a great capability for automated representation learning, earlier works have revealed the vulnerability of DLs to various attacks. The vulnerabilities include adversarial samples, model poisoning, and fault injection attacks. On the one hand, these security threats could divert the behavior of the DL model and lead to incorrect decisions in critical tasks. On the other hand, the susceptibility of DLs to potential attacks might thwart trustworthy technology transfer as well as reliable DL deployment. In this work, we investigate the existing defense techniques to protect DLs against the above-mentioned security threats. Particularly, we review end-to-end defense schemes for robust deep learning in both centralized and federated learning settings. Our comprehensive taxonomy and horizontal comparisons reveal an important fact that defense strategies developed using DL/software/hardware co-design outperform the DL/software-only counterparts and show how they can achieve very efficient and latency-optimized defenses for real-world applications. We believe our systemization of knowledge sheds light on the promising performance of hardware-software co-design of DL security methodologies and can guide the development of future defenses.