Artificial Intelligence Inference Research Articles

Automated Backend Allocation for Multi-Model, On-Device AI Inference

On-Device Artificial Intelligence (AI) services such as face recognition, object tracking and voice recognition are rapidly scaling up deployments on embedded, memory-constrained hardware devices. These services typically delegate AI inference models for execution on CPU and GPU computing backends. While GPU delegation is a common practice to achieve high speed computation, the approach suffers from degraded throughput and completion times under multi-model scenarios, i.e., concurrently executing services. This paper introduces a solution to sustain performance in multi-model, on-device AI contexts by dynamically allocating a combination of CPU and GPU backends per model. The allocation is feedback-driven, and guided by a knowledge of model-specific, multi-objective pareto fronts comprising inference latency and memory consumption. Our backend allocation algorithm that runs online per model, and achieves 25-100% improvement in throughput over static allocations as well as load-balancing scheduler solutions targeting multi-model scenarios.

Applications of Artificial Intelligence Techniques in Healthcare and Medicine

Abstract—Artificial intelligence (AI) is swiftly coming into fitness care and serving foremost roles, from automating drudgery and ordinary responsibilities in scientific exercise to dealing with sufferers and scientific resources. As builders create AI structures to tackle those responsibilities, numerous dangers and demanding situations emerge, along with the hazard of accidents to sufferers from AI gadget errors, the hazard to affected person privateness of information acquisition and AI inference, and more. Potential answers are complicated however contain funding in infrastructure for high-quality, consultant information; collaborative oversight via way of means of each the Food and Drug Administration and different fitness-care actors; and adjustments to scientific schooling with a purpose to put together companies for moving roles in an evolving gadget. Keywords — Dermatology, Radiology, Screening, Psychiatry, Primary care, Disease diagnosis, Telemedicine, Electronic health records.

High Performance Platform to Detect Faults in the Smart Grid by Artificial Intelligence Inference

High Performance Platform to Detect Faults in the Smart Grid by Artificial Intelligence Inference

GA optimization-based BRB AI reasoning algorithm for determining the factors affecting customer churn for operators

Customer churn directly leads to the weakening of the economic efficiency of an operator and even the loss of its competitive advantage in the market share. The identification and prediction of customer churn in the era of interconnection has become more complex, as it not only is based on the analysis of customer information and customer churn data but also must take into account the intricate relationships between the market economy and culture. In the era of big data, numerous predictive models are based on more redundant features, which increases the complexity of the algorithms and the difficulty of analyzing customer churn. Therefore, in this paper, a belief rule base (BRB) artificial intelligence inference algorithm based on GA optimization to determine the factors affecting customer churn for operators proposed. First, customer churn data from a website are analyzed, and features are extracted. Second, a BRB is established from the experience of operator experts, and a BRB inference prediction model is constructed for predicting and analyzing customer churn. Finally, the BRB model is optimized by GA optimization, the input characteristics with high feature weights are obtained, and the accuracy of the churn analysis is verified according to the obtained features. The results show that this method not only outperforms the comparative SVM and BP neural network models for predicting customer churn but also provides better judgment of the main inputs of customer churn, thus reducing the reliance on input information, optimizing the complexity of the algorithm, and enabling operators to obtain a more accurate understanding of the main factors that lead to churn.

Automated Backend Allocation for Multi-Model, On-Device AI Inference

On-Device Artificial Intelligence (AI) services such as face recognition, object tracking and voice recognition are rapidly scaling up deployments on embedded, memory-constrained hardware devices. These services typically delegate AI inference models for execution on CPU and GPU computing backends. While GPU delegation is a common practice to achieve high speed computation, the approach suffers from degraded throughput and completion times under multi-model scenarios, i.e. concurrently executing services. This paper introduces a solution to sustain performance in multi-model, on-device AI contexts by dynamically allocating a combination of CPU and GPU backends per model. The allocation is feedback-driven, and guided by a knowledge of model-specific, multi-objective pareto fronts comprising inference latency and memory consumption. Primary contribution of this paper is a backend allocation algorithm that runs online per model, and achieves 25-100% improvement in throughput over static allocations as well as load-balancing scheduler solutions targeting multi-model scenarios. Other noteworthy contributions include a novel pareto front estimator that runs on-device, and also a software-based GPU profiler with a lightweight algorithm to detect changing GPU workloads. Specifically, the pareto front estimator outperforms state of the art algorithms NSGA-II and SPEA2 by 94% on pareto coverage, and by almost 2x on computational overhead.

What is the educational value and clinical utility of artificial intelligence for intraoperative and postoperative video analysis? A survey of surgeons and trainees.

Surgical complications often occur due to lapses in judgment and decision-making. Advances in artificial intelligence (AI) have made it possible to train algorithms that identify anatomy and interpret the surgical field. These algorithms can potentially be used for intraoperative decision-support and postoperative video analysis and feedback. Despite the very early success of proof-of-concept algorithms, it remains unknown whether this innovation meets the needs of end-users or how best to deploy it. This study explores users' opinion on the value, usability and design for adapting AI in operating rooms. A device-agnostic web-accessible software was developed to provide AI inference either (1) intraoperatively on a live video stream (synchronous mode), or (2) on an uploaded video or image file (asynchronous mode) postoperatively for feedback. A validated AI model (GoNoGoNet), which identifies safe and dangerous zones of dissection during laparoscopic cholecystectomy, was used as the use case. Surgeons and trainees performing laparoscopic cholecystectomy interacted with the AI platform and completed a 5-point Likert scale survey to evaluate the educational value, usability and design of the platform. Twenty participants (11 surgeons and 9 trainees) evaluated the platform intraoperatively (n = 10) and postoperatively (n = 11). The majority agreed or strongly agreed that AI is an effective adjunct to surgical training (81%; neutral = 10%), effective for providing real-time feedback (70%; neutral = 20%), postoperative feedback (73%; neutral = 27%), and capable of improving surgeon confidence (67%; neutral = 29%). Only 40% (neutral = 50%) and 57% (neutral = 43%) believe that the tool is effective in improving intraoperative decisions and performance, or beneficial for patient care, respectively. Overall, 38% (neutral = 43%) reported they would use this platform consistently if available. The majority agreed or strongly agreed that the platform was easy to use (81%; neutral = 14%) and has acceptable resolution (62%; neutral = 24%), while 30% (neutral = 20%) reported that it disrupted the OR workflow, and 20% (neutral = 0%) reported significant time lag. All respondents reported that such a system should be available "on-demand" to turn on/off at their discretion. Most found AI to be a useful tool for providing support and feedback to surgeons, despite several implementation obstacles. The study findings will inform the future design and usability of this technology in order to optimize its clinical impact and adoption by end-users.

Abstract 21 Machine Learning Approach for Automatic Enumeration of Cord Blood Stem Cells by Flow Cytometry

Abstract Introduction Stem cell laboratories measure the number of viable CD34+ and CD45+ cells in their products by flow cytometry following a standard protocol (ISHAGE). Although the ISHAGE protocol is well documented, its correct application can be challenging. Automated gating algorithms have been created to address this issue, but they have limitations and can only be used in certain conditions. Recent developments in artificial intelligence (AI) and machine learning (ML) have made these approaches more accessible and practical to use. For flow cytometry data, the Cytobank platform now offers the Automatic Gating Algorithm (AGA), an AI-ML-based feature allowing users to easily implement their own gating strategies. Objectives The performance of the AGA based on the ISHAGE protocol gating was evaluated using cord blood post-thaw flow cytometry data. Methods In this study, the AGA was trained using 29 flow cytometry cord blood data files analyzed according to the ISHAGE gating strategy. Using the model, an inference analysis was conducted on two series of 12 data samples acquired from two independent operators. The enumeration of viable CD34+ cells was manually performed and compared to the AI analyses. Results The mean differences and standard deviation in vCD34+ counts between the manual and the automatic gating are 6.0% ± 2.7 for Operator #1 and 7.7% ± 2.1 for Operator #2. The correlation coefficients between vCD34+ results are 0.9981 for Operator #1 vs AI, and 0.9699 for Operator #2 vs AI. These results show that the AI inference provides results comparable to human manual gating, and are within the accepted difference of 10% between duplicates as defined by ISHAGE convention. Discussion This study provides a testing ground for using AI-ML-based automatic gating and supports its use in stem cell laboratories with acceptable performance and accuracy. The application of AI and ML for vCD34+ enumeration can potentially lead to the development of more widely applicable automatic gating algorithms. This could improve standardization by providing a common tool for analyzing flow cytometry data.

HPC Platform for Railway Safety-Critical Functionalities Based on Artificial Intelligence

The automation of railroad operations is a rapidly growing industry. In 2023, a new European standard for the automated Grade of Automation (GoA) 2 over European Train Control System (ETCS) driving is anticipated. Meanwhile, railway stakeholders are already planning their research initiatives for driverless and unattended autonomous driving systems. As a result, the industry is particularly active in research regarding perception technologies based on Computer Vision (CV) and Artificial Intelligence (AI), with outstanding results at the application level. However, executing high-performance and safety-critical applications on embedded systems and in real-time is a challenge. There are not many commercially available solutions, since High-Performance Computing (HPC) platforms are typically seen as being beyond the business of safety-critical systems. This work proposes a novel safety-critical and high-performance computing platform for CV- and AI-enhanced technology execution used for automatic accurate stopping and safe passenger transfer railway functionalities. The resulting computing platform is compatible with the majority of widely-used AI inference methodologies, AI model architectures, and AI model formats thanks to its design, which enables process separation, redundant execution, and HW acceleration in a transparent manner. The proposed technology increases the portability of railway applications into embedded systems, isolates crucial operations, and effectively and securely maintains system resources.

Efficient Convolution Network to Assist Breast Cancer Diagnosis and Target Therapy.

Breast cancer is the leading cause of cancer-related deaths among women worldwide, and early detection and treatment has been shown to significantly reduce fatality rates from severe illness. Moreover, determination of the human epidermal growth factor receptor-2 (HER2) gene amplification by Fluorescence in situ hybridization (FISH) and Dual in situ hybridization (DISH) is critical for the selection of appropriate breast cancer patients for HER2-targeted therapy. However, visual examination of microscopy is time-consuming, subjective and poorly reproducible due to high inter-observer variability among pathologists and cytopathologists. The lack of consistency in identifying carcinoma-like nuclei has led to divergences in the calculation of sensitivity and specificity. This manuscript introduces a highly efficient deep learning method with low computing cost. The experimental results demonstrate that the proposed framework achieves high precision and recall on three essential clinical applications, including breast cancer diagnosis and human epidermal receptor factor 2 (HER2) amplification detection on FISH and DISH slides for HER2 target therapy. Furthermore, the proposed method outperforms the majority of the benchmark methods in terms of IoU by a significant margin (p<0.001) on three essential clinical applications. Importantly, run time analysis shows that the proposed method obtains excellent segmentation results with notably reduced time for Artificial intelligence (AI) training (16.93%), AI inference (17.25%) and memory usage (18.52%), making the proposed framework feasible for practical clinical usage.

A Survey on Collaborative DNN Inference for Edge Intelligence

With the vigorous development of artificial intelligence (AI), intelligence applications based on deep neural networks (DNNs) have changed people’s lifestyles and production efficiency. However, the large amount of computation and data generated from the network edge becomes the major bottleneck, and the traditional cloud-based computing mode has been unable to meet the requirements of realtime processing tasks. To solve the above problems, by embedding AI model training and inference capabilities into the network edge, edge intelligence (EI) becomes a cutting-edge direction in the field of AI. Furthermore, collaborative DNN inference among the cloud, edge, and end devices provides a promising way to boost EI. Nevertheless, at present, EI oriented collaborative DNN inference is still in its early stage, lacking systematic classification and discussion of existing research efforts. Motivated by it, we have comprehensively investigated recent studies on EI-oriented collaborative DNN inference. In this paper, we first review the background and motivation of EI. Then, we classify four typical collaborative DNN inference paradigms for EI, and analyse their characteristics and key technologies. Finally, we summarize the current challenges of collaborative DNN inference, discuss future development trends and provide future research directions.

Joint Optimization of Video-Based AI Inference Tasks in MEC-Assisted Augmented Reality Systems

The high computational complexity and energy consumption of artificial intelligence (AI) algorithms hinder their application in augmented reality (AR) systems. However, mobile edge computing (MEC) makes it possible to solve this problem. This paper considers the scene of completing video-based AI inference tasks in the MEC system. We formulate a mixed-integer nonlinear programming problem (MINLP) to reduce inference delays, energy consumption and to improve recognition accuracy. We give a simplified expression of the inference complexity model and accuracy model through derivation and experimentation. The problem is then solved iteratively by using alternating optimization. Specifically, by assuming that the offloading decision is given, the problem is decoupled into two sub-problems, i.e., the resource allocation problem for the devices set that completes the inference tasks locally, and that for the devices set that offloads tasks. For the problem of offloading decision optimization, we propose a Channel-Aware heuristic algorithm. To further reduce the complexity, we propose an alternating direction method of multipliers (ADMM) based distributed algorithm. The ADMM-based algorithm has a low computational complexity that grows linearly with the number of devices. Numerical experiments show the effectiveness of proposed algorithms. The trade-off relationship between delay, energy consumption, and accuracy is also analyzed.

A 16-nm SoC for Noise-Robust Speech and NLP Edge AI Inference With Bayesian Sound Source Separation and Attention-Based DNNs

The proliferation of personal artificial intelligence (AI) -assistant technologies with speech-based conversational AI interfaces is driving the exponential growth in the consumer Internet of Things (IoT) market. As these technologies are being applied to keyword spotting (KWS), automatic speech recognition (ASR), natural language processing (NLP), and text-to-speech (TTS) applications, it is of paramount importance that they provide uncompromising performance for context learning in long sequences, which is a key benefit of the attention mechanism, and that they work seamlessly in polyphonic environments. In this work, we present a 25-mm2 system-on-chip (SoC) in 16-nm FinFET technology, codenamed SM6, which executes end-to-end speech-enhancing attention-based ASR and NLP workloads. The SoC includes: 1) FlexASR, a highly reconfigurable NLP inference processor optimized for whole-model acceleration of bidirectional attention-based sequence-to-sequence (seq2seq) deep neural networks (DNNs); 2) a Markov random field source separation engine (MSSE), a probabilistic graphical model accelerator for unsupervised inference via Gibbs sampling, used for sound source separation; 3) a dual-core Arm Cortex A53 CPU cluster, which provides on-demand single Instruction/multiple data (SIMD) fast fourier transform (FFT) processing and performs various application logic (e.g., expectation–maximization (EM) algorithm and 8-bit floating-point (FP8) quantization); and 4) an always-ON M0 subsystem for audio detection and power management. Measurement results demonstrate the efficiency ranges of 2.6–7.8 TFLOPs/W and 4.33–17.6 Gsamples/s/W for FlexASR and MSSE, respectively; MSSE denoising performance allowing 6 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> smaller ASR model to be stored on-chip with negligible accuracy loss; and 2.24-mJ energy consumption while achieving real-time throughput, end-to-end, and per-frame ASR latencies of 18 ms.

Research and Implementation of High Computational Power for Training and Inference of Convolutional Neural Networks

Algorithms and computing power have consistently been the two driving forces behind the development of artificial intelligence. The computational power of a platform has a significant impact on the implementation cost, performance, power consumption, and flexibility of an algorithm. Currently, AI algorithmic models are mainly trained using high-performance GPU platforms, and their inferencing can be implemented using GPU, CPU, and FPGA. On the one hand, due to its high-power consumption and extreme cost, GPU is not suitable for power and cost-sensitive application scenarios. On the other hand, because the training and inference of the neural network use different computing power platforms, the data of the neural network model needs to be transmitted on platforms with varying computing power, which affects the data processing capability of the network and affects the real-time performance and flexibility of the neural network. This paper focuses on the high computing power implementation method of the integration of convolutional neural network (CNN) training and inference in artificial intelligence and proposes to implement the process of CNN training and inference by using high-performance heterogeneous architecture (HA) devices with field programmable gate array (FPGA) as the core. Numerous repeated multiplication and accumulation operations in the process of CNN training and inference have been implemented by programmable logic (PL), which significantly improves the speed of CNN training and inference and reduces the overall power consumption, thus providing a modern implementation method for neural networks in an application field that is sensitive to power, cost, and footprint. First, based on the data stream containing the training and inference process of the CNN, this study investigates methods to merge the training and inference data streams. Secondly, high-level language was used to describe the merged data stream structure, and a high-level description was converted to a hardware register transfer level (RTL) description by the high-level synthesis tool (HLS), and the intellectual property (IP) core was generated. The PS was used for overall control, data preprocessing, and result analysis, and it was then connected to the IP core via an on-chip AXI bus interface in the HA device. Finally, the integrated implementation method was tested and validated with the Xilinx HA device, and the MNIST handwritten digit validation set was used in the tests. According to the test results, compared with using a GPU, the model trained in the HA device PL achieves the same convergence rate with only 78.04 percent training time. With a processing time of only 3.31 ms and 0.65 ms for a single frame image, an average recognition accuracy of 95.697%, and an overall power consumption of only 3.22 W @ 100 MHz, the two convolutional neural networks mentioned in this paper are suitable for deployment in lightweight domains with limited power consumption.

Efficient AI Reasoning Model Based on FPGA

The artificial intelligence inference model is currently the best mathematical model based on artificial neural networks. The intelligent reasoning mode based on artificial intelligence has a large construction scale. In a completed inference, many multiplication and addition operations need to be completed. To establish an effective artificial intelligence inference model, its computational complexity is dozens or even more times higher than traditional artificial intelligence algorithms. This article focused on the research on the isomerization acceleration of FPGA (Field Programmable Gate Array). This article was looking for a method that reduces both system changes and migration, while maximizing the flexibility of FPGA programmability. In order to build on this foundation, a fast interface for fast computation using FPGA was designed. The characteristic of this project was that it could combine the FPGA acceleration platform with ordinary computers or servers, and could effectively utilize traditional computer software and toolchains. This article compared the computational time consumption between the inference model and the CPU. In the scale calculation logic scenario of the inference model designed in this article, it could achieve the same computing power as the CPU when processing 70000 multiplication calculations. The experimental results indicated that the increase in CPU computing power was greater than that of the inference model. This indicated that the larger the amount of computational data, the more significant the acceleration effect of the inference model in this article would be.

Evaluation Methods and Measures for Causal Learning Algorithms

The convenient access to copious multifaceted data has encouraged machine learning researchers to reconsider correlation-based learning and embrace the opportunity of causality-based learning, i.e., causal machine learning (causal learning). Recent years have, therefore, witnessed great effort in developing causal learning algorithms aiming to help artificial intelligence (AI) achieve human-level intelligence. Due to the lack of ground-truth data, one of the biggest challenges in current causal learning research is algorithm evaluations. This largely impedes the cross-pollination of AI and causal inference and hinders the two fields to benefit from the advances of the other. To bridge from conventional causal inference (i.e., based on statistical methods) to causal learning with Big Data (i.e., the intersection of causal inference and machine learning), in this survey, we review commonly used datasets, evaluation methods, and measures for causal learning using an evaluation pipeline similar to conventional machine learning. We focus on the two fundamental causal inference tasks and causality-aware machine learning tasks. Limitations of current evaluation procedures are also discussed. We, then, examine popular causal inference tools/packages and conclude with primary challenges and opportunities for benchmarking causal learning algorithms in the era of Big Data. The survey seeks to bring to the forefront the urgency of developing publicly available benchmarks and consensus-building standards for causal learning evaluation with observational data. In doing so, we hope to broaden the discussions and facilitate collaboration to advance the innovation and application of causal learning.

Nigeria Generating Station Capacity and Load Demand Requirement Using Regression Analysis (Least Square) and Artificial Neuro-Fuzzy Inference System (Anfis) for Reliable Power Supply

The increasing demand of electrical energy is growing exponential rate while there is mismatched between power supply and power demand. This means that there is strong record to determine the existing state of the system for purpose of future projection of power generation to avoid system collapse. Evidently, to conduct the analysis of system forecast for energy demand, many mathematical techniques and artificial machine learning which is based on artificial intelligence. This paper considered the application of least – square regression technique and artificial neuro-fuzzy inference system (ANFIS) and artificial neural network (ANN) principle, on the view to train the data set for Nigeria energy consumption (residential. Commercial and industrial) on the view to determine total system consumption pattern for reliable power supply. The data used ranges from (2000-2012) as input for the projection plan of 2013-2035, was determined for Nigeria power system. This is modelled and simulated in matlab software tool (version 19.0). the purposed artificial intelligence system based on Artificial neural fuzzy inference system (ANFIS) which provided high machine learning fixtures to predicts close to the actual value as compared to the traditional least square algorithms. The results were evaluated using mean absolute percentage error (MAPE) of 5.73%. The proposed technique ANFIS predicted load forecast (2013-2035) to provide energy data for electric utilities and to plan successfully for efficient power generation. Application of Adaptive Neuron – fuzzy inference in artificial intelligence involved both neural network (NN) and fuzzy logic (FL) principle together an adaptive neuro-fuzzy inference system (ANFIS), is an artificial neural network that is based on a fuzzy inference system. ANFIS is a very useful system to extract numerical model from numerical data since it integrates both neural network and fuzzy logic principle together, it is capable of adapting the benefits of both in a single framework. An adaptive neuro-fuzzy inference system (ANFIS) is an application of adaptive neuro-fuzzy logic that uses frame work of artificial intelligence (AI). Hence its inference system corresponds to a code of IF – THEN RULE in fuzzy neuro network. It has a learning capability to approximate non-linear functions hence adaptive neurofuzzy inference system (ANFIS) deserves to be considered as good estimator.

Bandwidth-efficient multi-task AI inference with dynamic task importance for the Internet of Things in edge computing

Over the past years, artificial intelligence (AI) models have been utilized for the Internet of Things (IoT) in applications such as remote assistance based on augmented reality (AR) in smart factories, as well as powerline inspection and precision agriculture missions performed by unmanned aerial vehicles (UAVs). Due to the limited battery capacity and computing power of these devices (e.g., AR glasses and UAVs), edge computing is recognized as a means to empower the Internet of Things (IoT) with AI. Considering that multiple AI model inference tasks (e.g., point cloud classification and fault detection) are typically performed on the same stream of sensory data (e.g., UAV camera feed), we propose TORC ( T asks- Or iented Edge C omputing) to reduce the bandwidth requirement . By incorporating AI into data transmission, the lightweight framework of TORC preserves edge computing servers’ ability to reconstruct/restore data into the original form, ensuring the proper coexistence of AI inference tasks and traditional non-AI tasks like human inspection, as well as simultaneous localization and mapping. It encodes and decodes sensory data with neural networks , whose training is driven by the AI inference tasks, in order to reduce bandwidth consumption and latency without impairing the accuracy of the AI inference tasks. Additionally, taking into account the mobility of the IoT and changes in the environment, TORC can adapt to variation in the bandwidth budget, as well as the temporally dynamic importance of AI inference tasks, without the need to train multiple neural networks for each setting. As a demonstration, empirical results conducted on the Cityscapes dataset and tasks related to autonomous driving show that, at the same level of accuracy, TORC reduces the bandwidth consumption by up to 48% and latency by up to 26%.

AI-PiM—Extending the RISC-V processor with Processing-in-Memory functional units for AI inference at the edge of IoT

The recent advances in Artificial Intelligence (AI) achieving “better-than-human” accuracy in a variety of tasks such as image classification and the game of Go have come at the cost of exponential increase in the size of artificial neural networks. This has lead to AI hardware solutions becoming severely memory-bound and scrambling to keep-up with the ever increasing “von Neumann bottleneck”. Processing-in-Memory (PiM) architectures offer an excellent solution to ease the von Neumann bottleneck by embedding compute capabilities inside the memory and reducing the data traffic between the memory and the processor. But PiM accelerators break the standard von Neumann programming model by fusing memory and compute operations together which impedes their integration in the standard computing stack. There is an urgent requirement for system-level solutions to take full advantage of PiM accelerators for end-to-end acceleration of AI applications. This article presents AI-PiM as a solution to bridge this research gap. AI-PiM proposes a hardware, ISA and software co-design methodology which allows integration of PiM accelerators in the RISC-V processor pipeline as functional execution units. AI-PiM also extends the RISC-V ISA with custom instructions which directly target the PiM functional units resulting in their tight integration with the processor. This tight integration is especially important for edge AI devices which need to process both AI and non-AI tasks on the same hardware due to area, power, size and cost constraints. AI-PiM ISA extensions expose the PiM hardware functionality to software programmers allowing efficient mapping of applications to the PiM hardware. AI-PiM adds support for custom ISA extensions to the complete software stack including compiler, assembler, linker, simulator and profiler to ensure programmability and evaluation with popular AI domain-specific languages and frameworks like TensorFlow, PyTorch, MXNet, Keras etc. AI-PiM improves the performance for vector-matrix multiplication (VMM) kernel by 17.63x and provides a mean speed-up of 2.74x for MLPerf Tiny benchmark compared to RV64IMC RISC-V baseline. AI-PiM also speeds-up MLPerf Tiny benchmark inference cycles by 2.45x (average) compared to state-of-the-art Arm Cortex-A72 processor.

Research on the Lightweight Deployment Method of Integration of Training and Inference in Artificial Intelligence

In recent years, the continuous development of artificial intelligence has largely been driven by algorithms and computing power. This paper mainly discusses the training and inference methods of artificial intelligence from the perspective of computing power. To address the issue of computing power, it is necessary to consider performance, cost, power consumption, flexibility, and robustness comprehensively. At present, the training of artificial intelligence models mostly are based on GPU platforms. Although GPUs offer high computing performance, their power consumption and cost are relatively high. It is not suitable to use GPUs as the implementation platform in certain application scenarios with demanding power consumption and cost. The emergence of high-performance heterogeneous architecture devices provides a new path for the integration of artificial intelligence training and inference. Typically, in Xilinx and Intel’s multi-core heterogeneous architecture, multiple high-performance processors and FPGAs are integrated into a single chip. When compared with the current separate training and inference method, heterogeneous architectures leverage a single chip to realize the integration of AI training and inference, providing a good balance of training and inference of different targets, further reducing the cost of training and implementation of AI inference and power consumption, so as to achieve the lightweight goals of computation, and to improve the flexibility and robustness of the system. In this paper, based on the LeNet-5 network structure, we first introduced the process of network training using a multi-core CPU in Xilinx’s latest multi-core heterogeneous architecture device, MPSoC. Then, the method of converting the network model into hardware logic implementation was studied, and the model parameters were transferred from the processing system of the device to the hardware accelerator structure, composed of programmable logic through the bus interface AXI provided on the chip. Finally, the integrated implementation method was tested and verified in Xilinx MPSoC. According to the test results, the recognition accuracy of this lightweight deployment scheme on MNIST dataset and CIFAR-10 dataset reached 99.5 and 75.4% respectively, while the average processing time of the single frame was only 2.2 ms. In addition, the power consumption of the network within the SoC hardware accelerator is only 1.363 W at 100 MHz.

CHIMERA: A 0.92-TOPS, 2.2-TOPS/W Edge AI Accelerator With 2-MByte On-Chip Foundry Resistive RAM for Efficient Training and Inference

Implementing edge artificial intelligence (AI) inference and training is challenging with current memory technologies. As deep neural networks (DNNs) grow in size, this problem is only getting worse. This article presents CHIMERA, the first non-volatile DNN chip for both edge AI training and inference using foundry on-chip resistive RAM (RRAM) macros and no off-chip memory, fabricated in 40-nm CMOS. CHIMERA&#x2019;s DNN accelerator is specifically optimized for RRAM and achieves 0.92-TOPS peak performance and 2.2-TOPS/W energy efficiency. We scale inference up to <inline-formula> <tex-math notation="LaTeX">$6\times $ </tex-math></inline-formula> larger DNNs by connecting six CHIMERAs in an illusion system with just 4&#x0025; overhead in measured execution time and 5&#x0025; in energy, enabled by communication-sparse DNN mappings that exploit RRAM non-volatility through quick chip wake-up and shutdown (<inline-formula> <tex-math notation="LaTeX">$ &lt; 33 ~\mu \text{s}$ </tex-math></inline-formula>). Our incremental edge AI training algorithm, called low-rank training, overcomes RRAM write energy, speed, and endurance challenges and achieves the same accuracy as traditional algorithms with up to <inline-formula> <tex-math notation="LaTeX">$283\times $ </tex-math></inline-formula> fewer RRAM weight update steps and <inline-formula> <tex-math notation="LaTeX">$340\times $ </tex-math></inline-formula> better energy-delay product. Combined with ENDUrance REsiliency using random Remapping (ENDURER), a hardware module that provides resilience to write endurance failures, we enable ten years of 20-samples/min incremental edge AI training.