Artificial Intelligence Hardware Research Articles

Special Issue on Testability and Dependability of Artificial Intelligence Hardware

Artificial intelligence (AI) hardware, including AI accelerators and neuromorphic computing processors, emerges as one new frontier in the field of computing. There is an expedited paradigm shift in embracing bold and radical innovation of computer architectures, aiming at the continuation of computing performance improvement despite the slowed-down physical device scaling. Testability and dependability of AI hardware need to be addressed before mainstream adoption, especially in latency or throughput-critical, safety-critical, mission-critical, or remotely controlled applications (e.g., computer vision, autonomous driving, smart healthcare, IoTs, and robotics).

Testability and Dependability of AI Hardware: Survey, Trends, Challenges, and Perspectives

Hardware realization of artificial intelligence (AI) requires new design styles and even underlying technologies than those used in traditional digital processors or logic circuits. Therefore, their dependability and testability requirements, challenges, and solutions are fundamentally different and unique. This survey article covers this very important topic and provides insight to such issues.—Mehdi Tahoori, Karlsruhe Institute of Technology

Special Issue on Testability and Dependability of Artificial Intelligence Hardware

<fig orientation="portrait" position="float" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <graphic orientation="portrait" position="float" xlink:href="pande-3243862.tif"/> </fig> The articles in this issue are divided into two groups: 1) the first group comprises articles that make up the Special Issue on Testability and Dependability of Artificial Intelligence Hardware and 2) the second group consists of general interest articles.

Optical and Electrical Memories for Analog Optical Computing

Key to recent successes in the field of artificial intelligence (AI) has been the ability to train a growing number of parameters which form fixed connectivity matrices between layers of nonlinear nodes. This “deep learning” approach to AI has historically required an exponential growth in processing power which far exceeds the growth in computational throughput of digital hardware as well as trends in processing efficiency. New computing paradigms are therefore required to enable efficient processing of information while drastically improving computational throughput. Emerging strategies for analog computing in the photonic domain have the potential to drastically reduce latency but require the ability to modify optical processing elements according to the learned parameters of the neural network. In this point-of-view article, we provide a forward-looking perspective on both optical and electrical memories coupled to integrated photonic hardware in the context of AI. We also show that for programmed memories, the READ energy-latency-product of photonic random-access memory (PRAM) can be orders of magnitude lower compared to electronic SRAMs. Our intent is to outline path for PRAMs to become an integral part of future foundry processes and give these promising devices relevance for emerging AI hardware.

A review of cryogenic neuromorphic hardware

The revolution in artificial intelligence (AI) brings up an enormous storage and data processing requirement. Large power consumption and hardware overhead have become the main challenges for building next-generation AI hardware. To mitigate this, neuromorphic computing has drawn immense attention due to its excellent capability for data processing with very low power consumption. While relentless research has been underway for years to minimize the power consumption in neuromorphic hardware, we are still a long way off from reaching the energy efficiency of the human brain. Furthermore, design complexity and process variation hinder the large-scale implementation of current neuromorphic platforms. Recently, the concept of implementing neuromorphic computing systems in cryogenic temperature has garnered intense interest thanks to their excellent speed and power metric. Several cryogenic devices can be engineered to work as neuromorphic primitives with ultra-low demand for power. Here, we comprehensively review the cryogenic neuromorphic hardware. We classify the existing cryogenic neuromorphic hardware into several hierarchical categories and sketch a comparative analysis based on key performance metrics. Our analysis concisely describes the operation of the associated circuit topology and outlines the advantages and challenges encountered by the state-of-the-art technology platforms. Finally, we provide insight to circumvent these challenges for the future progression of research.

Photonic machine learning with on-chip diffractive optics

Machine learning technologies have been extensively applied in high-performance information-processing fields. However, the computation rate of existing hardware is severely circumscribed by conventional Von Neumann architecture. Photonic approaches have demonstrated extraordinary potential for executing deep learning processes that involve complex calculations. In this work, an on-chip diffractive optical neural network (DONN) based on a silicon-on-insulator platform is proposed to perform machine learning tasks with high integration and low power consumption characteristics. To validate the proposed DONN, we fabricated 1-hidden-layer and 3-hidden-layer on-chip DONNs with footprints of 0.15 mm2 and 0.3 mm2 and experimentally verified their performance on the classification task of the Iris plants dataset, yielding accuracies of 86.7% and 90%, respectively. Furthermore, a 3-hidden-layer on-chip DONN is fabricated to classify the Modified National Institute of Standards and Technology handwritten digit images. The proposed passive on-chip DONN provides a potential solution for accelerating future artificial intelligence hardware with enhanced performance.

CNN Sensor Analytics With Hybrid-Float6 Quantization on Low-Power Embedded FPGAs

The use of artificial intelligence (AI) in sensor analytics is entering a new era based on the use of ubiquitous embedded connected devices. This transformation requires the adoption of design techniques that reconcile accurate results with sustainable system architectures. As such, improving the efficiency of AI hardware engines as well as backward compatibility must be considered. In this paper, we present the Hybrid-Float6 (HF6) quantization and its dedicated hardware design. We propose an optimized multiply-accumulate (MAC) hardware by reducing the mantissa multiplication to a multiplexor-adder operation. We exploit the intrinsic error tolerance of neural networks to further reduce the hardware design with approximation. To preserve model accuracy, we present a quantization-aware training (QAT) method, which in some cases improves accuracy. We demonstrate this concept in 2D convolution layers. We present a lightweight tensor processor (TP) implementing a pipelined vector dot-product. For compatibility and portability, the 6-bit floating-point (FP) is wrapped in the standard FP format, which is automatically extracted by the proposed hardware. The hardware/software architecture is compatible with TensorFlow (TF) Lite. We evaluate the applicability of our approach with a CNN-regression model for anomaly localization in a structural health monitoring (SHM) application based on acoustic emission (AE). The embedded hardware/software framework is demonstrated on XC7Z007S as the smallest Zynq-7000 SoC. The proposed implementation achieves a peak power efficiency and run-time acceleration of 5.7 GFLOPS/s/W and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$48.3\times $ </tex-math></inline-formula> , respectively.

Energy Efficient Hyperparameter Tuned Deep Neural Network to Improve Accuracy of Near-Threshold Processor

When it comes to decreasing margins and increasing energy efficiency in near-threshold and sub-threshold processors, timing error resilience may be viewed as a potentially lucrative alternative to examine. On the other hand, the currently employed approaches have certain restrictions, including high levels of design complexity, severe time constraints on error consolidation and propagation, and uncontaminated architectural registers (ARs). The design of near-threshold circuits, often known as NT circuits, is becoming the approach of choice for the construction of energy-efficient digital circuits. As a result of the exponentially decreased driving current, there was a reduction in performance, which was one of the downsides. Numerous studies have advised the use of NT techniques to chip multiprocessors as a means to preserve outstanding energy efficiency while minimising performance loss. Over the past several years, there has been a clear growth in interest in the development of artificial intelligence hardware with low energy consumption (AI). This has resulted in both large corporations and start-ups producing items that compete on the basis of varying degrees of performance and energy use. This technology’s ultimate goal was to provide levels of efficiency and performance that could not be achieved with graphics processing units or general-purpose CPUs. To achieve this objective, the technology was created to integrate several processing units into a single chip. To accomplish this purpose, the hardware was designed with a number of unique properties. In this study, an Energy Efficient Hyperparameter Tuned Deep Neural Network (EEHPT-DNN) model for Variation-Tolerant Near-Threshold Processor was developed. In order to improve the energy efficiency of artificial intelligence (AI), the EEHPT-DNN model employs several AI techniques. The notion focuses mostly on the repercussions of embedded technologies positioned at the network’s edge. The presented model employs a deep stacked sparse autoencoder (DSSAE) model with the objective of creating a variation-tolerant NT processor. The time-consuming method of modifying hyperparameters through trial and error is substituted with the marine predators optimization algorithm (MPO). This method is utilised to modify the hyperparameters associated with the DSSAE model. To validate that the proposed EEHPT-DNN model has a higher degree of functionality, a full simulation study is conducted, and the results are analysed from a variety of perspectives. This was completed so that the enhanced performance could be evaluated and analysed. According to the results of the study that compared numerous DL models, the EEHPT-DNN model performed significantly better than the other models.

Under the leadership of our president: ‘Potemkin AI’ and the Turkish approach to artificial intelligence

Recently, artificial intelligence (AI) has become a buzzword in scientific research and the majority of countries have increased their AI efforts so as not to miss first mover advantages. Turkey announced its first national AI strategy in 2021, which aims at transforming the Turkish economy. The present article argues that the proposed national AI strategy does not match with realities on the ground. There are two layers that play out as an obstacle. The first is a political layer that consists of authoritarian backsliding, and ensuing institutional decay, economic slowdown and brain drain. The second is the structural layer, which includes technical fundamentals for AI development, namely data, hardware, and talent pool. Drawing on these two layers, the present contribution concludes that Turkey needs a political and structural transformation to meet its short- and long-term targets.

Resonant Tunnelling Diode – Photodetectors for spiking neural networks

Spike-based neuromorphic devices promise to alleviate the energy greed of the artificial intelligence hardware by using spiking neural networks (SNNs), which employ neuron like units to process information through the timing of the spikes. These neuron-like devices only consume energy when active. Recent works have shown that resonant tunnelling diodes (RTDs) incorporating optoelectronic functionalities such as photodetection and light emission can play a major role on photonic SNNs. RTDs are devices that display an N-shaped current-voltage characteristics capable of providing negative differential conductance (NDC) over a range of the operating voltages. Specifically, RTD photodetectors (RTD-PDs) show promise due to their unique mixture of the structural simplicity while simultaneously providing highly complex non-linear behavior. The goal of this work is to present a systematic study of the how the thickness of the RTD-PD light absorption layers (100, 250, 500 nm) and the device size impacts on the performance of InGaAs RTD-PDs, namely on its responsivity and time response when operating in the third (1550 nm) optical transmission window. Our focus is on the overall characterization of the device optoelectronic response including the impact of the light absorption on the device static current-voltage characteristic, the responsivity and the photodetection time response. For the static characterization, the devices I-V curves were measured under dark conditions and under illumination, giving insights on the light induced I-V tunability effect. The RTD-PD responsivity was compared to the response of a commercial photodetector. The characterization of the temporal response included its capacity to generate optical induced neuronal-like electrical spike, that is, when working as an opto-to-electrical spike converter. The experimental data obtained at each characterization phase is being used for the evaluation and refinement of a behavioral model for RTD-PD devices under construction.

AI-PiM—Extending the RISC-V processor with Processing-in-Memory functional units for AI inference at the edge of IoT

The recent advances in Artificial Intelligence (AI) achieving “better-than-human” accuracy in a variety of tasks such as image classification and the game of Go have come at the cost of exponential increase in the size of artificial neural networks. This has lead to AI hardware solutions becoming severely memory-bound and scrambling to keep-up with the ever increasing “von Neumann bottleneck”. Processing-in-Memory (PiM) architectures offer an excellent solution to ease the von Neumann bottleneck by embedding compute capabilities inside the memory and reducing the data traffic between the memory and the processor. But PiM accelerators break the standard von Neumann programming model by fusing memory and compute operations together which impedes their integration in the standard computing stack. There is an urgent requirement for system-level solutions to take full advantage of PiM accelerators for end-to-end acceleration of AI applications. This article presents AI-PiM as a solution to bridge this research gap. AI-PiM proposes a hardware, ISA and software co-design methodology which allows integration of PiM accelerators in the RISC-V processor pipeline as functional execution units. AI-PiM also extends the RISC-V ISA with custom instructions which directly target the PiM functional units resulting in their tight integration with the processor. This tight integration is especially important for edge AI devices which need to process both AI and non-AI tasks on the same hardware due to area, power, size and cost constraints. AI-PiM ISA extensions expose the PiM hardware functionality to software programmers allowing efficient mapping of applications to the PiM hardware. AI-PiM adds support for custom ISA extensions to the complete software stack including compiler, assembler, linker, simulator and profiler to ensure programmability and evaluation with popular AI domain-specific languages and frameworks like TensorFlow, PyTorch, MXNet, Keras etc. AI-PiM improves the performance for vector-matrix multiplication (VMM) kernel by 17.63x and provides a mean speed-up of 2.74x for MLPerf Tiny benchmark compared to RV64IMC RISC-V baseline. AI-PiM also speeds-up MLPerf Tiny benchmark inference cycles by 2.45x (average) compared to state-of-the-art Arm Cortex-A72 processor.

The Impact of Thermal Enhance Layers on the Relaxation Effect in Analog RRAM

Computing-in-memory (CIM) with analog resistive random access memory (RRAM) has recently shown great potential in building energy-efficient hardware for artificial intelligence (AI). However, the relaxation effect of analog RRAM featuring post-programming conductance drift has become a key performance-limiting factor. In this work, a comprehensive study of the relaxation effect is presented from the analysis of its causes to the strategy for device optimization as well as the impact on CIM applications. An application-oriented quantitative indicator (relative deviation [RD]) is proposed to fairly evaluate the relaxation effect of different devices. In particular, the influence of oxygen content in different thermal enhanced layers (TELs) on the relaxation and maximum conductance value <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${G}_{max}$ </tex-math></inline-formula> of analog RRAM is studied. A theory of ternary oxide TEL is proposed to mitigate relaxation while maintaining low <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${G}_{max}$ </tex-math></inline-formula> , which is experimentally validated by TaTiO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>x</i></sub> as TEL. Furthermore, neural network simulation is carried out to analyze the requirement for RRAM relaxation for CIM applications. This work provides a useful strategy for device optimization to suppress the relaxation effect by engineering the TEL.

Advancing Science and Technology of High-k Dielectric at ECS

Historically SiO2 was the main driver as the transistor gate dielectric in CMOS technology. Once the thickness of SiO2 reached the onset of direct tunneling region (<1.5 nm) HfO2 -based high-k insulators were introduced to suppress the direct-tunneling leakage current. ECS started a symposium on Physics and Technology of High-k Gate Dielectrics in 2002 describing the evolution of dielectric science in nanoelectronics. In recent years transistor has transformed from a planar device to a three-dimensional device to a gate all around device. The electrical performance in these devices depends on the dielectric deposition process, precise selection of deposition parameters, pre-deposition surface treatments and subsequent thermal budget. By considering the interface reaction kinetics and thermal budget various atomic layer deposition (ALD) schemes were identified to get a stable dielectric with a robust interface. The deposition of Hf1-xZrxO2 on both silicon and germanium, a higher mobility channel material, were discussed. By doping HfO2 with ZrO2 and modifying the structural phase provides higher dielectric constant than the standard amorphous or monoclinic phase. Some of the reliability challenges of these dialectics based on dielectric-semiconductor interface were addressed and resolved. Several applications of high-k dielectrics have emerged including ferroelectric FETs and resistive random-access memory (ReRAM) devices that are being investigated for possible implementation of artificial intelligence hardware. Dr. Dolf Landheer presented his paper in the 1st symposium and from 2nd symposium onwards he was an organizer. He contributed significantly toward the advancing science and technology of high-k dielectrics and its dissemination for a long time in addition to other areas.

Artificial Intelligence System for Automatic Quantitative Analysis and Radiology Reporting of Leg Length Radiographs.

Leg length discrepancies are common orthopedic problems with the potential for poor functional outcomes. These are frequently assessed using bilateral leg length radiographs. The objective was to determine whether an artificial intelligence (AI)-based image analysis system can accurately interpret long leg length radiographic images. We built an end-to-end system to analyze leg length radiographs and generate reports like radiologists, which involves measurement of lengths (femur, tibia, entire leg) and angles (mechanical axis and pelvic tilt), describes presence and location of orthopedic hardware, and reports laterality discrepancies. After IRB approval, a dataset of 1,726 extremities (863 images) from consecutive examinations at a tertiary referral center was retrospectively acquired and partitioned into train/validation and test sets. The training set was annotated and used to train a fasterRCNN-ResNet101 object detection convolutional neural network. A second-stage classifier using a EfficientNet-D0 model was trained to recognize the presence or absence of hardware within extracted joint image patches. The system was deployed in a custom web application that generated a preliminary radiology report. Performance of the system was evaluated using a holdout 220 image test set, annotated by 3 musculoskeletal fellowship trained radiologists. At the object detection level, the system demonstrated a recall of 0.98 and precision of 0.96 in detecting anatomic landmarks. Correlation coefficients between radiologist and AI-generated measurements for femur, tibia, and whole-leg lengths were > 0.99, with mean error of < 1%. Correlation coefficients for mechanical axis angle and pelvic tilt were 0.98 and 0.86, respectively, with mean absolute error of < 1°. AI hardware detection demonstrated an accuracy of 99.8%. Automatic quantitative and qualitative analysis of leg length radiographs using deep learning is feasible and holds potential in improving radiologist workflow.

Query processing on tensor computation runtimes

The huge demand for computation in artificial intelligence (AI) is driving unparalleled investments in hardware and software systems for AI. This leads to an explosion in the number of specialized hardware devices, which are now offered by major cloud vendors. By hiding the low-level complexity through a tensor-based interface, tensor computation runtimes (TCRs) such as PyTorch allow data scientists to efficiently exploit the exciting capabilities offered by the new hardware. In this paper, we explore how database management systems can ride the wave of innovation happening in the AI space. We design, build, and evaluate Tensor Query Processor (TQP): TQP transforms SQL queries into tensor programs and executes them on TCRs. TQP is able to run the full TPC-H benchmark by implementing novel algorithms for relational operators on the tensor routines. At the same time, TQP can support various hardware while only requiring a fraction of the usual development effort. Experiments show that TQP can improve query execution time by up to 10X over specialized CPU- and GPU-only systems. Finally, TQP can accelerate queries mixing ML predictions and SQL end-to-end, and deliver up to 9X speedup over CPU baselines.

Palimpsest memories stored in memristive synapses.

Biological synapses store multiple memories on top of each other in a palimpsest fashion and at different time scales. Palimpsest consolidation is facilitated by the interaction of hidden biochemical processes governing synaptic efficacy during varying lifetimes. This arrangement allows idle memories to be temporarily overwritten without being forgotten, while previously unseen memories are used in the short term. While embedded artificial intelligence can greatly benefit from this functionality, a practical demonstration in hardware is missing. Here, we show how the intrinsic properties of metal-oxide volatile memristors emulate the processes supporting biological palimpsest consolidation. Our memristive synapses exhibit an expanded doubled capacity and protect a consolidated memory while up to hundreds of uncorrelated short-term memories temporarily overwrite it, without requiring specialized instructions. We further demonstrate this technology in the context of visual working memory. This showcases how emerging memory technologies can efficiently expand the capabilities of artificial intelligence hardware toward more generalized learning memories.

Guest Editorial Special Issue on New Frontiers in Extremely Efficient Reservoir Computing

With the penetration of artificial intelligence (AI) technology into industrial applications, not only computational effectiveness but also computational efficiency in machine learning (ML) methods has been increasingly demanded. Reservoir computing (RC) is an ML framework leveraging a dynamic <i>reservoir</i> for a nonlinear transformation of sequential inputs and a <i>readout</i> for mapping the reservoir state to a desired output. Since only the readout is trained with a simple learning algorithm, RC has attracted much attention as a promising approach to enhance compatibility between high computational performance and low learning cost. In addition, recent studies on physical reservoirs implemented with various physical substrates have boosted the potential of RC in the development of effective and efficient AI hardware. Therefore, it is time to further explore the new frontiers in extremely efficient RC.

Noise-mitigation strategies in physical feedforward neural networks.

Physical neural networks are promising candidates for next generation artificial intelligence hardware. In such architectures, neurons and connections are physically realized and do not leverage digital concepts with their practically infinite signal-to-noise ratio to encode, transduce, and transform information. They, therefore, are prone to noise with a variety of statistical and architectural properties, and effective strategies leveraging network-inherent assets to mitigate noise in a hardware-efficient manner are important in the pursuit of next generation neural network hardware. Based on analytical derivations, we here introduce and analyze a variety of different noise-mitigation approaches. We analytically show that intra-layer connections in which the connection matrix's squared mean exceeds the mean of its square fully suppress uncorrelated noise. We go beyond and develop two synergistic strategies for noise that is uncorrelated and correlated across populations of neurons. First, we introduce the concept of ghost neurons, where each group of neurons perturbed by correlated noise has a negative connection to a single neuron, yet without receiving any input information. Second, we show that pooling of neuron populations is an efficient approach to suppress uncorrelated noise. As such, we developed a general noise-mitigation strategy leveraging the statistical properties of the different noise terms most relevant in analog hardware. Finally, we demonstrate the effectiveness of this combined approach for a trained neural network classifying the modified National Institute of Standards and Technology handwritten digits, for which we achieve a fourfold improvement of the output signal-to-noise ratio. Our noise mitigation lifts the 92.07% classification accuracy of the noisy neural network to 97.49%, which is essentially identical to the 97.54% of the noise-free network.

Effect of biologically-motivated energy constraints on liquid state machine dynamics and classification performance

Abstract Equipping edge devices with intelligent behavior opens up new possibilities for automating the decision making in extreme size, weight, and power-constrained application domains. To this end, several recent lines of research are aimed at the design of artificial intelligence hardware accelerators that have significantly reduced footprint and power demands compared to conventional CPU/GPU systems. However, despite some key advancements, the majority of work in this area assumes that there is an unlimited supply of energy available for computation, which is not realistic in the case of battery-powered and energy harvesting devices. In this paper, we address this gap by exploring the computational effects of energy constraints on a popular class of brain-inspired spiking neural networks–liquid state machines (LSMs). Energy constraints were applied by limiting the spiking activity in subsets of LSM neurons. We tested our designs on two biosignal processing tasks: epileptic seizure detection and biometric gait identification. For both tasks, we show that energy constraints can significantly improve classification accuracy. This demonstrates that in the design of neuromorphic systems, reducing energy and increasing performance are not always competing goals.

The Other Artificial Intelligence Hardware Problem

The consolidation of leading-edge semiconductor manufacturing in one company in one country poses two problems: a short-term strategic risk and a long-term innovation risk. Both problems remain underexplored because advances in artificial intelligence (AI) algorithms have compensated for a relative lack of advances in AI hardware.