Tensor Processing Unit Research Articles

THPoseLite, a Lightweight Neural Network for Detecting Pose in Thermal Images

Nowadays, Smart Environments (SEs) enable the monitoring of people with physical disabilities by incorporating activity recognition. Thermal cameras are being incorporated as they preserve privacy. Some Deep Learning (DL) solutions use the pose of the users because it removes external noise. Although there are robust DL solutions in the visible spectrum, they fail in the thermal domain. Thus, we propose THPoseLite (Thermal Human Pose Lite), a Convolutional Neural Network (CNN) based on MobileNetV2 that extracts pose from Thermal Images (TIs). In a novel way, an auto-labeling approach has been developed. It includes a background removal using an optical flow estimator. It also integrates Blazepose (a pose estimator for Visible spectrum images (VSIs)) to obtain the poses in the pre-processed TIs. Results show that the pre-processing increases the percentage of detected poses by Blazepose from 19.55% to 76.85%. This allows the recording of Human Pose Estimation (HPE) datasets in the visible spectrum without requiring visible spectrum cameras or manually annotating datasets. Furthermore, THPoseLite has been embedded in an Internet of Things (IoT) device incorporating an edge Tensor Processing Unit (TPU) accelerator, which can process TIs recorded at 9 Frames Per Second (FPS) in real-time (12.28 FPS). It requires fewer than 6W of energy to run. It has been achieved using model quantization, decreasing the accuracy in estimating the poses by only 1%. The MSE of MobileNetV2 in test images is 35.48, obtaining accurate poses in 21% of the images that Blazepose is not able to detect any pose.

An optical scheme of on-chip matrixing by phase-change based tunable weighting of photonic tensor unit

In this paper, we propose an optical scheme of on-chip matrixing for matrix-vector multiplications (MVMs) by configuring each matrix element into a photonic tensor processing unit (TPU) with wavelength division multiplexing and the actively tunable weighting for scalar multiplication. The low loss chalcogenide phase change material of Ge2Sb2Se4Te (GSST) is employed and modeled with intermediate states for multilevel tunable weighting of each TPU. The dynamic electro-thermal process of GSST phase transition using ITO for electrical heating is simulated and well confirms the switchable weighting of TPUs. Simulation results reveal that a 7 V voltage pulse of 500 ns duration followed by a 9 V voltage pulse of 1500 ns duration and another pulse of 14 V voltage and 500 ns duration can set data value of ‘0’ and ‘1’ for each TPU. Taking a set of incident light with varied wavelengths and powers as input vectors, the quantized MVM outputs of photocurrents with varied amplitude are obtained after photoelectric conversion. Finally, the photonic integrated circuit level simulations by Lumerical INTERCONNECT perfectly confirm our scheme of on-chip matrixing for optical MVMs and computing.

Analysis of Convolutional Neural Networks for Facial Expression Recognition on GPU, TPU and CPU

In light of the increasing computational capacity provided by Central Processing Units (CPUs), Graphics Processing Units (GPUs), and Tensor Processing Units (TPUs), all of these were designed to speed up deep learning workloads, and the fact that this iteration of human-computer interaction is becoming more natural and social, it is clear that the field of human-computer interaction is poised for significant growth. The scientific community has found emotion recognition to be of tremendous interest and significance. Despite these advances, it is still desired that research into computational methods for identifying and recognizing emotions at the same ease as humans. This study uses Convolutional Neural Networks (CNN) for human emotion identification from facial expressions to delve deeper into this topic. The results demonstrated that training an Artificial Neural Networks (ANN) on GPUs might cut computational time by as much as 90% while accuracy could be raised up to 65%.

Short-Term Electrical Load Forecasting in Power Systems Using Deep Learning Techniques

The use of big data in deep neural networks has recently surpassed traditional machine learning techniques in many application areas. The main reasons for the use of deep neural networks are the increase in computational power made possible by graphics processing units and tensor processing units, and the new algorithms created by recurrent neural networks and CNNs. In addition to traditional machine learning methods, deep neural networks have applications in anticipating electricity load. Using a real dataset for one-step forecasting, this article compares three deep learning algorithms for short-term power load forecasting: LSTM, GRUs, and CNN. The statistics come from the Turkish city of Zonguldak and include hourly electricity usage loads and temperatures over a period of three years, commencing in 2019 and ending in 2021. The mean absolute percentage error is used to compare the performances of the techniques. Forecasts are made for twelve representative months from each season. The main reason for the significant deviations in the forecasts for January, May, September, and December is the presence of religious and national holidays in these months. This was solved by adding the information obtained from religious and national holidays to the modeling. This is not to say that CNNs are not good at capturing long-term dependencies and modeling sequential data. In all experiments, LSTM, GRUs and encoder-decoder LSTM outperformed simple CNN designs. In the future, these new architectural methods can be applied to long- or short-term electric charge predictions and their results can be compared to LSTM, GRUs and their variations.

Efficient Hardware Acceleration Techniques for Deep Learning on Edge Devices: A Comprehensive Performance Analysis

Implementing deep learning models on edge devices presents significant challenges due to the limited computing power, memory limitations, and processing power of these devices As deep learning models become more complex, cf ensuring proper execution on an edge platform is critical for real-time implementation Hardware -Addresses these challenges by exploring ways to accelerate. The problem lies in the resource-hungry nature of today’s deep learning models, which are typically designed for cloud environments with high computing capacity, making them unsuitable for edge environments with restricted resources. The main objective of this review is to analyze and compare various hardware acceleration strategies, such as graphics processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs); . . . . (pruning, quantization, and knowledge storage), memory management, and data flow optimization to improve the performance and energy efficiency of deep learning models on edge devices The results of this comprehensive performance analysis suggests that hardware accelerators can significantly improve throughput and reduce latency while maintaining acceptable levels of power consumption and accuracy. In addition to techniques such as quantization and pruning that are seen to reduce computational load and memory footprint, enabling more efficient deep learning inference on edge platforms, the study highlights trade-offs between speed, consumption power efficiency and model accuracy between for each hardware accelerator are emphasized. The findings suggest that by choosing the right hardware and applying the right optimization techniques, edge devices may be able to optimize deep learning models, meeting the requirements of real-time AI applications in resource-constrained environments handle the role there.

X-Former: In-Memory Acceleration of Transformers

Transformers have achieved great success in a wide variety of natural language processing (NLP) tasks due to the self-attention mechanism, which assigns an importance score for every word relative to other words in a sequence. However, these models are very large, often reaching hundreds of billions of parameters, and therefore require a large number of dynamic random access memory (DRAM) accesses. Hence, traditional deep neural network (DNN) accelerators such as graphical processing units (GPUs) and tensor processing units (TPUs) face limitations in processing Transformers efficiently. In-memory accelerators based on nonvolatile memory (NVM) promise to be an effective solution to this challenge, since they provide high storage density while performing massively parallel matrix–vector multiplications (MVMs) within memory arrays. However, attention score computations, which are frequently used in Transformers unlike convolutional neural networks (CNNs) and recurrent neural network (RNNs), require MVMs where both the operands change dynamically for each input. As a result, conventional NVM-based accelerators incur high write latency and write energy when used for Transformers and further suffer from the low endurance of most NVM technologies. To address these challenges, we present, a hybrid in-memory hardware accelerator that consists of both NVM and CMOS processing elements to execute transformer workloads efficiently. To improve the hardware utilization of, we also propose a sequence blocking dataflow, which overlaps the computations of the two processing elements and reduces execution time. Across several benchmarks, we show that achieves up to 69.8 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> and 13 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> improvements in latency and energy over a NVIDIA GeForce GTX 1060 GPU and up to 24.1 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> and 7.95 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> improvements in latency and energy over a state-of-the-art in-memory NVM accelerator.

Single-Event Effects on Commercial-Off-the-Shelf Edge-Processing Artificial Intelligence ASICs

Single-event effects test results are presented for heavy ions and high-energy protons on two commercial edge-processing artificial intelligence application-specific integrated circuits, particularly the Google Coral Accelerator Module, which employs a Google Coral Edge Tensor Processing Unit, and the Intel Neural Compute Stick 2, which employs the Intel Movidius Myriad X. The observed SEE signatures, probable causes within the devices, and single-event functional interrupt cross-sections are discussed. Ultimately, when SEUs or SEFIs are observed, there are negligible changes in the measured precision or accuracy of the algorithm, and rarely do the errors result in incorrect identification or classification.

Magnum.np: a PyTorch based GPU enhanced finite difference micromagnetic simulation framework for high level development and inverse design

magnum.np is a micromagnetic finite-difference library completely based on the tensor library PyTorch. The use of such a high level library leads to a highly maintainable and extensible code base which is the ideal candidate for the investigation of novel algorithms and modeling approaches. On the other hand magnum.np benefits from the device abstraction and optimizations of PyTorch enabling the efficient execution of micromagnetic simulations on a number of computational platforms including graphics processing units and potentially Tensor processing unit systems. We demonstrate a competitive performance to state-of-the-art micromagnetic codes such as mumax3 and show how our code enables the rapid implementation of new functionality. Furthermore, handling inverse problems becomes possible by using PyTorch’s autograd feature.

End-to-end differentiability and tensor processing unit computing to accelerate materials’ inverse design

Numerical simulations have revolutionized material design. However, although simulations excel at mapping an input material to its output property, their direct application to inverse design has traditionally been limited by their high computing cost and lack of differentiability. Here, taking the example of the inverse design of a porous matrix featuring targeted sorption isotherm, we introduce a computational inverse design framework that addresses these challenges, by programming differentiable simulation on TensorFlow platform that leverages automated end-to-end differentiation. Thanks to its differentiability, the simulation is used to directly train a deep generative model, which outputs an optimal porous matrix based on an arbitrary input sorption isotherm curve. Importantly, this inverse design pipeline leverages the power of tensor processing units (TPU)—an emerging family of dedicated chips, which, although they are specialized in deep learning, are flexible enough for intensive scientific simulations. This approach holds promise to accelerate inverse materials design.

Evaluating the Performance of Pre-Trained Convolutional Neural Network for Audio Classification on Embedded Systems for Anomaly Detection in Smart Cities.

Environmental Sound Recognition (ESR) plays a crucial role in smart cities by accurately categorizing audio using well-trained Machine Learning (ML) classifiers. This application is particularly valuable for cities that analyzed environmental sounds to gain insight and data. However, deploying deep learning (DL) models on resource-constrained embedded devices, such as Raspberry Pi (RPi) or Tensor Processing Units (TPUs), poses challenges. In this work, an evaluation of an existing pre-trained model for deployment on Raspberry Pi (RPi) and TPU platforms other than a laptop is proposed. We explored the impact of the retraining parameters and compared the sound classification performance across three datasets: ESC-10, BDLib, and Urban Sound. Our results demonstrate the effectiveness of the pre-trained model for transfer learning in embedded systems. On laptops, the accuracy rates reached 96.6% for ESC-10, 100% for BDLib, and 99% for Urban Sound. On RPi, the accuracy rates were 96.4% for ESC-10, 100% for BDLib, and 95.3% for Urban Sound, while on RPi with Coral TPU, the rates were 95.7% for ESC-10, 100% for BDLib and 95.4% for the Urban Sound. Utilizing pre-trained models reduces the computational requirements, enabling faster inference. Leveraging pre-trained models in embedded systems accelerates the development, deployment, and performance of various real-time applications.

Deep Learning Model Compression Techniques: Advances, Opportunities, and Perspective

Recently, deep learning (DL) models have excelled in a wide range of fields. All of these successes are built on intricate DL models. The hundreds of millions or even billions of parameters and high-performance computing graphical processing units or tensor processing units are largely responsible for their achievement. DL model integration into real-time devices with tight latency limitations, limited memory, and power-constrained requirements is the key driving force behind investigation of DL model compression techniques. Also, there is an increase in data availability that encourages multimodal fusion in DL models to boost the models' predictive accuracy. In order to create compact DL models for deployment that is memory- and computationally efficient, the data included in the network parameters is compressed as much as possible, leaving only the bits necessary to carry out the task. A better trade-off between compression rate and accuracy loss should be established to take model acceleration and compression into consideration without severely reducing the model's performance. In this paper, we examine various DL model compression techniques used for both single- modality and multi-modal deep learning tasks. We explore over numerous DL model compression methods that have advanced in a number of applications. We then come up with the benefits and drawbacks of various compression and acceleration methods such as ineffectiveness in compressing more complicated networks with dimensionality-dependent complex structures, and, ultimately, the field's future prospects are given.

Leveraging Google's Tensor Processing Units for tsunami-risk mitigation planning in the Pacific Northwest and beyond

Abstract. Tsunami-risk mitigation planning has particular importance for communities like those of the Pacific Northwest, where coastlines are extremely dynamic and a seismically active subduction zone looms large. The challenge does not stop here for risk managers: mitigation options have multiplied since communities have realized the viability and benefits of nature-based solutions. To identify suitable mitigation options for their community, risk managers need the ability to rapidly evaluate several different options through fast and accessible tsunami models, but they may lack high-performance computing infrastructure. The goal of this work is to leverage Google's Tensor Processing Unit (TPU), a high-performance hardware device accessible via the Google Cloud framework, to enable the rapid evaluation of different tsunami-risk mitigation strategies available to all communities. We establish a starting point through a numerical solver of the nonlinear shallow-water equations that uses a fifth-order weighted essentially non-oscillatory method with the Lax–Friedrichs flux splitting and a total variation diminishing third-order Runge–Kutta method for time discretization. We verify numerical solutions through several analytical solutions and benchmarks, reproduce several findings about one particular tsunami-risk mitigation strategy, and model tsunami runup at Crescent City, California whose topography comes from a high-resolution digital elevation model. The direct measurements of the simulation's performance, energy usage, and ease of execution show that our code could be a first step towards a community-based, user-friendly virtual laboratory that can be run by a minimally trained user on the cloud thanks to the ease of use of the Google Cloud platform.

Characterizing the Performance of Accelerated Jetson Edge Devices for Training Deep Learning Models

Deep Neural Network (DNN) models are becoming ubiquitous in a variety of contemporary domains such as Autonomous Vehicles, Smart cities and Healthcare. They help drones to navigate, identify suspicious activities from safety cameras, and perform diagnostics over medical imaging. Fast DNN inferencing close to the data source is enabled by a growing class of accelerated edge devices such as NVIDIA Jetson and Google Coral which host low-power Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) along with ARM CPUs in a compact form-factor to offer a superior performance-to-energy ratio. E.g., the NVIDIA Jetson AGX Xavier kit has a 512-core Volta GPU, an 8-core ARM CPU and 32GB LPDDR4x memory, that operates within 65W of power, costs US999 and is smaller than a paperback novel. Recently, there has been a push towards training DNN models on the edge [2]. This is driven by the massive growth in data collected from edge devices in Cyber-Physical Systems (CPS) and Internet of Things (IoT), the need to refresh the models periodically, the bandwidth constraints in moving all this data to Cloud data centers for training, and a heightened emphasis on privacy by retaining data on the edge. This has led to techniques like federated and geo-distributed learning that train DNN models locally on data on an edge device and aggregate them centrally. In this abstract, we summarise and highlight key results from our full paper [5].

Integrated Photonic Tensor Processing Unit for a Matrix Multiply: A Review

The explosion of artificial intelligence and machine-learning algorithms, connected to the exponential growth of the exchanged data, is driving a search for novel application-specific hardware accelerators. Among the many, the photonics field appears to be in the perfect spotlight for this global data explosion, thanks to its almost infinite bandwidth capacity associated with limited energy consumption. In this review, we will overview the major advantages that photonics has over electronics for hardware accelerators, followed by a comparison between the major architectures implemented on Photonics Integrated Circuits (PIC) for both the linear and nonlinear parts of Neural Networks. By the end, we will highlight the main driving forces for the next generation of photonic accelerators, as well as the main limits that must be overcome.

Design and Implementation of Artificial Intelligence-based Real-time Pothole Detection for IoT Smart Infrastructure

The Internet of Things (IoT) has been extensively deployed for Smart Cities due to its ability to process many different and heterogeneous end systems. The IoT innovation encourages artificial intelligence applications to process data. In Smart City infrastructure, the road is a critical component of transportation infrastructure that supports the economic, social, and cultural things of community life and various aspects of community life. Road conditions affect a variety of community activities. Good roads enhance comfort and support local businesses. However, many roads remain in bad condition, such as potholes. Various methods have been attempted to identify potholes, especially the two-dimensional imaging method. This paper proposes the real-time Artificial Intelligence detection of potholes using the Convolutional Neural Network (CNN), which leverages the Edge Tensor Processing Unit (TPU) with the MobileNet SSD v2. The system was set up on a Jetson Nano with a few extras, including a camera and GPS, to support the IoT infrastructure. Evaluation for the model consists of device implementation, model evaluation, GPS position deviation, and on-road implementation. The effectiveness is confirmed through experiments using a system test-bed that generates ideal mAP off 0.22 and recall values.

Utilization of Deep Convolutional Neural Networks for Accurate Chest X-Ray Diagnosis and Disease Detection.

Chest radiography is a widely used diagnostic imaging procedure in medical practice, which involves prompt reporting of future imaging tests and diagnosis of diseases in the images. In this study, a critical phase in the radiology workflow is automated using the three convolutional neural network (CNN) models, viz. DenseNet121, ResNet50, and EfficientNetB1 for fast and accurate detection of 14 class labels of thoracic pathology diseases based on chest radiography. These models were evaluated on an AUC score for normal versus abnormal chest radiographs using 112120 chest X-ray14 datasets containing various class labels of thoracic pathology diseases to predict the probability of individual diseases and warn clinicians of potential suspicious findings. With DenseNet121, the AUROC scores for hernia and emphysema were predicted as 0.9450 and 0.9120, respectively. Compared to the score values obtained for each class on the dataset, the DenseNet121 outperformed the other two models. This article also aims to develop an automated server to capture fourteen thoracic pathology disease results using a tensor processing unit (TPU). The results of this study demonstrate that our dataset can be used to train models with high diagnostic accuracy for predicting the likelihood of 14 different diseases in abnormal chest radiographs, enabling accurate and efficient discrimination between different types of chest radiographs. This has the potential to bring benefits to various stakeholders and improve patient care.

Data-Driven Fracture Morphology Prognosis from High Pressured Modified Proppants Based on Stochastic-Adam-RMSprop Optimizers; tf.NNR Study

Data-driven models with some evolutionary optimization algorithms, such as particle swarm optimization (PSO) and ant colony optimization (ACO) for hydraulic fracturing of shale reservoirs, have in recent times been validated as one of the best-performing machine learning algorithms. Log data from well-logging tools and physics-driven models is difficult to collate and model to enhance decision-making processes. The study sought to train, test, and validate synthetic data emanating from CMG’s numerically propped fracture morphology modeling to support and enhance productive hydrocarbon production and recovery. This data-driven numerical model was investigated for efficient hydraulic-induced fracturing by using machine learning, gradient descent, and adaptive optimizers. While satiating research curiosities, the online predictive analysis was conducted using the Google TensorFlow tool with the Tensor Processing Unit (TPU), focusing on linear and non-linear neural network regressions. A multi-structured dense layer with 1000, 100, and 1 neurons was compiled with mean absolute error (MAE) as loss functions and evaluation metrics concentrating on stochastic gradient descent (SGD), Adam, and RMSprop optimizers at a learning rate of 0.01. However, the emerging algorithm with the best overall optimization process was found to be Adam, whose error margin was 101.22 and whose accuracy was 80.24% for the entire set of 2000 synthetic data it trained and tested. Based on fracture conductivity, the data indicates that there was a higher chance of hydrocarbon production recovery using this method.

Edge computing on TPU for brain implant signal analysis

The ever-increasing number of recording sites of silicon-based probes imposes a great challenge for detecting and evaluating single-unit activities in an accurate and efficient manner. Currently separate solutions are available for high precision offline evaluation and separate solutions for embedded systems where computational resources are more limited.We propose a deep learning-based spike sorting system, that utilizes both unsupervised and supervised paradigms to learn a general feature embedding space and detect neural activity in raw data as well as predict the feature vectors for sorting. The unsupervised component uses contrastive learning to extract features from individual waveforms, while the supervised component is based on the MobileNetV2 architecture. One of the key advantages of our system is that it can be trained on multiple, diverse datasets simultaneously, resulting in greater generalizability than previous deep learning-based models.We demonstrate that the proposed model does not only reaches the accuracy of current state-of-art offline spike sorting methods but has the unique potential to run on edge Tensor Processing Units (TPUs), specialized chips designed for artificial intelligence and edge computing. We compare our model performance with state of art solutions on paired datasets as well as on hybrid recordings as well. The herein demonstrated system paves the way to the integration of deep learning-based spike sorting algorithms into wearable electronic devices, which will be a crucial element of high-end brain–computer interfaces.

Reaching for the Sky: Maximizing Deep Learning Inference Throughput on Edge Devices with AI Multi-Tenancy

The wide adoption of smart devices and Internet-of-Things (IoT) sensors has led to massive growth in data generation at the edge of the Internet over the past decade. Intelligent real-time analysis of such a high volume of data, particularly leveraging highly accurate deep learning (DL) models, often requires the data to be processed as close to the data sources (or at the edge of the Internet) to minimize the network and processing latency. The advent of specialized, low-cost, and power-efficient edge devices has greatly facilitated DL inference tasks at the edge. However, limited research has been done to improve the inference throughput (e.g., number of inferences per second) by exploiting various system techniques. This study investigates system techniques, such as batched inferencing, AI multi-tenancy, and cluster of AI accelerators, which can significantly enhance the overall inference throughput on edge devices with DL models for image classification tasks. In particular, AI multi-tenancy enables collective utilization of edge devices’ system resources (CPU, GPU) and AI accelerators (e.g., Edge Tensor Processing Units; EdgeTPUs). The evaluation results show that batched inferencing results in more than 2.4× throughput improvement on devices equipped with high-performance GPUs like Jetson Xavier NX. Moreover, with multi-tenancy approaches, e.g., concurrent model executions (CME) and dynamic model placements (DMP), the DL inference throughput on edge devices (with GPUs) and EdgeTPU can be further improved by up to 3× and 10×, respectively. Furthermore, we present a detailed analysis of hardware and software factors that change the DL inference throughput on edge devices and EdgeTPUs, thereby shedding light on areas that could be further improved to achieve high-performance DL inference at the edge.

Density Matrix Renormalization Group with Tensor Processing Units

A novel architecture is used to boost the density-matrix renormalization group, an important and widely used algorithm, impressively pushing the boundary of the current state of the art by at least an order of magnitude in complexity.