Full Precision Research Articles

Technical characteristics of global navigation satellite systems and their role in precision agriculture

Originally developed for military applications, Global Navigation Satellite Systems (GNSS) have become a part of our daily lives. It is also essential in modern agricultural production. For example, precision crop production and precision livestock farming work with a lot of position (stationary or dynamic) data provided by navigation systems. The first global navigation system was called Global Positioning System (GPS). Since then, a number of similar independent systems have been developed because of the fact that the service can be restricted or even disabled due to an event. Modern terrestrial navigation devices measure their position across multiple GNSS systems, thus increasing accuracy.
 In this article we will illustrate the general operational features with an example of the oldest navigation system, the NAVSTAR GPS of the USA. In addition, we present the main features of Galileo, the system set up by the European Union, which is expected to be fully operational in 2020. We present the market situation for GNSS systems and illustrate its use through some precision farming applications.
 We conducted in-depth interviews to learn about farmers’ knowledge and motivations about GNSS technology and precision farming. All those who are not yet or only partially precision farmers want to switch to full precision farming in the long run. Respondents have all heard of Galileo, but their knowledge is partial. Users are waiting for Galileo to be fully operational.

Exploring the Connection Between Binary and Spiking Neural Networks.

On-chip edge intelligence has necessitated the exploration of algorithmic techniques to reduce the compute requirements of current machine learning frameworks. This work aims to bridge the recent algorithmic progress in training Binary Neural Networks and Spiking Neural Networks—both of which are driven by the same motivation and yet synergies between the two have not been fully explored. We show that training Spiking Neural Networks in the extreme quantization regime results in near full precision accuracies on large-scale datasets like CIFAR-100 and ImageNet. An important implication of this work is that Binary Spiking Neural Networks can be enabled by “In-Memory” hardware accelerators catered for Binary Neural Networks without suffering any accuracy degradation due to binarization. We utilize standard training techniques for non-spiking networks to generate our spiking networks by conversion process and also perform an extensive empirical analysis and explore simple design-time and run-time optimization techniques for reducing inference latency of spiking networks (both for binary and full-precision models) by an order of magnitude over prior work. Our implementation source code and trained models are available at https://github.com/NeuroCompLab-psu/SNN-Conversion.

Inexact methods for the low rank solution to large scale Lyapunov equations

The rational Krylov subspace method (RKSM) and the low-rank alternating directions implicit (LR-ADI) iteration are established numerical tools for computing low-rank solution factors of large-scale Lyapunov equations. In order to generate the basis vectors for the RKSM, or extend the low-rank factors within the LR-ADI method, the repeated solution to a shifted linear system of equations is necessary. For very large systems this solve is usually implemented using iterative methods, leading to inexact solves within this inner iteration (and therefore to “inexact methods”). We will show that one can terminate this inner iteration before full precision has been reached and still obtain very good accuracy in the final solution to the Lyapunov equation. In particular, for both the RKSM and the LR-ADI method we derive theory for a relaxation strategy (e.g. increasing the solve tolerance of the inner iteration, as the outer iteration proceeds) within the iterative methods for solving the large linear systems. These theoretical choices involve unknown quantities, therefore practical criteria for relaxing the solution tolerance within the inner linear system are then provided. The theory is supported by several numerical examples, which show that the total amount of work for solving Lyapunov equations can be reduced significantly.

RTN: Reparameterized Ternary Network

To deploy deep neural networks on resource-limited devices, quantization has been widely explored. In this work, we study the extremely low-bit networks which have tremendous speed-up, memory saving with quantized activation and weights. We first bring up three omitted issues in extremely low-bit networks: the squashing range of quantized values; the gradient vanishing during backpropagation and the unexploited hardware acceleration of ternary networks. By reparameterizing quantized activation and weights vector with full precision scale and offset for fixed ternary vector, we decouple the range and magnitude from direction to extenuate above problems. Learnable scale and offset can automatically adjust the range of quantized values and sparsity without gradient vanishing. A novel encoding and computation pattern are designed to support efficient computing for our reparameterized ternary network (RTN). Experiments on ResNet-18 for ImageNet demonstrate that the proposed RTN finds a much better efficiency between bitwidth and accuracy and achieves up to 26.76% relative accuracy improvement compared with state-of-the-art methods. Moreover, we validate the proposed computation pattern on Field Programmable Gate Arrays (FPGA), and it brings 46.46 × and 89.17 × savings on power and area compared with the full precision convolution.

Sparse BD-Net

In this work, we propose a multiplication-less binarized depthwise-separable convolution neural network, called BD-Net. BD-Net is designed to use binarized depthwise separable convolution block as the drop-in replacement of conventional spatial-convolution in deep convolution neural network (DNN). In BD-Net, the computation-expensive convolution operations (i.e., Multiplication and Accumulation) are converted into energy-efficient Addition/Subtraction operations. For further compressing the model size while maintaining the dominant computation in addition/subtraction, we propose a brand-new sparse binarization method with a hardware-oriented structured sparsity pattern. To successfully train such sparse BD-Net, we propose and leverage two techniques: (1) a modified group-lasso regularization whose group size is identical to the capacity of basic computing core in accelerator and (2) a weight penalty clipping technique to solve the disharmony issue between weight binarization and lasso regularization. The experiment results show that the proposed sparse BD-Net can achieve comparable or even better inference accuracy, in comparison to the full precision CNN baseline. Beyond that, a BD-Net customized process-in-memory accelerator is designed using SOT-MRAM, which owns characteristics of high channel expansion flexibility and computation parallelism. Through the detailed analysis from both software and hardware perspectives, we provide an intuitive design guidance for software/hardware co-design of DNN acceleration on mobile embedded systems. Note that this journal submission is the extended version of our previous published paper in ISVLSI 2018 [24].

An AI approach to automated magnetic formation mapping beneath cover

SummaryMost regional scale magnetic maps are dominated by the magnetic characteristics of steeply dipping basement units truncated by an unconformity surface. It is easy to demonstrate that 80 to 90% of each total field magnetic anomaly is contributed by this intersecting surface. We approach this problem by mapping the boundaries between contrasting magnetic units along each line in the magnetic survey using the full precision of the line data and 3D information from the magnetic gradient tensor. Additionally, we derive the azimuth of each boundary, depth to the unconformity and magnetic properties of the anomalous units. The segments are overlain on any image such as existing geological maps, satellite imagery, gravity or magnetic imagery to provide a new geological interpretation concept. This method provides a new way to interpret new and old magnetic surveys.Eigenvector analysis of the magnetic tensor and normalised source strength (NSS) are combined with an artificial intelligence (AI) approach to estimate the basement properties. The method is applied to full tensor magnetic survey data or a grid of the total magnetic intensity data is processed using FFT transformations to derive the magnetic gradient tensor. These data are used as input to the pre-trained AI process for calculation of depth, width, azimuth, magnetic susceptibility and magnetisation direction. The rock properties and depth information can be used for 3D visualisation of the unconformity and 2D mapping of the magnetic lithology of the unconformity surface.

Embedded Fault-Tolerant Accelerator Architecture for Synthetic-Aperture Radar Backprojection

This paper presents an optimized multicore embedded architecture to speed up and improve the resilience of the execution of a backprojection algorithm targeting onboard synthetic-aperture radar (SAR) imaging systems for the space environment. The proposed architecture produces SAR images by computing approximations instead of the full precision of some arithmetic operations. The proposed solution was implemented on a Xilinx SoC device with a dual-core processor. The novel fault-tolerant system exhibits graceful degradation characteristics as it is able to produce images considered acceptable, that is, signal-to-noise ratio and on average with an SNR 0.65 dB less than the fault-free image, while providing a time reduction up to 33%, and a decrease up to 40% in energy consumption.

SCELib4.0: The new program version for computing molecular properties in the Single Center Approach

SCELib4.0: The new program version for computing molecular properties in the Single Center Approach

Producing marine geophysical archive files from raw underway data

Preserving costly marine geophysical trackline data is of paramount importance but variability in priorities, funding, personnel, and technology impact our data archival capacity. We have addressed one crucial facet of this dilemma by devising an open source approach to merge and reduce underway geophysical data and to generate marine geophysical archive files using common command line programs along with the Generic Mapping Tools and its mgd77 supplement. Archive files generated using this approach retain full precision and may be converted automatically to MGD77T, MGD77+, as well as MGD77 formats. We successfully applied the approach to 340 geophysical data sets acquired by R/V Kilo Moana from 2002 to 2018 and in the near term we plan to submit the non-proprietary archive files to the National Centers for Environmental Information’s trackline geophysics archive. We encourage international oceanographic communities to explore our methodology as a larger user-base will strengthen the software and the procedures.

A High-Throughput and Power-Efficient FPGA Implementation of YOLO CNN for Object Detection

Convolutional neural networks (CNNs) require numerous computations and external memory accesses. Frequent accesses to off-chip memory cause slow processing and large power dissipation. For real-time object detection with high throughput and power efficiency, this paper presents a Tera-OPS streaming hardware accelerator implementing a you-only-look-once (YOLO) CNN. The parameters of the YOLO CNN are retrained and quantized with the PASCAL VOC data set using binary weight and flexible low-bit activation. The binary weight enables storing the entire network model in block RAMs of a field-programmable gate array (FPGA) to reduce off-chip accesses aggressively and, thereby, achieve significant performance enhancement. In the proposed design, all convolutional layers are fully pipelined for enhanced hardware utilization. The input image is delivered to the accelerator line-by-line. Similarly, the output from the previous layer is transmitted to the next layer line-by-line. The intermediate data are fully reused across layers, thereby eliminating external memory accesses. The decreased dynamic random access memory (DRAM) accesses reduce DRAM power consumption. Furthermore, as the convolutional layers are fully parameterized, it is easy to scale up the network. In this streaming design, each convolution layer is mapped to a dedicated hardware block. Therefore, it outperforms the “one-size-fits-all” designs in both performance and power efficiency. This CNN implemented using VC707 FPGA achieves a throughput of 1.877 tera operations per second (TOPS) at 200 MHz with batch processing while consuming 18.29 W of on-chip power, which shows the best power efficiency compared with the previous research. As for object detection accuracy, it achieves a mean average precision (mAP) of 64.16% for the PASCAL VOC 2007 data set that is only 2.63% lower than the mAP of the same YOLO network with full precision.

Fast, reliable and unrestricted iterative computation of Gauss–Hermite and Gauss–Laguerre quadratures

Methods for the computation of classical Gaussian quadrature rules are described which are effective both for small and large degree. These methods are reliable because the iterative computation of the nodes has guaranteed convergence, and they are fast due to their fourth-order convergence and its asymptotic exactness for an appropriate selection of the variables. For Gauss--Hermite and Gauss--Laguerre quadratures, local Taylor series can be used for computing efficiently the orthogonal polynomials involved, with exact initial values for the Hermite case and first values computed with a continued fraction for the Laguerre case. The resulting algorithms have almost unrestricted validity with respect to the parameters. Full relative precision is reached for the Hermite nodes, without any accuracy loss and for any degree, and a mild accuracy loss occurs for the Hermite and Laguerre weights as well as for the Laguerre nodes. These fast methods are exclusively based on convergent processes, which, together with the high order of convergence of the underlying iterative method, makes them particularly useful for high accuracy computations. We show examples of very high accuracy computations (of up to $1000$ digits of accuracy).

An Optimized Recurrent Unit for Ultra-Low-Power Keyword Spotting

There is growing interest in being able to run neural networks on sensors, wearables and internet-of-things (IoT) devices. However, the computational demands of neural networks make them difficult to deploy on resource-constrained edge devices. To meet this need, our work introduces a new recurrent unit architecture that is specifically adapted for on-device low power acoustic event detection (AED). The proposed architecture is based on the gated recurrent unit ('GRU' -- introduced by Cho et al. [9]) but features optimizations that make it implementable on ultra-low power micro-controllers such as the Arm Cortex M0+. Our new architecture, the Embedded Gated Recurrent Unit (eGRU) is demonstrated to be highly efficient and suitable for short-duration AED and keyword spotting tasks. A single eGRU cell is 60× faster and 10× smaller than a GRU cell. Despite its optimizations, eGRU compares well with GRU across tasks of varying complexities. The practicality of eGRU is investigated in a wearable acoustic event detection application. An eGRU model is implemented and tested on the Arm Cortex M0-based Atmel ATSAMD21E18 processor. The Arm M0+ implementation of the eGRU model compares favorably with a full precision GRU that is running on a workstation. The embedded eGRU model achieves a classification accuracy 95.3%, which is only 2% less than the full precision GRU.

Full arch precision of six intraoral scanners in vitro

PurposeIntraoral scanners may offer an alternative to traditional impressions. That intraoral scanners produce precise scans is essential. Popular methods used to evaluate precision tend to rely on mean distance deviation between repeated scans. Mean value measurements may underestimate errors resulting in misleading conclusions and clinical decisions. This study investigated the precision of six intraoral scanners using the traditional method of measuring mean error, and a proposed method considering only the most extreme and clinically relevant aspects of a scan. MethodsAn edentulous model was scanned five times using six intraoral scanners. The repeated scans were aligned, uniformly trimmed and mean surface deviation measured across all 20 scan combinations within each scanner group. All scan combinations were then measured by arranging scan vertices from greatest to smallest unsigned distance from its compared scan and measuring the median value within the 1% of most greatly deviating points. Traditional mean deviation results and upper-bound deviations were compared. ResultsThe upper-bound deviation within a scan reported errors up to two times greater than those found when measuring global mean distances. Results revealed clinically relevant errors of more than 0.3mm in scans produced by the Planmeca and Dentalwings scanners, findings not seen when measuring mean distance error of the complete scan. ConclusionsUpper-bound deviation of a cropped scan may provide a clinically useful metric for scanner precision. The Aadva, 3Shape, CEREC and TDS produced scans potentially appropriate for clinical use while Planmeca and Dentalwings produced deviations greater than 0.3mm when measuring the upper-bound deviation.

A Review of Binarized Neural Networks

In this work, we review Binarized Neural Networks (BNNs). BNNs are deep neural networks that use binary values for activations and weights, instead of full precision values. With binary values, BNNs can execute computations using bitwise operations, which reduces execution time. Model sizes of BNNs are much smaller than their full precision counterparts. While the accuracy of a BNN model is generally less than full precision models, BNNs have been closing accuracy gap and are becoming more accurate on larger datasets like ImageNet. BNNs are also good candidates for deep learning implementations on FPGAs and ASICs due to their bitwise efficiency. We give a tutorial of the general BNN methodology and review various contributions, implementations and applications of BNNs.

Binary neural networks for speech recognition

Recently, deep neural networks (DNNs) significantly outperform Gaussian mixture models in acoustic modeling for speech recognition. However, the substantial increase in computational load during the inference stage makes deep models difficult to directly deploy on low-power embedded devices. To alleviate this issue, structure sparseness and low precision fixed-point quantization have been applied widely. In this work, binary neural networks for speech recognition are developed to reduce the computational cost during the inference stage. A fast implementation of binary matrix multiplication is introduced. On modern central processing unit (CPU) and graphics processing unit (GPU) architectures, a 5–7 times speedup compared with full precision floatingpoint matrix multiplication can be achieved in real applications. Several kinds of binary neural networks and related model optimization algorithms are developed for large vocabulary continuous speech recognition acoustic modeling. In addition, to improve the accuracy of binary models, knowledge distillation from the normal full precision floating-point model to the compressed binary model is explored. Experiments on the standard Switchboard speech recognition task show that the proposed binary neural networks can deliver 3–4 times speedup over the normal full precision deep models. With the knowledge distillation from the normal floating-point models, the binary DNNs or binary convolutional neural networks (CNNs) can restrict the word error rate (WER) degradation to within 15.0%, compared to the normal full precision floating-point DNNs or CNNs, respectively. Particularly for the binary CNN with binarization only on the convolutional layers, the WER degradation is very small and is almost negligible with the proposed approach.

Development of a 3-PRR Precision Tracking System with Full Closed-Loop Measurement and Control.

A 3-PRR (three links with each link consisting of a prismatic pair and two rotating pairs) parallel platform was designed for application in a vacuum environment. To meet the requirement of high tracking accuracy of the 3-PRR parallel platform, a full closed-loop control precision tracking system with laser displacement sensors and linear grating encoders was analysed and implemented. Equally-spaced laser displacement sensors and linear grating encoders were adopted not only for measurement but also for feedback control. A feed-forward control method was applied for comparison before conducting the closed-loop feedback control experiments. The closed-loop control experiments were conducted by adopting the PI (proportion and integration) feedback control and RBF (radial basis function) neural network control algorithms. The experimental results demonstrate that the feed-forward control, PI feedback control, and RBF neural-network control algorithms all have a better control effect than that of semi-closed-loop control, which proves the validity of the designed full closed-loop control system based on the combination of laser displacement sensors and linear grating encoders.

A Scalable Near-Memory Architecture for Training Deep Neural Networks on Large In-Memory Datasets

Most investigations into near-memory hardware accelerators for deep neural networks have primarily focused on inference, while the potential of accelerating training has received relatively little attention so far. Based on an in-depth analysis of the key computational patterns in state-of-the-art gradient-based training methods, we propose an efficient near-memory acceleration engine called NTX that can be used to train state-of-the-art deep convolutional neural networks at scale. Our main contributions are: (i) a loose coupling of RISC-V cores and NTX co-processors reducing offloading overhead by <inline-formula><tex-math notation="LaTeX">$7\times$</tex-math></inline-formula> over previously published results; (ii) an optimized IEEE 754 compliant data path for fast high-precision convolutions and gradient propagation; (iii) evaluation of near-memory computing with NTX embedded into residual area on the Logic Base die of a Hybrid Memory Cube; and (iv) a scaling analysis to meshes of HMCs in a data center scenario. We demonstrate a <inline-formula><tex-math notation="LaTeX">$2.7\times$</tex-math></inline-formula> energy efficiency improvement of NTX over contemporary GPUs at <inline-formula><tex-math notation="LaTeX">$4.4\times$</tex-math></inline-formula> less silicon area, and a compute performance of 1.2 Tflop/s for training large state-of-the-art networks with full floating-point precision. At the data center scale, a mesh of NTX achieves above 95 percent parallel and energy efficiency, while providing <inline-formula><tex-math notation="LaTeX">$2.1\times$</tex-math></inline-formula> energy savings or <inline-formula><tex-math notation="LaTeX">$3.1\times$</tex-math></inline-formula> performance improvement over a GPU-based system.

A study on observed ultrasonic motor-induced magnetic resonance imaging (MRI) artifacts

BackgroundThe safe performance of magnetic resonance imaging (MRI)-guided robot-assisted interventions requires full control and high precision of assistive devices. Because many currently available tools are not MRI-compatible, the characterization of existing tools and development of new ones are necessary. The purpose of this research is to identify and minimize the image artifacts generated by a USM in MR images. MethodsThe behavior of an ultrasonic motor (USM), the most common MRI-safe actuator, in a high-field scanner was investigated. The motor was located in three orientations with respect to the bore axis with the power on or off. The induced image artifacts were compared across four sequences. Three artifact reduction methods (employing ultrashort sequences, slice thickness reductions, and bandwidth increments) were tested. ResultsSignal voids, pileups, and geometric distortions were observed when the motor was off. The artifact size was minimal when the motor shaft was aligned with the bore axis. In addition to the above artifacts, zipper and motion artifacts were noted when the motor was running, and these artifacts increased with increasing motor speed. Increasing the bandwidth slightly reduced the artifacts. However, decreasing the slice thickness from 5 mm to 3 mm and from 5 mm to 1 mm reduced artifact size from 30% to 40% and from 60% to 75%, respectively. ConclusionThe image artifacts were due to the non-homogenous nature of the static and gradient fields caused by the motor structure. The operating motor interferes with the RF field, causing zipper and motion artifacts.

Stochastic Quantization for Learning Accurate Low-Bit Deep Neural Networks

Low-bit deep neural networks (DNNs) become critical for embedded applications due to their low storage requirement yet high computing efficiency. However, they suffer much from the non-negligible accuracy drop. This paper proposes the stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The motivation is due to the following observation. Existing training algorithms approximate the real-valued weights with low-bit representation all together in each iteration. The quantization error may be small for some elements/filters, while is remarkable for others, which leads to inappropriate gradient directions during training, and thus brings notable accuracy drop. Instead, SQ quantizes a portion of elements/filters to low-bit values with a stochastic probability inversely proportional to the quantization error, while keeping the other portion unchanged with full precision. The quantized and full precision portions are updated with their corresponding gradients separately in each iteration. The SQ ratio, which measures the ratio of the quantized weights to all weights, is gradually increased until the whole network is quantized. This procedure can greatly compensate for the quantization error and thus yield better accuracy for low-bit DNNs. Experiments show that SQ can consistently and significantly improve the accuracy for different low-bit DNNs on various datasets and various network structures, no matter whether activation values are quantized or not.

Blended coarse gradient descent for full quantization of deep neural networks

Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights, hence mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36\% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46\%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data, and prove that the expected coarse gradient correlates positively with the underlying true gradient.