Resource-constrained Mobile Platforms Research Articles

A Real-Time Object Detection Processor With xnor-Based Variable-Precision Computing Unit

Object detection algorithms using convolutional neural networks (CNNs) achieve high detection accuracy, but it is challenging to realize real-time object detection due to their high computational complexity, especially on resource-constrained mobile platforms. In this article, we propose an algorithm-hardware co-optimization approach to designing a real-time object detection system. We first develop a compact object detection model based on a binarized neural network (BNN), which employs a new layer structure, the DenseToRes layer, to mitigate information loss due to deep quantization. We also propose an efficient object detection processor that runs object detection with high throughput using limited hardware rescources. We develop a resource-efficient processing unit supporting variable precision with minimal hardware overheads. Implemented in field-programmable gate array (FPGA), the object detection processor achieves 64.51 frames/s throughput with 64.92 mean average precision (mAP) accuracy. Compared to prior FPGA-based designs for object detection, our design achieves high throughput with competitive accuracy and lower hardware implementation costs.

Genie in the Model

Advances in deep neural networks (DNNs) have fostered a wide spectrum of intelligent mobile applications ranging from voice assistants on smartphones to augmented reality with smart-glasses. To deliver high-quality services, these DNNs should operate on resource-constrained mobile platforms and yield consistent performance in open environments. However, DNNs are notoriously resource-intensive, and often suffer from performance degradation in real-world deployments. Existing research strives to optimize the resource-performance trade-off of DNNs by compressing the model without notably compromising its inference accuracy. Accordingly, the accuracy of these compressed DNNs is bounded by the original ones, leading to more severe accuracy drop in challenging yet common scenarios such as low-resolution, small-size, and motion-blur. In this paper, we propose to push forward the frontiers of the DNN performance-resource trade-off by introducing human intelligence as a new design dimension. To this end, we explore human-in-the-loop DNNs (H-DNNs) and their automatic performance-resource optimization. We present H-Gen, an automatic H-DNN compression framework that incorporates human participation as a new hyperparameter for accurate and efficient DNN generation. It involves novel hyperparameter formulation, metric calculation, and search strategy in the context of automatic H-DNN generation. We also propose human participation mechanisms for three common DNN architectures to showcase the feasibility of H-Gen. Extensive experiments on twelve categories of challenging samples with three common DNN structures demonstrate the superiority of H-Gen in terms of the overall trade-off between performance (accuracy, latency), and resource (storage, energy, human labour).

Efficient Neural Image Decoding via Fixed-Point Inference

Recent learned image coding has emerged with superior efficiency to conventional methods. It, however, is criticized for its complexity-exhaustive deep neural network (DNN) architectures, especially on resource-constrained mobile platforms. Thus we devise a two-stage approach: first, a range pre-processing is applied to constrain the dynamic range of feature map activation by leveraging its sparsity nature with densely clustered distribution, then a layer-wise range-adaptive quantization for convolutional parameter (e.g., weight, bias), and simple yet efficient linear scaling and range-dependent normalization for activation are executed, leading to a fully fixed-point inference architecture. All arithmetic operations and associated data tensors are processed using low-bit-width fixed-point numbers, yielding significant reductions of the computational complexity, memory space, and the elimination of platform-dependent inconsistency induced by floating-point operations. We first exemplify such fixed-point inference in a DNN-based image decoder, showing the comparable coding efficiency with its native floating-point model, against the same anchor using the High-Efficiency Video Coding (HEVC)-based intra image coder. We also extend proposed approach to the super-resolution network for learned resolution scaling-based video streaming, and VGG network-based classification tasks, both of which present negligible performance loss. These evidence the generalization of our approach for efficient DNN processing of various tasks. All materials are made publicly accessible at http://njuvision.github.io/fixed-point/.

One-Shot Learning Hand Gesture Recognition Based on Lightweight 3D Convolutional Neural Networks for Portable Applications on Mobile Systems

Though deep convolutional neural networks (CNNs) have made great breakthroughs in the field of vision-based gesture recognition, however it is challenging to deploy these high-performance networks to resource-constrained mobile platforms and acquire large numbers of labeled samples for deep training of CNNs. Furthermore, there are some application scenarios with only a few samples or even a single one for a new gesture class so that the recognition method based on CNNs cannot achieve satisfactory classification performance. In this paper, a well-designed lightweight network based on I3D with spatial-temporal separable 3D convolutions and Fire module is proposed as an effective tool for the extraction of discriminative features. Then some effective capacity by deep training of large samples from related categories can be transferred and utilized to enhance the learning ability of the proposed network instead of training from scratch. In this way, the implementation of one-shot learning hand gesture recognition (OSLHGR) is carried out by a rational decision with distance measure. Moreover, a kind of mechanism of discrimination evolution with innovation of new sample and voting integration based on multi-classifiers is established to improve the learning and classification performance of the proposed method. Finally, a series of experiments and tests on the IsoGD and Jester datasets are conducted to demonstrate the effectiveness of our improved lightweight I3D. Meanwhile, a specific dataset of gestures with variant angles and directions, BSG 2.0, and the ChaLearn gesture dataset (CGD) are used for the test of OSLHGR. The results on different experiment platforms verify and validate the performance advantages of satisfied classification and real-time response speed.

Quantization and training of object detection networks with low-precision weights and activations

As convolutional neural networks have demonstrated state-of-the-art performance in object recognition and detection, there is a growing need for deploying these systems on resource-constrained mobile platforms. However, the computational burden and energy consumption of inference for these networks are significantly higher than what most low-power devices can afford. To address these limitations, this paper proposes a method to train object detection networks with low-precision weights and activations. The probability density functions of weights and activations of each layer are first directly estimated using piecewise Gaussian models. Then, the optimal quantization intervals and step sizes for each convolution layer are adaptively determined according to the distribution of weights and activations. As the most computationally expensive convolutions can be replaced by effective fixed point operations, the proposed method can drastically reduce computation complexity and memory footprint. Performing on the tiny you only look once (YOLO) and YOLO architectures, the proposed method achieves comparable accuracy to their 32-bit counterparts. As an illustration, the proposed 4-bit and 8-bit quantized versions of the YOLO model achieve a mean average precision of 62.6% and 63.9%, respectively, on the Pascal visual object classes 2012 test dataset. The mAP of the 32-bit full-precision baseline model is 64.0%.

Mobile expressive renderings: the state of the art.

Mobile applications are incorporating underlying platforms' pervasiveness in many innovative ways. Performance barriers due to resource constraints are slowly vanishing, and people are increasingly using mobile devices to perform many daily tasks they previously performed on desktop computers. Although a mobile platform's ability to handle graphics-related tasks requires further investigation, researchers have already made substantial progress. One particular related research area is nonphotorealistic rendering (NPR). NPR involves inherent abstraction, and mobile platforms offer relatively less computing power. So, a convergence of these areas can help deal with producing complex renderings on resource-constrained mobile platforms. This tutorial describes the state of NPR techniques for mobile devices, especially PDAs, tablets, and mobile phones, to motivate the development of efficient mobile NPR apps. In particular, the article addresses NPR advantages, challenges, and solutions. It also discusses mobile NPR visualizations, usability concerns, and future research directions.