Efficiency Of Deep Neural Networks Research Articles

Adaptive habitat biogeography-based optimizer for optimizing deep CNN hyperparameters in image classification

Deep Convolutional Neural Networks (DCNNs) have shown remarkable success in image classification tasks, but optimizing their hyperparameters can be challenging due to their complex structure. This paper develops the Adaptive Habitat Biogeography-Based Optimizer (AHBBO) for tuning the hyperparameters of DCNNs in image classification tasks. In complicated optimization problems, the BBO suffers from premature convergence and insufficient exploration. In this regard, an adaptable habitat is presented as a solution to these problems; it would permit variable habitat sizes and regulated mutation. Better optimization performance and a greater chance of finding high-quality solutions across a wide range of problem domains are the results of this modification's increased exploration and population diversity. AHBBO is tested on 53 benchmark optimization functions and demonstrates its effectiveness in improving initial stochastic solutions and converging faster to the optimum. Furthermore, DCNN-AHBBO is compared to 23 well-known image classifiers on nine challenging image classification problems and shows superior performance in reducing the error rate by up to 5.14%. Our proposed algorithm outperforms 13 benchmark classifiers in 87 out of 95 evaluations, providing a high-performance and reliable solution for optimizing DNNs in image classification tasks. This research contributes to the field of deep learning by proposing a new optimization algorithm that can improve the efficiency of deep neural networks in image classification.

Introducing shape priors in Siamese networks for image classification

The efficiency of deep neural networks is increasing, and so is the amount of annotated data required for training them. We propose a solution improving the learning process of a classification network with less labeled data. Our approach is to inform the classifier of the elements it should focus on to make its decision by supplying it with some shape priors. These shape priors are expressed as binary masks, giving a rough idea of the shape of the relevant elements for a given class. We resort to Siamese architecture and feed it with image/mask pairs. By inserting shape priors, only the relevant features are retained. This provides the network with significant generalization power without requiring a specific domain adaptation step. This solution is tested on some standard cross-domain digit classification tasks and on a real-world video surveillance application. Extensive tests show that our approach outperforms the classical classifier by generating a good latent space with less training data. Code is available at https://github.com/halqasir/MG-Siamese.

Single-shot optical neural network.

Analog optical and electronic hardware has emerged as a promising alternative to digital electronics to improve the efficiency of deep neural networks (DNNs). However, previous work has been limited in scalability (input vector length K ≈ 100 elements) or has required nonstandard DNN models and retraining, hindering widespread adoption. Here, we present an analog, CMOS-compatible DNN processor that uses free-space optics to reconfigurably distribute an input vector and optoelectronics for static, updatable weighting and the nonlinearity-with K ≈ 1000 and beyond. We demonstrate single-shot-per-layer classification of the MNIST, Fashion-MNIST, and QuickDraw datasets with standard fully connected DNNs, achieving respective accuracies of 95.6, 83.3, and 79.0% without preprocessing or retraining. We also experimentally determine the fundamental upper bound on throughput (∼0.9 exaMAC/s), set by the maximum optical bandwidth before substantial increase in error. Our combination of wide spectral and spatial bandwidths enables highly efficient computing for next-generation DNNs.

CFormerFaceNet: Efficient Lightweight Network Merging a CNN and Transformer for Face Recognition

Most face recognition methods rely on deep convolutional neural networks (CNNs) that construct multiple layers of processing units in a cascaded form and employ convolution operations to fuse local features. However, these methods are not conducive to modeling the global semantic information of the face and lack attention to important facial feature regions and their spatial relationships. In this work, a Group Depth-Wise Transpose Attention (GDTA) block is designed to effectively capture both local and global representations, mitigate the issue of limited receptive fields in CNNs, and establish long-range dependencies among different feature regions. Based on GDTA and CNNs, a novel, efficient, and lightweight face recognition model called CFormerFaceNet, which combines a CNN and Transformer, is proposed. The model significantly reduces the parameters and computational cost without compromising performance, greatly improving the computational efficiency of deep neural networks in face recognition tasks. The model achieves competitive accuracy on multiple challenging benchmark face datasets, including LFW, CPLFW, CALFW, SLLFW, CFP_FF, CFP_FP, and AgeDB-30, while maintaining the minimum computational cost compared to all other advanced face recognition models. The experimental results using computers and embedded devices also demonstrate that it can meet real-time requirements in practical applications.

Evaluation Platform of Time-Domain Computing-in-Memory Circuits

Computing-in-memory (CIM) architecture is considered an effective way to reduce the energy efficiency of deep neural networks (DNNs). Compared with conventional all digital implementations, time-domain CIM designs have shown great potential with better energy efficiency and less area cost. However, due to the non-idealities of devices and PVT variations, time-domain CIM may face computational errors, resulting in the reduction of network accuracy. In this brief, we proposed an evaluation platform based on typical time-domain CIM circuits to optimize the design process. Non-idealities in CIM calculation and the impact of key parameters on the network accuracy were analyzed. Based on this evaluation platform, a 28nm time-domain CIM test chip with configurable computing channels was fabricated. Under 2–128 computing channels, the measurement results show the same trend as proposed evaluation platform. And this chip achieved 66.8 TOPS/W energy efficiency and 80.68% inference accuracy based on the CIFAR-10 dataset.

Improved Imagery Throughput via Cascaded Uncertainty Pruning on U-Net++

The extensive use of machine learning inferences in real-life earth observation and remote sensing cases has grown over recent years. Network pruning has been carefully studied in various applications to speed up the machine learning workflow, but mainstream pruning strategies often focus on specific connection significancy rather than the sample difficulty. U-Net++ as a well-versed and capable semantic segmentation deep convolutional neural network architecture, as well as its equivalents, are all facing the challenge of overconfidence, which will create barriers for a robust uncertainty-based pruning strategy to be designed. In the following study, we analyzed the efficiency of deep neural networks and semantic segmentation in satellite imagery analysis, and proposed a new tailored workf low of dynamic pruning for U-Net++ by combining the ideas of network calibration and uncertainty and defining the inference complexity of network input samples. We tested and illustrated the capability of this new workflow and delivered a successful comparative study on its effectiveness on the DeepGlobe satellite imagery road extraction dataset and how it can greatly reduce the computational cost with little performance drop.

Impact of Mixed Precision Techniques on Training and Inference Efficiency of Deep Neural Networks

Impact of Mixed Precision Techniques on Training and Inference Efficiency of Deep Neural Networks

Efficient Layer Compression without Pruning.

Network pruning is one of the chief means for improving the computational efficiency of Deep Neural Networks (DNNs). Pruning-based methods generally discard network kernels, channels, or layers, which however inevitably will disrupt original well-learned network correlation and thus lead to performance degeneration. In this work, we propose an Efficient Layer Compression (ELC) approach to efficiently compress serial layers by decoupling and merging rather than pruning. Specifically, we first propose a novel decoupling module to decouple the layers, enabling us readily merge serial layers that include both nonlinear and convolutional layers. Then, the decoupled network is losslessly merged based on the equivalent conversion of the parameters. In this way, our ELC can effectively reduce the depth of the network without destroying the correlation of the convolutional layers. To our best knowledge, we are the first to exploit the mergeability of serial convolutional layers for lossless network layer compression. Experimental results conducted on two datasets demonstrate that our method retains superior performance with a FLOPs reduction of 74.1% for VGG-16 and 54.6% for ResNet-56, respectively. In addition, our ELC improves the inference speed by 2× on Jetson AGX Xavier edge device.

Iterative deep neural networks based on proximal gradient descent for image restoration

The algorithm unfolding networks with explainability of algorithms and higher efficiency of Deep Neural Networks (DNN) have received considerable attention in solving ill-posed inverse problems. Under the algorithm unfolding network framework, we propose a novel end-to-end iterative deep neural network and its fast network for image restoration. The first one is designed making use of proximal gradient descent algorithm of variational models, which consists of denoiser and reconstruction sub-networks. The second one is its accelerated version with momentum factors. For sub-network of denoiser, we embed the Convolutional Block Attention Module (CBAM) in previous U-Net for adaptive feature refinement. Experiments on image denoising and deblurring demonstrate that competitive performances in quality and efficiency are gained by compared with several state-of-the-art networks for image restoration. Proposed unfolding DNN can be easily extended to solve other similar image restoration tasks, such as image super-resolution, image demosaicking, etc.

Deep Neural Network Regression of MANO from a Single Image

In order to improve the precision and efficiency of estimating the pose and shape of the 3D human hand from a monocular RGB image and accelerate 3D reconstruction, combining the efficiency of deep neural networks and accuracy of traditional optimization, DeepMANO is proposed. First, a DNN model is used to extract sparse features from image, then using these features to regress MANO model parameters. The regressed parameters are then used to initialize the iterative optimization routine which fits MANO to 3D joints. Finally, the parameters obtained by the routine are employed to supervise the network. Two large hand motion datasets, FreiHand and ObMan, are utilized to train the network. The approach assumes a weak perspective model. Experimental results show that the hand pose error is reduced by 52% and mesh error is reduced by 59% compared with state-of-the-art (SOTA) methods. The running time is about three times faster than SOTA methods.

Wind Farm Power Generation Control Via Double-Network-Based Deep Reinforcement Learning

A model-free deep reinforcement learning (DRL) method is proposed in this article to maximize the total power generation of wind farms through the combination of induction control and yaw control. Specifically, a novel double-network (DN)-based DRL approach is designed to generate control policies for thrust coefficients and yaw angles simultaneously and separately. Two sets of critic-actor networks are constructed to this end. They are linked by a central power-related reward, providing a coordinated control structure while inheriting the critic-actor mechanism's advantages. Compared with conventional DRL methods, the proposed DN-based DRL strategy can adapt to the distinctive and incompatible features of different control inputs, guaranteeing a reliable training process and ensuring superior performance. Also, the prioritized experience replay strategy is utilized to improve the training efficiency of deep neural networks. Simulation tests based on a dynamic wind farm simulator show that the proposed method can significantly increase the power generation for wind farms with different layouts.

TD-SRAM: Time-Domain-Based In-Memory Computing Macro for Binary Neural Networks

In-Memory Computing (IMC), which takes advantage of analog multiplication-accumulation (MAC) insides memory, is promising to alleviate the Von-Neumann bottleneck and improve the energy efficiency of deep neural networks (DNNs). Since the time-domain (TD) computing is also an energy-efficient analog computing paradigm, we present an 8kb mixed-signal IMC macro, TD-SRAM, by combining IMC with TD computing. A dual-edge single input (DESI) TD computing topology is proposed, which can significantly improve the area and power efficiencies of TD cell. The TD-SRAM bitcell consisting of a 6T DESI based TD cell and a 6T-SRAM cell supports binary DNNs. In the IMC mode, 60 columns work in parallel and 96-input binary-MAC operations are processed in each column. Implemented in a standard 40-nm CMOS process, the TD-SRAM achieves the high energy efficiency of 537 TOPS/W at 0.9-V supply. With different DNN topologies, the test chips achieve the accuracy of 95.90%-98.00% with a dual 2-bit time-to-digital converter (TDC) in the MNIST dataset.

OpEvo: An Evolutionary Method for Tensor Operator Optimization

Training and inference efficiency of deep neural networks highly rely on the performance of tensor operators on hardware platforms. Manually optimizing tensor operators has limitations in terms of supporting new operators or hardware platforms. Therefore, automatically optimizing device code configurations of tensor operators is getting increasingly attractive. However, current methods for tensor operator optimization usually suffer from poor sample-efficiency due to the combinatorial search space. In this work, we propose a novel evolutionary method, OpEvo, which efficiently explores the search spaces of tensor operators by introducing a topology-aware mutation operation based on q-random walk to leverage the topological structures over the search spaces. Our comprehensive experiment results show that compared with state-of-the-art(SOTA) methods OpEvo can find the best configuration with the lowest variance and least efforts in the number of trials and wall-clock time. All code of this work is available online.

In situ Parallel Training of Analog Neural Network Using Electrochemical Random-Access Memory.

In-memory computing based on non-volatile resistive memory can significantly improve the energy efficiency of artificial neural networks. However, accurate in situ training has been challenging due to the nonlinear and stochastic switching of the resistive memory elements. One promising analog memory is the electrochemical random-access memory (ECRAM), also known as the redox transistor. Its low write currents and linear switching properties across hundreds of analog states enable accurate and massively parallel updates of a full crossbar array, which yield rapid and energy-efficient training. While simulations predict that ECRAM based neural networks achieve high training accuracy at significantly higher energy efficiency than digital implementations, these predictions have not been experimentally achieved. In this work, we train a 3 × 3 array of ECRAM devices that learns to discriminate several elementary logic gates (AND, OR, NAND). We record the evolution of the network’s synaptic weights during parallel in situ (on-line) training, with outer product updates. Due to linear and reproducible device switching characteristics, our crossbar simulations not only accurately simulate the epochs to convergence, but also quantitatively capture the evolution of weights in individual devices. The implementation of the first in situ parallel training together with strong agreement with simulation results provides a significant advance toward developing ECRAM into larger crossbar arrays for artificial neural network accelerators, which could enable orders of magnitude improvements in energy efficiency of deep neural networks.

Efficiency of deep neural networks for joint angle modeling in digital gait assessment

Reliability and user compliance of the applied sensor system are two key issues of digital healthcare and biomedical informatics. For gait assessment applications, accurate joint angle measurements are important. Inertial measurement units (IMUs) have been used in a variety of applications and can also provide significant information on gait kinematics. However, the nonlinear mechanism of human locomotion results in moderate estimation accuracy of the gait kinematics and thus joint angles. To develop “digital twins” as a digital counterpart of body lower limb joint angles, three-dimensional gait kinematic data were collected. This work investigates the estimation accuracy of different neural networks in modeling lower body joint angles in the sagittal plane using the kinematic records of a single IMU attached to the foot. The evaluation results based on the root mean square error (RMSE) show that long short-term memory (LSTM) networks deliver superior performance in nonlinear modeling of the lower limb joint angles compared to other machine learning (ML) approaches. Accordingly, deep learning based on the LSTM architecture is a promising approach in modeling of gait kinematics using a single IMU, and thus can reduce the required physical IMUs attached on the subject and improve the practical application of the sensor system.

An acoustic model and linguistic analysis for Malayalam disyllabic words: a low resource language

Automatic Speech Recognition (ASR) has reaped a lot of attention in recent years. Despite the recent advancements in ASR, the potential for extracting the raw features from speech remains lacking. This paper proposes an Automatic Speech Recognition system on Malayalam speech data using spectrogram images and Convolutional Neural Network (CNN). The voicegram/spectrogram images of sound files are generated, which is fed into CNN. Convolutional Neural Network topology is defined with a set of Convolution and Fully Connected layers and used Softmax layer for classification. An accuracy of 93.33% achieved with this proposed model indicates that spectrogram image-based approaches have promising results in speech-based recognition. An analysis of acoustic characteristics of Malayalam disyllabic words selected to design the ASR system with formant analysis, voice onset time and spectral moments from 4000 tokens produced by 20 speakers is also conducted. A comparison between CNN model and multiple classifiers with acoustic features have been performed and proved the efficiency of deep Neural Networks over raw features.

Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks

Quantization has been proven to be a vital method for improving the inference efficiency of deep neural networks (DNNs). However, it is still challenging to strike a good balance between accuracy and efficiency while quantizing DNN weights or activation values from high-precision formats to their quantized counterparts. We propose a new method called elastic significant bit quantization (ESB) that controls the number of significant bits of quantized values to obtain better inference accuracy with fewer resources. We design a unified mathematical formula to constrain the quantized values of the ESB with a flexible number of significant bits. We also introduce a distribution difference aligner (DDA) to quantitatively align the distributions between the full-precision weight or activation values and quantized values. Consequently, ESB is suitable for various bell-shaped distributions of weights and activation of DNNs, thus maintaining a high inference accuracy. Benefitting from fewer significant bits of quantized values, ESB can reduce the multiplication complexity. We implement ESB as an accelerator and quantitatively evaluate its efficiency on FPGAs. Extensive experimental results illustrate that ESB quantization consistently outperforms state-of-the-art methods and achieves average accuracy improvements of 4.78%, 1.92%, and 3.56% over AlexNet, ResNet18, and MobileNetV2, respectively. Furthermore, ESB as an accelerator can achieve 10.95 GOPS peak performance of 1k LUTs without DSPs on the Xilinx ZCU102 FPGA platform. Compared with CPU, GPU, and state-of-the-art accelerators on FPGAs, the ESB accelerator can improve the energy efficiency by up to 65x, 11x, and 26x, respectively.

EDense: a convolutional neural network with ELM-based dense connections

The explosive growth of geospatial data is increasing requirements for automatic and efficient data learning abilities. Many deep learning methods have been widely applied for geospatial data understanding tasks, such as road networks and geospatial object detection. However, the demands for more accurate learning of high-level features require the use of deeper neural networks. To further improve the learning efficiency of deep neural networks, in this paper, we propose an improved convolutional neural network named EDense. First, we use its dense connectivity to integrate a CNN with an extreme learning machine. Then, we expand the kernels in the convolutional layers to increase the width of the network model. Furthermore, we propose one-feature EDense (OF-EDense), which is a simplified version of EDense, to fit conditions in which the number of parameters is strictly limited. Finally, the experimental results fully demonstrate the strong learning ability and high learning efficiency of EDense.

2-Bit-Per-Cell RRAM-Based In-Memory Computing for Area-/Energy-Efficient Deep Learning

In-memory computing (IMC) has emerged as a promising technique for enhancing energy efficiency of deep neural networks (DNNs). While embedded nonvolatile memory, such as resistive RAM (RRAM) is a good alternative to SRAM/DRAM for IMC owing to high density, low leakage, and nondestructive read, most prior works have not demonstrated using multilevel RRAM devices for array-level IMC operations. In this letter, we present an IMC prototype with 2-bit-per-cell RRAM devices for area-/energy-efficient DNN inference. Optimizations on four-level conductance distribution and peripheral circuits with input-splitting scheme have been performed, enabling high DNN accuracy and low area/energy consumption. The prototype chip that monolithically integrated 90-nm CMOS and 2-bit-per- cell RRAM array achieves 87% (83%) CIFAR-10 accuracy and 25 (51) TOPS/W energy efficiency at 1.2 V (0.9 V) supply. At 1.2 V, a stable accuracy of ~87% is maintained throughout 108 h.

Res-DNN: A Residue Number System-Based DNN Accelerator Unit

In this article, a technique, based on using Residue Number System (RNS) is suggested to improve the energy efficiency of Deep Neural Networks (DNNs). In the DNN architecture, which is fully RNS-based, only weights and the primary inputs in the main memory are in the binary number system (BNS). The architecture, which is called Res-DNN, offers a high energy saving while requiring higher bit count for data to handle the overflow compared to that of a BNS one. Scaling techniques in the processing elements are employed in the RNS-based computations to make the computation bit widths the same as the BNS bit width. In this architecture, the MAX pooling and ReLU activation function are implemented in the RNS format. To lower the memory usage and required memory bandwidth, we suggest a Huffman-based coding. Additionally, for accessing the weights stored in the main memory, to obtain further energy reduction, we propose a structural modification to the memory hierarchy where a lower level register file is added to the data path of these accesses. The effectiveness of the proposed architecture is evaluated under seven state-of-the-art DNNs with the datasets of ImageNet and CIFAR-10. The obtained results show that Res-DNN leads to $2.5\times $ lower energy for computations and an average of 30% overall energy reduction compared to those of the binary counterpart.