High Inference Speed Research Articles

SOLARNet: A single stage regression based framework for efficient and robust object recognition in aerial images

Object recognition and localization play a crucial role in aerial images and their applications. The aerial images are challenging due to the large aspect ratio, arbitrary orientation, variation in scales, and non-uniform and cluttered object distribution. To address these challenges, we propose an efficient and robust model called the Simultaneous Object Localization and Recognition Network (SOLARNet), which is a fusion network that integrates two different sub-networks: PixelAttentionDetector (PD) and RotationDetector (RD). The PD considers features from different scales and cluttered objects, while RD handles rotation invariance, giving horizontal and oriented object detection results. The state-of-the-art model fails to improve accuracy when images are adversarially attacked. SOLARNet is not only efficient in terms of accuracy but also robust concerning to Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Jacobian Based Saliency Map (JSM) adversarial attacks, which is the crucial factor for any mission-critical system. We have performed experiments and achieved the accuracy on the publicly available DOTA dataset (75.00% mAP, 66.40% mAP) and DIOR dataset (88.60% mAP, 81.50% mAP) for horizontal and oriented object recognition tasks respectively while having high inference speed. Qualitative and quantitative results reported in this paper substantiate the superiority of SOLARNet over other state-of-the-art methodologies.

Real-time multiple face mask and fever detection using YOLOv3 and TensorFlow lite platforms

COVID-19 has caused disruptions to many aspects of everyday life. To reduce the impact of this pandemic, its spreading must be controlled via face mask wearing. Manually mask-checking for everybody is embarrassing and uncontrollable. Hence, the proposed technique is used to help for automatic mask-checking based on deep learning platforms with real-time surveillance live infra-red (IR) camera. In this paper, two recent object detection platforms, named, you only look once version 3 (YOLOv3) and TensorFlow lite are adopted to accomplish this task. The two models are trained with a dataset consisting of images of persons with/without masks. This work is simulated with Google Colab then tested in real-time on an embedded device mated with fast GPU called Raspberry Pi 4 model B, 8 GB RAM. A comparison is made between the two models to verify their performance in relation to their precision rate and processing time. The work of this paper is also succeeded to realize multiple face masks real-time detection up to 10 facemasks in a single scene with high inference speed. Temperature is also measured using IR touchless sensor for each person with sound alarming to alert fever. The presented detector is cheap, light, small, and fast, with 99% accuracy rate during training and testing.

Two-Stage Multi-Channel Fault Detection and Remaining Useful Life Prediction Model of Internal Gear Pumps Based on Robust-ResNet.

The internal gear pump is simple in structure, small in size and light in weight. It is an important basic component that supports the development of hydraulic system with low noise. However, its working environment is harsh and complex, and there are hidden risks related to reliability and exposure of acoustic characteristics over the long term. In order to meet the requirements of reliability and low noise, it is very necessary to make models with strong theoretical value and practical significant to accurately monitor health and predict the remaininglife of the internal gear pump. This paper proposed a multi-channel internal gear pump health status management model based on Robust-ResNet. Robust-ResNet is an optimized ResNet model based on a step factor h in the Eulerian approach to enhance the robustness of the ResNet model. This model was a two-stage deep learning model that classified the current health status of internal gear pumps, and also predicted the remaining useful life (RUL) of internal gear pumps. The model was tested in an internal gear pump dataset collected by the authors. The model was also proven to be useful in the rolling bearing data from Case Western Reserve University (CWRU). The accuracy results of health status classification model were 99.96% and 99.94% in the two datasets. The accuracy of RUL prediction stage in the self-collected dataset was 99.53%. The results demonstrated that the proposed model achieved the best performance compared to other deep learning models and previous studies. The proposed method was also proven to have high inference speed; it could also achieve real-time monitoring of gear health management. This paper provides an extremely effective deep learning model for internal gear pump health management with great application value.

Long-Term Structural State Trend Forecasting Based on an FFT–Informer Model

Machine learning has been widely applied in structural health monitoring. While most existing methods, which are limited to forecasting structural state evolution of large infrastructures. forecast the structural state in a step-by-step manner, extracting feature of structural state trends and the negative effects of data collection under abnormal conditions are big challenges. To address these issues, a long-term structural state trend forecasting method based on long sequence time-series forecasting (LSTF) with an improved Informer model integrated with Fast Fourier transform (FFT) is proposed, named the FFT–Informer model. In this method, by using FFT, structural state trend features are represented by extracting amplitude and phase of a certain period of data sequence. Structural state trend, a long sequence, can be forecasted in a one-forward operation by the Informer model that can achieve high inference speed and accuracy of prediction based on the Transformer model. Furthermore, a Hampel filter that filters the abnormal deviation of the data sequence is integrated into the Multi-head ProbSparse self-attention in the Informer model to improve forecasting accuracy by reducing the effect of abnormal data points. Experimental results on two classical data sets show that the FFT–Informer model achieves high and stable accuracy and outperforms the comparative models in forecasting accuracy. It indicates that this model can effectively forecast the long-term state trend change of a structure and is proposed to be applied to structural state trend forecasting and early damage warning.

LASNet: A Light-Weight Asymmetric Spatial Feature Network for Real-Time Semantic Segmentation

In recent years, deep learning models have achieved great success in the field of semantic segmentation, which achieve satisfactory performance by introducing a large number of parameters. However, this achievement usually leads to high computational complexity, which seriously limits the deployment of semantic segmented applications on mobile devices with limited computing and storage resources. To address this problem, we propose a lightweight asymmetric spatial feature network (LASNet) for real-time semantic segmentation. We consider the network parameters, inference speed, and performance to design the structure of LASNet, which can make the LASNet applied to embedded devices and mobile devices better. In the encoding part of LASNet, we propose the LAS module, which retains and utilize spatial information. This module uses a combination of asymmetric convolution, group convolution, and dual-stream structure to reduce the number of network parameters and maintain strong feature extraction ability. In the decoding part of LASNet, we propose the multivariate concatenate module to reuse the shallow features, which can improve the segmentation accuracy and maintain a high inference speed. Our network attains precise real-time segmentation results in a wide range of experiments. Without additional processing and pre-training, LASNet achieves 70.99% mIoU and 110.93 FPS inference speed in the CityScapes dataset with only 0.8 M model parameters.

Oestrus detection in dairy cows by using atrous spatial pyramid and attention mechanism

The mounting behaviour of dairy cows is driven by oestrus and is also related to their welfare. To improve the accuracy and speed of the detection of mounting of dairy cows in dense scenes, a detection model for the mounting behaviour of dairy cows was developed in this study based on the feature enhancement module. Firstly, an improved attention module (C3GC-3) based on global context information and convolution was proposed to capture long-distance dependence. Then, a feature enhancement module based on atrous spatial pyramid pooling and the C3GC-3 was designed to realise the multiscale fusion of high-level semantic information and improve the feature extraction ability of the model for dense dairy cows images. Finally, the K-means clustering algorithm was used to obtain new anchors for the mounting behaviour of dairy cows, and CIoU was used to optimise the loss function. A camera was installed on a farm containing 200 dairy cows for data collection. The mosaic method was used to expand the 2668 images of the mounting behaviour of dairy cows to train the model, and the remaining 675 images were used to test the model. The experimental results showed that the proposed model had a high inference speed of 61 fps, and the detection time for each image was 16.3 ms, which met the real-time performance requirements for detecting mounting in dairy cows. The mean average precision of the model was 94.3%, which was 5.9% higher than that of YOLOv5l. The proposed model showed promising results regarding the mounting of dairy cows and might be used in any weather. • A mounting behaviour dataset of group-raised dairy cows in dense scenes was proposed.. • ASPP can improve the accuracy of multi-scale object detection. • Attention mechanism can improve the feature extraction ability of the model. • Mean average precision model was 94.3%, which is 5.9% higher than that of YOLOv5l.

Dynamic single-photon 3D imaging with a sparsity-based neural network.

Deep learning is emerging as an important tool for single-photon light detection and ranging (LiDAR) with high photon efficiency and image reconstruction quality. Nevertheless, the existing deep learning methods still suffer from high memory footprint and low inference speed, which undermine their compatibility when it comes to dynamic and long-range imaging with resource-constrained devices. By exploiting the sparsity of the data, we proposed an efficient neural network architecture which significantly reduces the storage and computation overhead by skipping the inactive sites with no photon counts. In contrast with the state-of-the-art deep learning methods, our method supports one-shot processing of data frames with high spatial resolution, and achieves over 90% acceleration in computation speed without sacrificing the reconstruction quality. In addition, the speed of our method is not sensitive to the detection distance. The experiment results on public real-world dataset and our home-built system have demonstrated the outstanding dynamic imaging capability of the algorithm, which is orders of magnitude faster than the competing methods and does not require any data pruning for hardware compatibility.

Fast Temporal Graph Convolutional Model for Skeleton-Based Action Recognition.

Human action recognition has a wide range of applications, including Ambient Intelligence systems and user assistance. Starting from the recognized actions performed by the user, a better human–computer interaction can be achieved, and improved assistance can be provided by social robots in real-time scenarios. In this context, the performance of the prediction system is a key aspect. The purpose of this paper is to introduce a neural network approach based on various types of convolutional layers that can achieve a good performance in recognizing actions but with a high inference speed. The experimental results show that our solution, based on a combination of graph convolutional networks (GCN) and temporal convolutional networks (TCN), is a suitable approach that reaches the proposed goal. In addition to the neural network model, we design a pipeline that contains two stages for obtaining relevant geometric features, data augmentation and data preprocessing, also contributing to an increased performance.

DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding

Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. To guarantee effective knowledge injection, previous studies integrate models with knowledge encoders for representing knowledge retrieved from knowledge graphs. The operations for knowledge retrieval and encoding bring significant computational burdens, restricting the usage of such models in real-world applications that require high inference speed. In this paper, we propose a novel KEPLM named DKPLM that decomposes knowledge injection process of the pre-trained language models in pre-training, fine-tuning and inference stages, which facilitates the applications of KEPLMs in real-world scenarios. Specifically, we first detect knowledge-aware long-tail entities as the target for knowledge injection, enhancing the KEPLMs' semantic understanding abilities and avoiding injecting redundant information. The embeddings of long-tail entities are replaced by ``pseudo token representations'' formed by relevant knowledge triples. We further design the relational knowledge decoding task for pre-training to force the models to truly understand the injected knowledge by relation triple reconstruction. Experiments show that our model outperforms other KEPLMs significantly over zero-shot knowledge probing tasks and multiple knowledge-aware language understanding tasks. We further show that DKPLM has a higher inference speed than other competing models due to the decomposing mechanism.

VAMPIRE: vectorized automated ML pre-processing and post-processing framework for edge applications

Machine learning techniques aim to mimic the human ability to automatically learn how to perform tasks through training examples. They have proven capable of tasks such as prediction, learning and adaptation based on experience and can be used in virtually any scientific application, ranging from biomedical, robotic, to business decision applications, and others. However, the lack of domain knowledge for a particular application can make feature extraction ineffective or even unattainable. Furthermore, even in the presence of pre-processed datasets, the iterative process of optimizing Machine Learning parameters, which do not translate from one domain to another, maybe difficult for inexperienced practitioners. To address these issues, we present in this paper a Vectorized Automated ML Pre-processIng and post-pRocEssing framework, approximately named (VAMPIRE), which implements feature extraction algorithms capable of converting large time-series recordings into datasets. Also, it introduces a new concept, the Activation Engine, which is attached to the output of a Multi Layer Perceptron and extracts the optimal threshold to apply binary classification. Moreover, a tree-based algorithm is used to achieve multi-class classification using the Activation Engine. Furthermore, the internet of things gives rise to new applications such as remote sensing and communications, so consequently applying Machine Learning to improve operation accuracy, latency, and reliability is beneficial in such systems. Therefore, all classifications in this paper were performed on the edge in order to reach high accuracy with limited resources. Moreover, forecasts were applied on three unrelated biomedical datasets, and on two other pre-processed urban and activity detection datasets. Features were extracted when required, and training and testing were performed on the Raspberry Pi remotely, where high accuracy and inference speed were achieved in every experiment. Additionally, the board remained competitive in terms of power consumption when compared with a laptop which was optimized using a Graphical Processing Unit.

Lightweight and Effective Human Pose Estimation Model Based on Multi-Angle Knowledge Distillation

In the field of human pose estimation, most of the existing methods focus on improving the generalization performance of the model, while ignoring the significant efficiency issues. This leads to an increasing amount of model parameters and needs to take up more and more computing resources, which greatly reduces the practical value of the model. In order to solve this problem, we propose a novel lightweight network structure called Effective and Lightweight Pose Network (ELPN). At the same time, for the sake of alleviating the difficulty of lightweight model training, we propose a Multi-Angle Pose Distillation (MAPD) model training method that can more effectively train particularly small pose network models. In quantitative experiments, our models have excellent performance on two mainstream benchmark datasets: the MPII and the COCO. In qualitative testing, our models can accurately locate the keypoints of complex human movements. These fully demonstrates the efficiency and effectiveness of our methods. Our models have the characteristics of high precision, small size and fast inference speed. It is a cost-effective model with greater practical value.

An Efficient Method for Biomedical Entity Linking Based on Inter- and Intra-Entity Attention

Biomedical entity linking is an important research problem for many downstream tasks, such as biomedical intelligent question answering, information retrieval, and information extraction. Biomedical entity linking is the task of mapping mentions in medical texts to standard entities in a given knowledge base. Recently, BERT-based models have achieved state-of-the-art results on the biomedical entity linking task. Although this type of method is effective, it brings challenges for fine-tuning and online services in practical industries due to a large number of model parameters and long inference time. In addition, due to the numerous surface variants of biomedical mentions, it is difficult for a single matching module to achieve good results. To address the challenge, we propose an efficient biomedical entity linking method that integrates inter- and intra-entity attention to better capture the information between medical entity mentions and candidate entities themselves and each other, and the model in this paper is more lightweight. Experimental results show that our method achieves competitive performance on two biomedical benchmark datasets, NCBI and ADR, with an accuracy rate of 91.28% and 93.13%, respectively. Moreover, it also achieves comparable or even better results compared to the BERT-based entity linking method while having far fewer model parameters and very high inference speed.

Optimizing the Hyperparameter Tuning of YOLOv5 for Underwater Detection

This study optimized the latest YOLOv5 framework, including its subset models, with training on different datasets that differed in image contrast and cloudiness to assess model performances based on quantitative metrics and image processing speed. The hyperparameter in the feature-extraction phase was configured based on the learning rate and momentum and further improved based on the adaptive moment estimation (ADAM) optimizer and the function reducing-learning-rate-on-plateau to optimize the model’s training scheme. The optimized YOLOv5s achieved a better performance, with a mean average precision of 98.6% and a high inference speed of 106 frames per second. The ADAM optimizer with a detailed learning rate (0.0001) and momentum (0.99) fine-tuning yielded a sufficient convergence rate (0.69% at 55th epoch) to assist YOLOv5s in attaining a more precise detection for underwater objects.

Are Microcontrollers Ready for Deep Learning-Based Human Activity Recognition?

The last decade has seen exponential growth in the field of deep learning with deep learning on microcontrollers a new frontier for this research area. This paper presents a case study about machine learning on microcontrollers, with a focus on human activity recognition using accelerometer data. We build machine learning classifiers suitable for execution on modern microcontrollers and evaluate their performance. Specifically, we compare Random Forests (RF), a classical machine learning technique, with Convolutional Neural Networks (CNN), in terms of classification accuracy and inference speed. The results show that RF classifiers achieve similar levels of classification accuracy while being several times faster than a small custom CNN model designed for the task. The RF and the custom CNN are also several orders of magnitude faster than state-of-the-art deep learning models. On the one hand, these findings confirm the feasibility of using deep learning on modern microcontrollers. On the other hand, they cast doubt on whether deep learning is the best approach for this application, especially if high inference speed and, thus, low energy consumption is the key objective.

Efficient Single-Shot Multi-Object Tracking for Vehicles in Traffic Scenarios.

Multi-object tracking is a significant field in computer vision since it provides essential information for video surveillance and analysis. Several different deep learning-based approaches have been developed to improve the performance of multi-object tracking by applying the most accurate and efficient combinations of object detection models and appearance embedding extraction models. However, two-stage methods show a low inference speed since the embedding extraction can only be performed at the end of the object detection. To alleviate this problem, single-shot methods, which simultaneously perform object detection and embedding extraction, have been developed and have drastically improved the inference speed. However, there is a trade-off between accuracy and efficiency. Therefore, this study proposes an enhanced single-shot multi-object tracking system that displays improved accuracy while maintaining a high inference speed. With a strong feature extraction and fusion, the object detection of our model achieves an AP score of 69.93% on the UA-DETRAC dataset and outperforms previous state-of-the-art methods, such as FairMOT and JDE. Based on the improved object detection performance, our multi-object tracking system achieves a MOTA score of 68.5% and a PR-MOTA score of 24.5% on the same dataset, also surpassing the previous state-of-the-art trackers.

An Efficient Road Surveillance Approach to Detect, Recognize & Tracking Vehicles Using Deep Learning Methods

In the current scenario on the increasing number of motor vehicles day by day, so traffic regulation faces many challenges on intelligent road surveillance and governance, this is one of the important research areas in the artificial intelligence or deep learning. Among various technologies, computer vision and machine learning algorithms have the most efficient, as a huge vehicles video or image data on road is available for study. In this paper, we proposed computer vision-based an efficient approach to vehicle detection, recognition and Tracking. We merge with one-stage (YOLOv4) and two-stage (R-FCN) detectors methods to improve vehicle detection accuracy and speed results. Two-stage object detection methods provide high localization and object recognition precision, even as one-stage detectors achieve high inference and test speed. Deep-SORT tracker method applied for detects bounding boxes to estimate trajectories. We analyze the performance of the Mask RCNN benchmark, YOLOv3 and Proposed YOLOv4 + R-FCN on the UA-DETRAC dataset and study with certain parameters like Mean Average Precisions (mAP), Precision recall.

SIMPLE: SIngle-network with Mimicking and Point Learning for Bottom-up Human Pose Estimation

The practical application requests both accuracy and efficiency on multi-person pose estimation algorithms. But the high accuracy and fast inference speed are dominated by top-down methods and bottom-up methods respectively. To make a better trade-off between accuracy and efficiency, we propose a novel multi-person pose estimation framework, SIngle-network with Mimicking and Point Learning for Bottom-up Human Pose Estimation (SIMPLE). Specifically, in the training process, we enable SIMPLE to mimic the pose knowledge from the high-performance top-down pipeline, which significantly promotes SIMPLE's accuracy while maintaining its high efficiency during inference. Besides, SIMPLE formulates human detection and pose estimation as a unified point learning framework to complement each other in single-network. This is quite different from previous works where the two tasks may interfere with each other. To the best of our knowledge, both mimicking strategy between different method types and unified point learning are firstly proposed in pose estimation. In experiments, our approach achieves the new state-of-the-art performance among bottom-up methods on the COCO, MPII and PoseTrack datasets. Compared with the top-down approaches, SIMPLE has comparable accuracy and faster inference speed.

PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text

Scene text detection and recognition have been well explored in the past few years. Despite the progress, efficient and accurate end-to-end spotting of arbitrarily-shaped text remains challenging. In this work, we propose an end-to-end text spotting framework, termed PAN++, which can efficiently detect and recognize text of arbitrary shapes in natural scenes. PAN++ is based on the kernel representation that reformulates a text line as a text kernel (central region) surrounded by peripheral pixels. By systematically comparing with existing scene text representations, we show that our kernel representation can not only describe arbitrarily-shaped text but also well distinguish adjacent text. Moreover, as a pixel-based representation, the kernel representation can be predicted by a single fully convolutional network, which is very friendly to real-time applications. Taking the advantages of the kernel representation, we design a series of components as follows: 1) a computationally efficient feature enhancement network composed of stacked Feature Pyramid Enhancement Modules (FPEMs); 2) a lightweight detection head cooperating with Pixel Aggregation (PA); and 3) an efficient attention-based recognition head with Masked RoI. Benefiting from the kernel representation and the tailored components, our method achieves high inference speed while maintaining competitive accuracy. Extensive experiments show the superiority of our method. For example, the proposed PAN++ achieves an end-to-end text spotting F-measure of 64.9 at 29.2 FPS on the Total-Text dataset, which significantly outperforms the previous best method. Code will be available at: git.io/PAN.

Facial Expressions Recognition for Human-Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer.

The interaction between humans and an NAO robot using deep convolutional neural networks (CNN) is presented in this paper based on an innovative end-to-end pipeline method that applies two optimized CNNs, one for face recognition (FR) and another one for the facial expression recognition (FER) in order to obtain real-time inference speed for the entire process. Two different models for FR are considered, one known to be very accurate, but has low inference speed (faster region-based convolutional neural network), and one that is not as accurate but has high inference speed (single shot detector convolutional neural network). For emotion recognition transfer learning and fine-tuning of three CNN models (VGG, Inception V3 and ResNet) has been used. The overall results show that single shot detector convolutional neural network (SSD CNN) and faster region-based convolutional neural network (Faster R-CNN) models for face detection share almost the same accuracy: 97.8% for Faster R-CNN on PASCAL visual object classes (PASCAL VOCs) evaluation metrics and 97.42% for SSD Inception. In terms of FER, ResNet obtained the highest training accuracy (90.14%), while the visual geometry group (VGG) network had 87% accuracy and Inception V3 reached 81%. The results show improvements over 10% when using two serialized CNN, instead of using only the FER CNN, while the recent optimization model, called rectified adaptive moment optimization (RAdam), lead to a better generalization and accuracy improvement of 3%-4% on each emotion recognition CNN.

Progressive Boundary Refinement Network for Temporal Action Detection

Temporal action detection is a challenging task due to vagueness of action boundaries. To tackle this issue, we propose an end-to-end progressive boundary refinement network (PBRNet) in this paper. PBRNet belongs to the family of one-stage detectors and is equipped with three cascaded detection modules for localizing action boundary more and more precisely. Specifically, PBRNet mainly consists of coarse pyramidal detection, refined pyramidal detection, and fine-grained detection. The first two modules build two feature pyramids to perform the anchor-based detection, and the third one explores the frame-level features to refine the boundaries of each action instance. In the fined-grained detection module, three frame-level classification branches are proposed to augment the frame-level features and update the confidence scores of action instances. Evidently, PBRNet integrates the anchor-based and frame-level methods. We experimentally evaluate the proposed PBRNet and comprehensively investigate the effect of the main components. The results show PBRNet achieves the state-of-the-art detection performances on two popular benchmarks: THUMOS'14 and ActivityNet, and meanwhile possesses a high inference speed.