Front-end Network Research Articles

A Survey on Regression-Based Crowd Counting Techniques

Traditional detect and count strategy can’t well handle the extremely crowded footage in computer vision-based counting task. In recent years, deep learning approaches have been widely explored to tackle this challenge. By regressing visual features to density map, the total crowd number can be predicted while avoids the detection of their actual positions. Efforts of improving performance distribute at various phases of the detecting pipeline, such as feature extraction and eliminating deviation of regressed density map etc. In this article, we conduct a thorough review on the most representative and state-of-the-art techniques. The efforts are systematically categorized into three topics: the evolving of front-end network, the handling of unbalanced density map prediction, and the selection of loss function. After the evaluation of most significant techniques, innovations of the state-of-the-art are inspected in detail to analyze specific reasons to achieve high performances. As conclusion, possible directions of enhancement are discussed to provide insights of future research.

Keypoint-Based Automated Component Placement Inspection for Printed Circuit Boards

This study aims to develop novel automated computer vision algorithms and systems for component replacement inspection for printed circuit boards (PCBs). The proposed algorithms are able to identify the locations and sizes of different components. They are object detection algorithms based on key points of the target components. The algorithms can be implemented as neural networks consisting of two portions: frontend networks and backend networks. The frontend networks are used for the feature extractions of input images. The backend networks are adopted to produce component inspection results. Each component class can has its own frontend and backend networks. In this way, the neural model for the component class can be effectively reused for different PCBs. To reduce the computation time for the inference of the networks, different component classes can share the same frontend networks. A two-stage training process is proposed to effectively explore features of different components for accurate component inspection. The proposed algorithm has the advantages of simplicity in training for data collection, high accuracy in defect detection, and high reusability and flexibility for online inspection. The algorithm is an effective alternative for automated inspection in smart factories, with growing demand for product quality and diversification.

Modeling and structural optimization design of switched reluctance motor based on fusing attention mechanism and CNN-BiLSTM

Six-degrees of freedom (6-DOF) parallel mechanisms driven by switched reluctance motors (SRMs) can realize flexible control with high precision. Efficiency is an important indicator to measure the speed control system of SRMs. There are many characteristic factors affecting efficiency and strong nonlinear relationships between different characteristic parameters, which makes it difficult for analytical models and traditional neural network models to express their spatial correlation. For this reason, a convolutional neural network (CNN)-bidirectional long short-term memory network (BiLSTM) efficiency regression prediction model (CNN-BiLSTM-SENet) that integrates the attention mechanism (SENet) is proposed. Firstly, customize a formula for sensitivity analysis to screen characteristic parameters of efficiency. Secondly, build the CNN-BiLSTM-SENet model and use sparrow search algorithm (SSA) for hyperparameter optimization, input data to CNN to extract high-dimensional feature vectors that reflect complex changing relationships between features and efficiency while establishing feature channels. Embed the SENet to adaptively perceive and assign different weights to the feature channels, enhancing the influence of key features. Input the feature vectors outputted by the front-end network to BiLSTM to bidirectionally learn coupling relationships between sequences and complete regression prediction. Finally, propose improved northern goshawk optimization (MNGO) to solve the regression model to obtain the maximum efficiency and corresponding characteristic parameters. The results proved that the SSA-optimized CNN-BiLSTM-SENet model has higher fitting exactness and better prediction effect for efficiency regression prediction, and the MNGO also has stronger search ability and faster convergence.

Crowd Counting by Multi-Scale Dilated Convolution Networks

The number of people in a crowd is crucial information in public safety, intelligent monitoring, traffic management, architectural design, and other fields. At present, the counting accuracy in public spaces remains compromised by some unavoidable situations, such as the uneven distribution of a crowd and the difference in head scale caused by people’s differing distances from the camera. To solve these problems, we propose a deep learning crowd counting model, multi-scale dilated convolution networks (MSDCNet), based on crowd density map estimation. MSDCNet consists of three parts. The front-end network uses the truncated VGG16 to obtain preliminary features of the input image, with a proposed spatial pyramid pooling (SPP) module replacing the max-pooling layer to extract features with scale invariance. The core network is our proposed multi-scale feature extraction network (MFENet) for extracting features in three different scales. The back-end network consists of consecutive dilation convolution layers instead of traditional alternate convolution and pooling to expand the receptive field, extract high-level semantic information and avoid the spatial feature loss of small-scale heads. The experimental results on three public datasets show that the proposed model solved the above problems satisfactorily and obtained better counting accuracy than representative models in terms of mean absolute error (MAE) and mean square error (MSE).

3D Mesh classification and panoramic image segmentation using spherical vector networks with rotation-equivariant self-attention mechanism

Spherical signals exist in many applications such as planetary data, lidar scanning and digitization of 3D objects, so we need models that can effectively process spherical data. When the spherical data is simply projected onto a two-dimensional plane and then convolutional neural networks (CNNs) are used, the performance of the previous algorithms that exist in the literature is poor due to the distortion caused by the projection and the invalid translational equivariance. We propose a spherical vector network with rotation-equivariant self-attention mechanism for part-whole relationships learning to avoid a certain degree of distortion in this paper. Specifically, we take first the spherical convolutional network as the front-end network to obtain primary vectors, then we achieve the part-whole relationships between vectors through proposed rotation-equivariant self-attention mechanism to obtain advanced vectors which can represent the existence probability of the entity and orientations. Experimental results show that the proposed method combined with the front-end network improves the 3D mesh classification accuracy of the front-end network by 9% when the training set is not rotated and the test set is rotated arbitrarily under the rigid ModelNet40 dataset. Similarly, the 3D mesh classification accuracy of the front-end network improves by 12.2% under the non-rigid SHREC15 dataset. In addition, our method is compared with the recent method in the spherical image semantic segmentation task, achieving an improvement of 2.2% in mean pixel accuracy and 1.3% in mean intersection over union.

I2D-Loc: Camera localization via image to LiDAR depth flow

Accurate camera localization in existing LiDAR maps is promising since it potentially allows exploiting strengths of both LiDAR-based and camera-based methods. However, effective methods that robustly address appearance and modality differences for 2D–3D localization are still missing. To overcome these problems, we propose the I2D-Loc, a scene-agnostic and end-to-end trainable neural network that estimates the 6-DoF pose from an RGB image to an existing LiDAR map with local optimization on an initial pose. Specifically, we first project the LiDAR map to the image plane according to a rough initial pose and utilize a depth completion algorithm to generate a dense depth image. We further design a confidence map to weight the features extracted from the dense depth to get a more reliable depth representation. Then, we propose to utilize a neural network to estimate the correspondence flow between depth and RGB images. Finally, we utilize the BPnP algorithm to estimate the 6-DoF pose, calculating the gradients of pose error and optimizing the front-end network parameters. Moreover, by decoupling the intrinsic camera parameters out of the end-to-end training process, I2D-Loc can be generalized to the images with different intrinsic parameters. Experiments on KITTI, Argoverse, and Lyft5 datasets demonstrate that the I2D-Loc can achieve centimeter localization performance. The source code, dataset, trained models, and demo videos are released at https://levenberg.github.io/I2D-Loc/.

Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation

Language resources are the main factor in speech-emotion-recognition (SER)-based deep learning models. Thai is a low-resource language that has a smaller data size than high-resource languages such as German. This paper describes the framework of using a pretrained-model-based front-end and back-end network to adapt feature spaces from the speech recognition domain to the speech emotion classification domain. It consists of two parts: a speech recognition front-end network and a speech emotion recognition back-end network. For speech recognition, Wav2Vec2 is the state-of-the-art for high-resource languages, while XLSR is used for low-resource languages. Wav2Vec2 and XLSR have proposed generalized end-to-end learning for speech understanding based on the speech recognition domain as feature space representations from feature encoding. This is one reason why our front-end network was selected as Wav2Vec2 and XLSR for the pretrained model. The pre-trained Wav2Vec2 and XLSR are used for front-end networks and fine-tuned for specific languages using the Common Voice 7.0 dataset. Then, feature vectors of the front-end network are input for back-end networks; this includes convolution time reduction (CTR) and linear mean encoding transformation (LMET). Experiments using two different datasets show that our proposed framework can outperform the baselines in terms of unweighted and weighted accuracies.

Research on fish bait particles counting model based on improved MCNN

• A model of fish bait particle counting based on improved MCNN network is proposed. • The method based on density map is first introduced to the study of counting fish bait particles. • A network suitable for feature extraction of small targets is provided. • A model fused with transposed convolutional layer can significantly generate higher-quality density maps. In intensive aquaculture, quantitative research on bait particles has important theoretical and practical significance for reducing breeding costs and realizing intelligent feeding, and can effectively reduce environmental pollution caused by excessive bait. However, the current methods based on acoustics and machine vision are difficult to count tiny or overlapped bait particles, resulting in low counting accuracy of model. In order to solve the above problems, this paper proposes a counting model for small bait particles, using MCNN as the basic network to achieve efficient and stable counting performance in aquaculture. The proposed model consists of two parts: a multi-column feature extractor and a density map generator. The multi-column feature extractor is used as the front-end network of model. By introducing multiple branches, convolution kernels with different size are used to extract feature information of different scales and adapt to the non-uniform bait particles. Thereby, the multi-scale problem caused by perspective variation of bait particles is solved. In addition, the density map generator is used as the back-end network of model. Instead of the 1 × 1 convolutional layer in the original MCNN, the density map generator takes advantage of transposed convolutional layer to restore the poor quality of the density map caused by the down-sample in the multi-column feature extraction network. Besides, the ReLU function is helpful to increase the fitting ability of the model. Moreover, in order to further improve the quality of the density map and the counting performance of the model, the Euclidean distance loss and the structural similarity loss are combined as the final loss function of the model. The fusion of loss is beneficial to solve the ambiguity of density map caused by the Euclidean loss function. As the experimental results showed, compared with the original MCNN network, the mean absolute error MAE of the proposed model is reduced by 69.35%, and the mean square error MSE is reduced by 70.87%. In summary, the proposed bait particles counting model has high counting accuracy and is more stable, it is in favor of providing theoretical support for intelligent feeding study in aquaculture.

Information Entropy Augmented High Density Crowd Counting Network

The research proposes an innovated structure of the density map-based crowd counting network augmented by information entropy. The network comprises of a front-end network to extract features and a back-end network to generate density maps. In order to validate the assumption that the entropy can boost the accuracy of density map generation, a multi-scale entropy map extraction process is imported into the front-end network along with a fine-tuned convolutional feature extraction process, In the back-end network, extracted features are decoded into the density map with a multi-column dilated convolution network. Finally, the decoded density map can be mapped as the estimated counting number. Experimental results indicate that the devised network is capable of accurately estimating the count in extremely high crowd density. Compared to similar structured networks which don’t adapt entropy feature, the proposed network exhibits higher performance. This result proves the feature of information entropy is capable of enhancing the efficiency of density map-based crowd counting approaches.

Front-End Survivability in Wireless Optical Broadband Access Network

Background: Survivability is the one of the key issue of wireless optical broadband (WOBAN) access network. Survivability means to provide continuous services to users if a device/link failure occurs in the network Objective: The component link failure can occur in the network due to any reason. In this condition huge data loss will occur in the network. Methods: In this paper, we consider the front-end survivability in WOBAN. We propose a novel maximum protection minimum link cost routing algorithm (MPMLC) that provides path protection with minimum link cost to front-end of the WOBAN. The proposed MPMLC algorithm assigns weight/cost to wireless link using link state prediction (LSP). Results: The path with minimum cost is selected to route the traffic (actual/affected due to failure) of front end network. The proposed algorithm outperforms in terms of reduced network delay and wireless link cost Conclusion: It is also observed from simulation results that when the multiple link failures occur in the network, proposed MPMLC algorithm exhibit better results in terms of reduced wireless link cost as compared to existing algorithm.

Learning Pairwise Potential CRFs in Deep Siamese Network for Change Detection

Very high resolution (VHR) images change detection plays an important role in many remote sensing applications, such as military reconnaissance, urban planning and natural resource monitoring. Recently, fully connected conditional random field (FCCRF)-facilitated deep convolutional neural networks have shown promising results in change detection. However, the FCCRF in change detection currently is still postprocessing based on the output of the front-end network, which is not a convenient end-to-end network model and cannot combine front-end network knowledge with the knowledge of pairwise potential. Therefore, we propose a new end-to-end deep Siamese pairwise potential CRFs network (PPNet) for VHR images change detection. Specifically, this method adds a conditional random field recurrent neural network (CRF-RNN) unit into the convolutional neural network and integrates the knowledge of unary potential and pairwise potential in the end-to-end training process, aiming to refine the edges of changed areas and to remove the distant noise. In order to correct the front-end network identification errors, the method uses effective channel attention (ECA) to further effectively distinguish the change areas. Our experimental results on two data sets verify that the proposed method has more advanced capability with almost no increase in the number of parameters and effectively avoids the overfitting phenomenon in the training process.

Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading

The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.

A Face Recognition Algorithm Based on Intermediate Layers Connected by the CNN

Face recognition is difficult due to the higher dimension of face image features and fewer training samples. Firstly, in order to improve the performance of feature extraction, we inventively construct a double hierarchical network structure convolution neural network (CNN) model. The front-end network adopts a relatively simple network model to achieve rough feature extraction from input images and obtain multiple suspect face candidate windows. The back-end network uses a relatively complex network model to filter the best detection window and return the face size and position by nonmaximum suppression. Then, in order to fully extract the face features in the optimal window, a face recognition algorithm based on intermediate layers connected by the deep CNN is proposed in this paper. Based on AlexNet, the front, intermediate and end convolution layers are combined by deep connection. Then, the feature vector describing the face image is obtained by the operation of the pooling layer and the full connection layer. Finally, the auxiliary classifier training method is used to train the model to ensure the effectiveness of the features of the intermediate layer. Experimental results based on open face database show that the recognition accuracy of the proposed algorithm is higher than that of other face recognition algorithms compared in this paper.

Data-Taking Network for COMET Phase-I

An experiment to search for mu-e conversion named COMET is being constructed at J-PARC. The experiment will be carried out using a two-stage approach of Phase-I and Phase-II. The data taking system of Phase-I is developed based on common network technology. The data taking system consists of two kinds of networks. One is a front-end network. Its network bundles around twenty front-end devices that have a 1-Gb optical network port. And a front-end computer accepts data from the devices via its network. The other is a back-end network that collects all event fragments from the front-end computers using a 10-Gb network. We used a low price 1Gb/10Gb optical network switch for the front-end network. And direct connection between the front-end PC and an event building PC using 10-Gb optical network devices was used for the back-end network. The event building PC has ten 10-Gb network ports. And each network port of the event building PC is connected to the front-end PC's port without using a network switch. We evaluated data taking performance with an event building on these two kinds of networks. The event building throughput of the front-end network achieved 337 MiB/s. And the event building throughput of the back-end networks achieved 1.2 GiB/s. It means that we could reduce the construction cost of the data taking network using this structure without deteriorating performance. Moreover, we evaluated the writing speed of the local storage RAID disk system connected to a back-end PC by a SAS interface, and a long-distance network copy from the experiment location to the lasting storage.

Crowd counting method based on cross column fusion attention mechanism

Deep learning has made substantial progress in crowd density estimation, but there are still some problems in existing methods, such as large population density, background interference, and scale change, which makes it difficult to count people. To solve the above problems, we proposed a crowd counting method based on a cross column fusion attention mechanism. First, the first ten layers of VGG16 with good migration ability and feature extraction ability are used as the front-end network to preliminarily extract human head features. Then, a cross column fusion attention module is designed. In this module, feature maps are fused across columns to make the network contain richer deep and shallow features. At the same time, to alleviate the background interference, the attention mechanism is used to guide the network to focus on the head position in the picture, and different weights are assigned to different positions according to the attention score map, so as to highlight the crowd and weaken the background, and finally get a high-quality density map. In addition, a shallow convolution module is designed as another branch. The output feature map of the shallow convolution module and the output feature map of the attention module of cross column fusion are fused to solve the problem of scale change effectively. Finally, in the last layer of the network, the convolution layer of 1 × 1 is used to replace the full connection layer, and fewer network parameters are used to reduce the calculation and the population density map is regressed. The experimental results show that the mean absolute error and mean square error of the proposed algorithm are significantly reduced compared with the comparison algorithm.

Region Pixel Voting Network (RPVNet) for 6D Pose Estimation from Monocular Image

Recent studies have shown that deep learning achieves superior results in the task of estimating 6D-pose of target object from an image. End-to-end techniques use deep networks to predict pose directly from image, avoiding the limitations of handcraft features, but rely on training dataset to deal with occlusion. Two-stage algorithms alleviate this problem by finding keypoints in the image and then solving the Perspective-n-Point (PnP) problem to avoid directly fitting the transformation from image space to 6D-pose space. This paper proposes a novel two-stage method using only local features for pixel voting, called Region Pixel Voting Network (RPVNet). Front-end network detects target object and predicts its direction maps, from which the keypoints are recovered by pixel voting using Random Sample Consensus (RANSAC). The backbone, object detection network and mask prediction network of RPVNet are designed based on Mask R-CNN. Direction map is a vector field with the direction of each point pointing to its source keypoint. It is shown that predicting an object’s keypoints is related to its own pixels and independent of other pixels, which means the influence of occlusion decreases in the object’s region. Based on this phenomenon, in RPVNet, local features instead of the whole features, i.e., the output of the backbone, are used by a well-designed Convolutional Neural Networks (CNN) to compute direction maps. The local features are extracted from the whole features through RoIAlign, based on the region provided by detection network. Experiments on LINEMOD dataset show that RPVNet’s average accuracy (86.1%) is almost equal to state-of-the-art (86.4%) when no occlusion occurs. Meanwhile, results on Occlusion LINEMOD dataset show that RPVNet outperforms state-of-the-art (43.7% vs. 40.8%) and is more accurate for small object in occluded scenes.

Emotion Knowledge Driven Video Highlight Detection

This paper addresses video highlight detection which aims to select a small subset of frames according to user's major or special interest. The performances of conventional methods highly depend on large-scale manually labeled training data which are time-consuming and labor-intensive to collect. To deal with this problem, we trace back to the original problem definition and find that whether a user is interested in a specific video segment heavily depends on human's subjective emotions. Leveraging this insight, we introduce an emotion knowledge driven video detection framework for modeling human's general emotion and inferencing highlight strength. Firstly, we obtain the concept-level representation of the video clip with a front-end network. The concepts are used as nodes to build an emotion-related knowledge graph, and their relationships in the graph are modeled via external public knowledge graphs. Then we adopt Siamese GCNs to model the dependencies between nodes in the graph and propagate messages along the edges. Finally, we compute the emotion-aware representation of the video clip based on the GCN layers and further use it to predict the highlight score. Our framework, including the front-end network, graph convolution layers and the highlight mapping network, can be trained in an end-to-end manner with the constraint of a ranking loss. Experiments on two benchmark datasets show that our proposed method performs favorably against the state-of-the-art methods.

Crowd counting via an inverse attention residual network

Deep learning has enabled substantial progress in crowd counting, but existing methods still face difficulties due to significant scale variations, severe occlusion, and overcrowding. To explicitly address these problems, we propose a crowd counting method based on the inverse attention residual network. Initially, the first 10 layers of VGG-16 with a strong transfer learning ability and a feature extraction ability are used as the front-end network for preliminary extraction of head features. Afterward, we use the inverse attention residual module to divide the background information and crowd information and retain the accurate pixel position information to reduce the impact of various types of noise in the input image. Finally, the mini multiscale dilated convolution is used as the backbone of the back-end network to perform the aggregation of feature maps, so the network can learn the details of heads at different scales and use the 1 × 1 convolutional layer to regress the feature maps to the density map. As a result, the above problems are resolved to a certain extent. Experimental results show that the mean absolute error and the mean square error of the proposed algorithm are significantly lower than those of the comparison algorithms.

Virtio Front-End Network Driver for RTEMS Operating System

Virtualization technology can provide a transparent and efficient software development environment at the development phase of embedded systems. In addition, virtualization can be exploited to provide strong resource isolation between real-time applications. However, since virtualization incurs noticeable overheads, we need to minimize these to utilize virtualization technology for either a software development environment or a runtime environment of real-time embedded systems. In this letter, we particularly focus on virtualization overheads of network I/O and present an implementation of standardized (i.e., virtio) front-end network driver for a real-time operating system (RTOS). Although there have been several studies for virtio on general-purpose OS, our implementation is the first open-source virtio for an RTOS. The measurement results show that our implementation can improve the bandwidth and latency up to 168% and 52%, respectively. In addition, the memory usage can be saved up to 36%.

Fire detection with infrared images using cascaded neural network

The existing equipment of civil aircraft cargo fire detection mainly uses photoelectric smoke detectors, which has a high false alarm rate. According to Federal Aviation Agency’s (FAA) statistics, the false alarm rate is as high as 99%.1In the cargo of civil aircraft, the traditional photoelectric detection technology cannot effectively distinguish interference particles from smoke particles. Since the video smoke detection technology has proven to be reliable in many large scenarios, a deep learning method of image processing for fire detection is proposed. The proposed convolutional neural network is constructed of front end network and back end network cascaded with the capsule network and the circularity computation for the dynamic infrared fire image texture extraction. In order to accurately identify whether there is a fire in the area and give the kind of burning substances, a series of fuels are selected, such as n-heptane, cyclohexane, and carton for combustion reaction, and infrared camera is used to take infrared images of all fuel combustion. Experimental results show that the proposed method can effectively detect fire at the early stage of fire which is applicable for fire detection in civil aircraft cargoes.