Final Density Map Research Articles

A novel framework for crowd counting using video and audio

Crowd counting is significant in many areas. The existing methods have poor accuracy for perspective scenes and low illumination scenes. Besides, the existing audio-assisted methods only use local audio, which fails to provide the spatial feature information of sound in all directions in space. To alleviate the above problems, a novel framework named Video and Audio-assisted Crowd Counting Network (VACCNet) is proposed. The framework consists of two submodules: Video Crowd Counting (VCC) module and Audio-assisted Crowd Counting (ACC) module. The visual features from the VCC module and the fused audio features from the ACC module are further combined to get the final density map. To prove the effects of VACCNet, a new self-collected dataset named multiPle dIrection Assistance couNting netwOrk (PIANO) is built. The experimental results based on existing benchmarks and PIANO show that the proposed method has a 14.23% improvement averagely to the conventional methods.

MHAMD-MST-CNN: multiscale head attention guided multiscale density maps fusion for video crowd counting via multi-attention spatial-temporal CNN

ABSTRACT Video-based crowd counting and density estimation (CCDE) is vital for crowd monitoring. The existing solutions lack in addressing issues like cluttered background and scale variation in crowd videos. To this end, a multiscale head attention-guided multiscale density maps fusion for video-based CCDE via multi-attention Spatial-Temporal CNN (MHAMD-MST-CNN) is proposed. The MHAMD-MST-CNN has three modules: a multi attention spatial stream (MASS), a multi attention temporal stream (MATS), and a final density map generation (FDMG) module. The spatial head attention modules (SHAMs) and temporal head attention modules (THAMs) are designed to eliminate the background influence from the MASS and the MATS, respectively, by mapping the multiscale spatial or temporal features to head maps. The multiscale de-backgrounded features are utilised by the density map generation (DMG) modules to generate multiscale density maps to deal with scale variation due to perspective distortion. The multiscale density maps are fused and fed into the FDMG module to obtain the final crowd density map. The MHAMD-MST-CNN has been trained and validated on three publicly available benchmark datasets: the Venice, the Mall, and the UCSD. The MHAMD-MST-CNN provides competitive results as compared with the state-of-the-arts in terms of mean absolute error (MAE) and root mean squared error (RMSE).

Object Counting Using a Refinement Network

To address the scale variance and uneven distribution of objects in scenarios of object-counting tasks, an algorithm called Refinement Network (RefNet) is exploited. The proposed top-down scheme sequentially aggregates multiscale features, which are laterally connected with low-level information. Trained by a multiresolution density regression loss, a set of intermediate-density maps are estimated on each scale in a multiscale feature pyramid, and the detailed information of the density map is gradually added through coarse-to-fine granular refinement progress to predict the final density map. We evaluate our RefNet on three crowd-counting benchmark datasets, namely, ShanghaiTech, UCF_CC_50, and UCSD, and our method achieves competitive performances on the mean absolute error and root mean squared error compared to the state-of-the-art approaches. We further extend our RefNet to cell counting, illustrating its effectiveness on relative counting tasks.

Multi-scale features fused network with multi-level supervised path for crowd counting

Many CNN-based methods which utilize the density map to regress the count number of crowd are introduced to solve the crowd counting problem lately. Due to the head scale variations caused by the perspective change and background noise, these methods cannot address these two problems well in highly crowded scenario. In order to solve these two problems, we introduce a multi-scale features fused network with multi-level supervised path to produce the high-quality density map in this paper. Our model utilizes the first 13 layers of VGG16 model as the backbone, the multi-level supervised path in our model employs the multi-level dilated convolution module (MLD) to supervise the whole network at multi-level, and generate the attention map for the density map, which is used to handle the scale variations. The other path is used to fuse multi-scale features to generate the density map with soft spatial-channel attention module (SSCA) which aims to produce a saliency weight map of same size. In the end, the final density map is captured by the feature map multiply the attention map. In addition, a new objective function is proposed to train our network. A large number of experimental results show that compared with other networks, our method achieves better experimental results on four challenging datasets (UCF_CC_50, ShanghaiTech, UCF-QRNF and WorldExpo’10 dataset).

Scale and Background Aware Asymmetric Bilateral Network for Unconstrained Image Crowd Counting

This paper attacks the two challenging problems of image-based crowd counting, that is, scale variation and complex background. To that end, we present a novel crowd counting method, called the Scale and Background aware Asymmetric Bilateral Network (SBAB-Net), which is able to handle scale variation and background noise in a unified framework. Specifically, the proposed SBAB-Net contains three main components, a pre-trained backbone convolutional neural network (CNN) as the feature extractor and two asymmetric branches to generate a density map. These two asymmetric branches have different structures and use features from different semantic layers. One branch is densely connected stacked dilated convolution (DCSDC) sub-network with different dilation rates, which relies on one deep feature layer and can handle scale variation. The other branch is parameter-free densely connected stacked pooling (DCSP) sub-network with various pooling kernels and strides, which relies on shallow feature and can fuse features with several receptive fields to reduce the impact of background noise. Two sub-networks are fused by the attention mechanism to generate the final density map. Extensive experimental results on three widely-used benchmark datasets have demonstrated the effectiveness and superiority of our proposed method: (1) We achieve competitive counting performance compared to state-of-the-art methods; (2) Compared with baseline, the MAE and MSE are decreased by at least 6.3% and 11.3%, respectively.

Attentive multi-stage convolutional neural network for crowd counting

Crowd counting is an important problem in computer vision, whose application can be found in a wide range of tasks. Although this problem has been well studied, how to effectively deal with scale variations and perspective distortions is still a challenge. High-quality crowd density map depends heavily on how well these problems are solved. In this paper, we propose a novel network architecture called Attentive Multi-stage CNN for Crowd Counting (AMCNN). The AMCNN contains two subnetworks, i.e., hierarchical density estimator(HDE) and auxiliary count classifier (AUCC). The HDE adopts a hierarchical strategy to mine semantic features in a coarse-to-fine manner to tackle the problem of scale changes and perspective distortions. And the obtained composite features are used to generate the final density map. In addition, to further improve the density map quality, a soft attention mechanism is integrated into the AMCNN to distinct the foreground and the background. Furthermore, the AUCC is employed to achieve the count classification task, which is complementary to the task of density estimation. We evaluate our model on three public datasets: ShanghaiTech, UCF_CC_50 and Mall. Extensive experiments demonstrate that our counting model is on par with some state-of-the-art methods. Source code will be released at:https://github.com/wxq-ahu/crowd-count-amcnn.

Arbitrary perspective crowd counting via local to global algorithm

Crowd counting is getting more and more attention. More and more collective activities, such as the Olympics Games and the World Expo, are also important to control the crowd number. In this paper, we address the problem of crowd counting in the crowded scene. Our model accurately estimated the count of people in the crowded scene. Firstly, we proposed a novel and simple convolutional neural network, called Global Counting CNN (GCCNN). The GCCNN can learn a mapping, transforms the appearance of image patches to estimated density maps. Secondly, the Local to Global counting CNN (LGCCNN), calculating the density map from local to global. Stiching the local patches constrains the final density map of the larger area, which makes up for the difference values in the perspective map. In general, it makes the final density map more accurate. The dataset we used is a set of public dataset, which are WorldExpo’10 dataset, Shanghaitech dataset, the UCF_CC_50 dataset and the UCSD dataset. The experiments have proved our method achieves the state-of-the-art result over other algorithms.

Multi-density map fusion network for crowd counting

In crowed scene, its hard to get the exact number of people due to the distorted perspectives, complex backgrounds, and scale changes. People in different locations have different sizes and dimensions in an image. To deal with this problem, we propose a new multi-density map fusion method to learn the mapping from the input image to the density map. Different form previous methods, our method mainly focuses on fusing different density maps information instead of fusing multi-scale feature of the same images. The major contributions are three paralleled branches and dynamic weighting strategy. First, our network employs the first ten layers of VGG16, and the network is combined with three paralleled branches. Each branch of our network extracts image information at different scales and each branch outputs a density map. Second, to ensure the quality of the final density map, we employ learnable relative weights to fuse the three density maps. Our method has been proved more robust than many state-of-art methods. Lots of experiments have been done in the ShanghaiTech, WorldExpo10, UCSD and UCF_CC_50 dataset to show the effectiveness of our proposed method.

Crowd counting with crowd attention convolutional neural network

Crowd counting is a challenging problem due to the scene complexity and scale variation. Although deep learning has achieved great improvement in crowd counting, scene complexity affects the judgement of these methods and they usually regard some objects as people mistakenly; causing potentially enormous errors in the crowd counting result. To address the problem, we propose a novel end-to-end model called Crowd Attention Convolutional Neural Network (CAT-CNN). Our CAT-CNN can adaptively assess the importance of a human head at each pixel location by automatically encoding a confidence map. With the guidance of the confidence map, the position of human head in estimated density map gets more attention to encode the final density map, which can avoid enormous misjudgements effectively. The crowd count can be obtained by integrating the final density map. To encode a highly refined density map, the total crowd count of each image is classified in a designed classification task and we first explicitly map the prior of the population-level category to feature maps. To verify the efficiency of our proposed method, extensive experiments are conducted on three highly challenging datasets. Results establish the superiority of our method over many state-of-the-art methods.

DA-Net: Learning the Fine-Grained Density Distribution With Deformation Aggregation Network

The major challenges for accurate crowd counting stem from the large variations in the scale, shape, and perspective. In fact, dealing with such difficulties depends on the geometric transformation capabilities of the network. Thus, we propose the deformation aggregation network (DA-Net) that can incrementally incorporate adaptive receptive fields to capture the fine-grained density distribution. Our model starts from a fundamental diamond network based on the successive steps of rising and falling on the channels, which utilizes small-fixed kernels accounting for preserving spatial information. For sake of accommodating to scale variations, we further apply deformable convolutions to deliver precise localization by augmenting the spatial sampling locations with additional offsets, capable of transformation-invariant speciality on diverse scenes. In addition, we assign adjustable weights to supervise network to learn the fusion proportion adaptively and aggregate multiple levels of abstractions to get the final density map. DA-Net is a fully convolutional network that is able to take input of arbitrary size. The proposed method delivers the state-of-the-art performance on four benchmarks (ShanghaiTech, WorldExpo’10, UCSD, and UCF_CC_50). To further verify the generality of our model, we conduct an extended experiment on vehicle data set TRANCOS and achieve significant results.

An atomic model of brome mosaic virus using direct electron detection and real-space optimization.

Advances in electron cryo-microscopy have enabled structure determination of macromolecules at near-atomic resolution. However, structure determination, even using de novo methods, remains susceptible to model bias and overfitting. Here we describe a complete workflow for data acquisition, image processing, all-atom modelling and validation of brome mosaic virus, an RNA virus. Data were collected with a direct electron detector in integrating mode and an exposure beyond the traditional radiation damage limit. The final density map has a resolution of 3.8 Å as assessed by two independent data sets and maps. We used the map to derive an all-atom model with a newly implemented real-space optimization protocol. The validity of the model was verified by its match with the density map and a previous model from X-ray crystallography, as well as the internal consistency of models from independent maps. This study demonstrates a practical approach to obtain a rigorously validated atomic resolution electron cryo-microscopy structure.

Structural Convergence between Cryo-EM and NMR Reveals Intersubunit Interactions Critical for HIV-1 Capsid Function

Mature HIV-1 particles contain conical-shaped capsids that enclose the viral RNA genome and perform essential functions in the virus life cycle. Previous structural analysis of two- and three-dimensional arrays of the capsid protein (CA) hexamer revealed three interfaces. Here, we present a cryoEM study of a tubular assembly of CA and a high-resolution NMR structure of the CA C-terminal domain (CTD) dimer. In the solution dimer structure, the monomers exhibit different relative orientations compared to previous X-ray structures. The solution structure fits well into the EM density map, suggesting that the dimer interface is retained in the assembled CA. We also identified a CTD-CTD interface at the local three-fold axis in the cryoEM map and confirmed its functional importance by mutagenesis. In the tubular assembly, CA intermolecular interfaces vary slightly, accommodating the asymmetry present in tubes. This provides the necessary plasticity to allow for controlled virus capsid dis/assembly.

Reconstruction of the P2X2 Receptor Reveals a Vase-Shaped Structure with Lateral Tunnels above the Membrane

In response to the intercellular messenger ATP, P2X receptors transfer various sensory information, including pain. Here we have reconstructed the structure of the P2X(2) receptor at 15 A resolution from more than 90,000 particle images, taken with a cryo-electron microscope equipped with a helium-cooled stage. This three-dimensional depiction, presumably in a closed state, revealed an elongated vase-shaped structure 202 A in height and 160 A in major diameter. The extracellular and transmembrane domains present a two-layered structure, in which a sparse outer layer surrounds a pore-forming inner density. The decreased diameter of a putative ion-conducting pathway at the middle of the membrane was considered to be the narrowest part of the pore, which has been predicted from electrophysiological studies. The sparse, extended structure of the P2X(2) receptor indicates a loose assembly of subunits, which could be a basis for the activation-dependent pore dilation of P2X receptors.

Classification and three-dimensional reconstruction of unevenly distributed or symmetry mismatched features of icosahedral particles

Methods for the three-dimensional reconstruction of icosahedral particles, such as spherical viruses, from electron micrographs are well established. These methods take advantage of the 60-fold symmetry of the icosahedral group. Several features within these particles, however, may deviate from icosahedral symmetry. Examples include viral genomes, symmetry mismatched vertex proteins, unique DNA packaging vertices, flexible proteins, and proteins that are present at less than 100% occupancy. Such asymmetrically distributed features are smeared in the final density map when icosahedral symmetry is applied. Here, we describe a novel approach to classifying, analysing, and obtaining three-dimensional reconstructions of such features. The approach uses the orientation information derived from the icosahedral orientation search to facilitate multivariate statistical analysis and to limit the orientational degrees of freedom for reconstruction. We demonstrate the application of this approach to images of Kelp fly Virus. In this case, each virion may have two different types of fivefold vertex. We use our approach to produce independent reconstructions of the two types of vertex.