Compact Descriptor Research Articles

A self-supervised deep Riemannian representation to classify parkinsonian fixational patterns

Parkinson’s disease (PD) is the second most prevalent neurodegenerative disorder, and it remains incurable. Currently there is no definitive biomarker for detecting PD, measuring its severity, or monitoring of treatments. Recently, oculomotor fixation abnormalities have emerged as a sensitive biomarker to discriminate Parkinsonian patterns from a control population, even at early stages. For oculomotor analysis, current experimental setups use invasive and restrictive capture protocols that limit the transfer in clinical routine. Alternatively, computational approaches to support the PD diagnosis are strictly based on supervised strategies, depending of large labeled data, and introducing an inherent expert-bias. This work proposes a self-supervised architecture based on Riemannian deep representation to learn oculomotor fixation patterns from compact descriptors. Firstly, deep convolutional features are recovered from oculomotor fixation video slices, and then encoded in compact symmetric positive matrices (SPD) to summarize second-order relationships. Each SPD input matrix is projected onto a Riemannian encoder until obtain a SPD embedding. Then, a Riemannian decoder reconstructs SPD matrices while preserving the geometrical manifold structure. The proposed architecture successfully recovers geometric patterns in the embeddings without any label diagnosis supervision, and demonstrates the capability to be discriminative regarding PD patterns. In a retrospective study involving 13 healthy adults and 13 patients diagnosed with PD, the proposed Riemannian representation achieved an average accuracy of 95.6% and an AUC of 99% during a binary classification task using a Support Vector Machine.

Extremely compact video representation for efficient near-duplicates detection

With the influx of online video uploads, a robust method for detecting video copies under various distortions is essential for copyright protection and search efficiency. We propose a compact video representation that uses a Siamese Neural Network for near-duplicate detection. This approach achieves up to 0.998 recall and 0.853 precision, even with distorted 56x56px miniatures. Consequently, we derive extremely compact video descriptors, facilitating video fragment retrieval in large datasets. Our method pre-selects frames from a video by identifying local maximums from the interframe differences curve, preserving characteristic sequence patterns even after extreme compression. Utilizing a simpler convolution-based model in the Siamese Neural Network, we improve results by up to 8% over VGG-16, with a 1.7-fold reduction in inference time. With compact descriptors of 1.875 kB, our models outperform others by more than two times. Additionally, we introduce a new dataset with dynamically changing scenes, enhancing its suitability for near-duplicate video detection.

RDLNet: A Regularized Descriptor Learning Network.

Local image descriptor learning has been instrumental in various computer vision tasks. Recent innovations lie with similarity measurement of descriptor vectors with metric learning for randomly selected Siamese or triplet patches. Local image descriptor learning focuses more on hard samples since easy samples do not contribute much to optimization. However, few studies focus on hard samples of image patches from the perspective of loss functions and design appropriate learning algorithms to obtain a more compact descriptor representation. This article proposes a regularized descriptor learning network (RDLNet) that makes the network focus on the learning of hard samples and compact descriptor with triplet networks. A novel hard sample mining strategy is designed to select the hardest negative samples in mini-batch. Then batch margin loss concerned with hard samples is adopted to optimize the distance of extreme cases. Finally, for a more stable network and preventing network collapsing, orthogonal regularization is designed to constrain convolutional kernels and obtain rich deep features. RDLNet provides a compact discriminative low-dimensional representation and can be embedded in other pipelines easily. This article gives extensive experimental results for large benchmarks in multiple scenarios and generalization in matching applications with significant improvements.

Structural Dynamics Descriptors for Metal Halide Perovskites.

Metal halide perovskites have shown extraordinary performance in solar energy conversion technologies. They have been classified as "soft semiconductors" due to their flexible corner-sharing octahedral networks and polymorphous nature. Understanding the local and average structures continues to be challenging for both modeling and experiments. Here, we report the quantitative analysis of structural dynamics in time and space from molecular dynamics simulations of perovskite crystals. The compact descriptors provided cover a wide variety of structural properties, including octahedral tilting and distortion, local lattice parameters, molecular orientations, as well as their spatial correlation. To validate our methods, we have trained a machine learning force field (MLFF) for methylammonium lead bromide (CH3NH3PbBr3) using an on-the-fly training approach with Gaussian process regression. The known stable phases are reproduced, and we find an additional symmetry-breaking effect in the cubic and tetragonal phases close to the phase-transition temperature. To test the implementation for large trajectories, we also apply it to 69,120 atom simulations for CsPbI3 based on an MLFF developed using the atomic cluster expansion formalism. The structural dynamics descriptors and Python toolkit are general to perovskites and readily transferable to more complex compositions.

Down syndrome identification and classification using facial features with neural network

The medical image tool named Ultrasound imaging is used for the analysis of fetus behaviour. Due to the presence of noise the quality of these images is low. Proper filtering is required to suppress this noise. Before the fatal development analysis, it is required image processing method. The processed images are then used for the analysis and it helps to take care of the health. Down syndrome is one of the crucial chromosomal disorders. It is due to the presence of a further copy of chromosome 21. The ultrasound imaging helps to point out DS in the earlier stage of pregnancy by handling the image in an efficient manner. In this work, nasal bone identification and texturing methods are used to detect the disease. Here we designed a nasal bone identification module for the image classification as normal or abnormal. Down syndrome detection utilizes a collection of facial expression images. A compact geometric descriptor is employed for extracting the facial features from the image set. There is no specific treatment for Down syndrome. Thus, early detection and screening of this disability are the best styles for Down syndrome prevention.

MFCC-based descriptor for bee queen presence detection

Monitoring and detecting hive health are essential for the development of ecosystem conservation strategies and sustainable beekeeping. The absence of the queen bee is a warning signal and indicates an anomalous situation. The literature contains promising machine learning and deep learning techniques for detecting the presence of the queen bee through the spectral temporal analysis of hive sounds. Despite works that consider handcrafted and deep features to describe the problem, there is a lack of feature selection to evaluate which descriptors could better discriminate the presence/absence of queen bee. Additionally, deep learning techniques lead to high-dimension descriptors and a high computational cost. In this context, we explore feature extraction and selection techniques to obtain efficient and compact descriptors that can perform classification in a real-time monitoring scenario. The results reveal that combining cepstral, time, and frequency features achieve 0.99 for accuracy, kappa, area under curve, specificity, and sensibility metrics, outperforming most state-of-the-art models for queen bee presence classification, including convolution neural network models. The feature selection step notably reduces the descriptor size and maintains classification performance. The 15 and 31 mel-frequency cepstrum coefficients descriptors have a low dimension and are based on only one feature, which reduces the number of calculation and library dependencies. Given this, we believe that our findings can support deployment in low-computational devices for non-intrusive and real-time hive health diagnoses.

Sequence-Level Reference Frames in Video Coding

The proliferation of low-cost DRAM chipsets now begins to allow for the consideration of substantially-increased decoded picture buffers in advanced video coding standards such as HEVC, VVC, and Google VP9. At the same time, the increasing demand for rapid scene changes and multiple scene repetitions in entertainment or broadcast content indicates that extending the frame referencing interval to tens of minutes or even the entire video sequence may offer coding gains, as long as one is able to identify frame similarity in a computationally- and memory-efficient manner. Motivated by these observations, we propose a “stitching” method that defines a reference buffer and a reference frame selection algorithm. Our proposal extends the referencing interval of inter-frame video coding to the entire length of video sequences. Our reference frame selection algorithm uses well-established feature descriptor methods that describe frame structural elements in a compact and semantically-rich manner. We propose to combine such compact descriptors with a similarity scoring mechanism in order to select the frames to be “stitched” to reference picture buffers of advanced inter-frame encoders like HEVC, VVC, and VP9 without breaking standard compliance. Our evaluation on synthetic and real-world video sequences with the HEVC and VVC reference encoders shows that our method offers significant rate gains, with complexity and memory requirements that remain manageable for practical encoders and decoders.

Exploring the Temporal Cues to Enhance Video Retrieval on Standardized CDVA

As the demand for large-scale video analysis increases, video retrieval research is also becoming more active. In 2014, ISO/IEC MPEG began standardizing compact descriptors for video analysis, known as CDVA, and it is now adopted as a standard. However, the standardized CDVA is not easily compared to other methods because the MPEG-CDVA dataset used for performance verification is not disclosed, despite the fact that follow-up studies are underway with multiple versions of the CDVA experimental model. In addition, analyses of modules constituting the CDVA framework are insufficient in previous studies. Therefore, we conduct self-evaluations of CDVA to analyze the impact of each module on the retrieval task. Furthermore, to overcome the obstacles identified through these self-evaluations, we suggest temporal nested invariance pooling, abbreviated as TNIP, which implies temporal robustness realized by improving nested invariance pooling, abbreviated as NIP, one of the features in CDVA. Finally, benchmarks of the existing CDVA and the proposed approach are provided on several public datasets. Through this, we show that the CDVA framework is capable of boosting the retrieval performance if utilizing the proposed approach.

Learning General and Distinctive 3D Local Deep Descriptors for Point Cloud Registration.

An effective 3D descriptor should be invariant to different geometric transformations, such as scale and rotation, robust to occlusions and clutter, and capable of generalising to different application domains. We present a simple yet effective method to learn general and distinctive 3D local descriptors that can be used to register point clouds that are captured in different domains. Point cloud patches are extracted, canonicalised with respect to their local reference frame, and encoded into scale and rotation-invariant compact descriptors by a deep neural network that is invariant to permutations of the input points. This design is what enables our descriptors to generalise across domains. We evaluate and compare our descriptors with alternative handcrafted and deep learning-based descriptors on several indoor and outdoor datasets that are reconstructed by using both RGBD sensors and laser scanners. Our descriptors outperform most recent descriptors by a large margin in terms of generalisation, and also become the state of the art in benchmarks where training and testing are performed in the same domain.

MHFP: Multi-view based hierarchical fusion pooling method for 3D shape recognition

• We propose a novel MHFP architecture, which can attain better multi-view features. • We design a 3D attention module for multi-view to construct the graph in our MHFP. • The results on ModelNet40 show the superiority of our multi-view based method. 3D shape recognition has received widespread attention in the field of computer vision. Since the 3D model contains much geometric information, which is difficult to extract but important to feature learning, effective description of 3D shape is still facing great challenges. With the rapid development of deep learning, a large number of methods have been proposed. However, these approaches always focus on the learning of view features but ignore the multi-view information protection in the process of feature fusion. In this work, we propose a novel Multi-view based Hierarchical Fusion Pooling Method (MHFP) for 3D Model Recognition, which hierarchically fuses the features of multi-view into a compact descriptor. Our approach considers the correlation between views, it can powerfully remove redundant information and retain a large amount of essential information. Meanwhile, we design a 3D attention module to dig out the correlation between the views, which prepares for the graph construction. To verify the effectiveness of our MHFP, we conduct experiments on the ModelNet40 dataset and compare it with some state-of-the-art methods. The final results demonstrate the superiority of our proposed approach in 3D shape recognition.

A novel fusion approach in the extraction of kernel descriptor with improved effectiveness and efficiency

Image representation using feature descriptors is crucial. A number of histogram-based descriptors are widely used for this purpose. However, histogram-based descriptors have certain limitations and kernel descriptors (KDES) are proven to overcome them. Moreover, the combination of more than one KDES performs better than an individual KDES. Conventionally, KDES fusion is performed by concatenating them after the gradient, colour and shape descriptors have been extracted. This approach has limitations in regard to the efficiency as well as the effectiveness. In this paper, we propose a novel approach to fuse different image features before the descriptor extraction, resulting in a compact descriptor which is efficient and effective. In addition, we have investigated the effect on the proposed descriptor when texture-based features are fused along with the conventionally used features. Our proposed descriptor is examined on two publicly available image databases and shown to provide outstanding performances.

Deep Learning Driven Venue Recommender for Event-Based Social Networks

Event-based online social platforms, such as Meetup and Plancast, have experienced increased popularity and rapid growth in recent years. In EBSN setup, selecting suitable venues for hosting events, which can attract a great turnout, is a key challenge. In this paper, we present a deep learning based venue recommendation system $DeepVenue$ D e e p V e n u e which provides context driven venue recommendations for the Meetup event-hosts to host their events. The crux of the proposed model relies on the notion of similarity between multiple Meetup entities such as events, venues, groups, etc. We develop deep learning techniques to compute a compact descriptor for each entity, such that two entities (say, venues) can be compared numerically. Notably, to mitigate the scarcity of venue related information in Meetup, we leverage on the cross domain knowledge transfer from popular LBSN service Yelp to extract rich venue related content. For hosting an event, the proposed $DeepVenue$ D e e p V e n u e model computes a success score for each candidate venue and ranks those venues according to the scores and finally recommend the top k venues. Our rigorous evaluation on the Meetup data collected for the city of Chicago shows that $DeepVenue$ D e e p V e n u e significantly outperforms the baselines algorithms. Precisely, for 84 percent of events, the correct hosting venue appears in the top 5 of the $DeepVenue$ D e e p V e n u e recommended list.

QuadrupletNet: A Novel Local Descriptor for Brain Tumor Detection and Segmentation

Brain tumor detection and segmentation from Magnetic Resonance Imaging (MRI) images is being one of the emerging fields in the biomedicine. A formidable undertaking in brain tumor surgery, medical care, treatment programme and quantitative assessment of MRI images is to precisely diagnose its location and extent. Recently, the convolutional neural network (CNN) based detection and segmentation method on brain tumor MRI images is being one of the emerging fields in the medical imaging as an automatic clinic treatment and evaluation solution. In this article, we put forward a brand new quadruplet loss in CNN framework, which achieves higher accuracy in brain tumor detection and segmentation than other pairwise loss and triplet loss methods. By applying the proposed quadruplet loss to the original L2Net CNN architecture leads to a more compact descriptor named QuadrupletNet. From our experiments, QuadrupletNet shows higher performance than other state-of-the-art loss functions e.g., the Triplet loss, as indicated in experiments on Multimodal Brain Tumor Image Segmentation (BRATS 2018) datasets, and on our own collected MRI brain tumor datasets (named MBTD).

Category-preserving binary feature learning and binary codebook learning for finger vein recognition

Local binary feature learning has attracted a lot of researches in image recognition due to its vital effectiveness. Generally, in the traditional local feature learning methods, a projection is learned to map the patches of image into binary features and then a codebook is generated by clustering the binary features with K-means clustering. However, these local feature learning methods, such as compact binary face descriptor and discriminative binary descriptor, ignore the category specific distributions of the original features during the feature learning process and use the real-valued clustering approach to generate the codebook, the discriminant of feature is degraded and the merits of binary feature are lost. To tack these problems, in this paper, we propose a novel category-preserving binary feature learning and binary codebook leaning (CPBFL-BCL) method for finger vein recognition. In CPBFL-BCL, the discrimination of learned binary features is generated by criteria of fisher discriminant analysis and category manifold preserving regularity during the feature learning process, and a novel binary clustering method based on K-means clustering is designed to generate binary codebook. Experimental results on recognition and retrieval tasks using two public finger vein databases are presented and demonstrate the effectiveness and efficiency of the proposed method over the state-of-the-art finger vein methods and a finger vein retrieval method.

Fast and Consistent Matching for Landmark-Based Place Recognition

Severe condition changes make visual place recognition a difficult task for robotics applications. Landmark-based methods that combine local image regions and features from a convolution neural network (CNN) have shown state-of-the-art performance. This paper focuses on improving the performance of landmark-based methods for place recognition. To save time in landmark matching, a novel method is presented to extract landmarks based directly on the lower layer of CNN features instead of using object proposals or superpixels. The centers of landmarks are projected onto the feature maps of the higher layer of the CNN to generate compact descriptors. We then extract potential images through similarities computed by direct matching. In the refinement stage, we exploit the Markov random field (MRF), combined with Bayes filters, to perform robust matchings. This stage integrates the spatial information of landmarks and introduces a new method for similarity measurements. We present extensive experiments using challenging datasets and compare the results to state-of-the-art methods to evaluate our approach. The results show that our approach ensures a remarkable performance under severe condition changes and significantly improves efficiency.

Recognition of 3D Shapes Based on 3V-DepthPano CNN

This paper proposes a convolutional neural network (CNN) with three branches based on the three-view drawing principle and depth panorama for 3D shape recognition. The three-view drawing principle provides three key views of a 3D shape. A depth panorama contains the complete 2.5D information of each view. 3V-DepthPano CNN is a CNN system with three branches designed for depth panoramas generated from the three key views. This recognition system, i.e., 3V-DepthPano CNN, applies a three-branch convolutional neural network to aggregate the 3D shape depth panorama information into a more compact 3D shape descriptor to implement the classification of 3D shapes. Furthermore, we adopt a fine-tuning technique on 3V-DepthPano CNN and extract shape features to facilitate the retrieval of 3D shapes. The proposed method implements a good tradeoff state between higher accuracy and training time. Experiments show that the proposed 3V-DepthPano CNN with 3 views obtains approximate accuracy to MVCNN with 12/80 views. But the 3V-DepthPano CNN frame takes much shorter time to obtain depth panoramas and train the network than MVCNN. It is superior to all other existing advanced methods for both classification and shape retrieval.

Video Alignment Using Bi-Directional Attention Flow in a Multi-Stage Learning Model

Recently, deep learning techniques have contributed to solving a multitude of computer vision tasks. In this paper, we propose a deep-learning approach for video alignment, which involves finding the best correspondences between two overlapping videos. We formulate the video alignment task as a variant of the well-known machine comprehension (MC) task in natural language processing. While MC answers a question about a given paragraph, our technique determines the most relevant frame sequence in the context video to the query video. This is done by representing the individual frames of the two videos by highly discriminative and compact descriptors. Next, the descriptors are fed into a multi-stage network that is able, with the help of the bidirectional attention flow mechanism, to represent the context video at various granularity levels besides estimating the query-aware context part. The proposed model was trained on 10k video-pairs collected from “YouTube”. The obtained results show that our model outperforms all known state of the art techniques by a considerable margin, confirming its efficacy.

LPD-AE: Latent Space Representation of Large-Scale 3D Point Cloud

The effective latent space representation of point cloud provides a foremost and fundamental manner that can be used for challenging tasks, including point cloud based place recognition and reconstruction, especially in large-scale dynamic environments. In this paper, we present a novel deep neural network, LPD-AE(Large-scale Place Description AutoEncoder Network), to obtain meaningful local and contextual features for the generation of latent space from 3D point cloud directly. The encoder network constructs the discriminative global descriptors to realize high accuracy and robust place recognition, which contributed by extracting the local neighbor geometric features and aggregating neighborhood relationships both in feature space and physical space. The decoder network performs hierarchical reconstruction on coarse key points and ultimately produce dense point clouds, which shows that it is capable of reconstructing a full point cloud frame from a single compact but high dimensional descriptor. Our proposed network demonstrates performance that is comparable to the state-of-the-art approaches. With the benefit of the LPD-AE, many computationally complex tasks that rely directly on point clouds can be effortlessly conducted on latent space with lower memory costs, such as relocalization, loop closure detection, and map compression reconstruction. Comprehensive validations on Oxford RobotCar dataset, KITTI dataset, and our freshly collected dataset, which contains multiple trials of repeated routes in different weather and at different times, manifest its potency for real robotic and self-driving implementation. The source code is available at https://github.com/Suoivy/LPD-AE .

Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics.

Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale. That is, one is with compactness and efficiency to serve for machine vision, and the other is with full fidelity, bowing to human perception. The recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, i.e. Compact Descriptors for Visual Search and Compact Descriptors for Video Analysis, promote the sustainable and fast development in their own directions, respectively. In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG standardization efforts1. Towards collaborative compression and intelligent analytics, VCM attempts to bridge the gap between feature coding for machine vision and video coding for human vision. Aligning with the rising Analyze then Compress instance Digital Retina, the definition, formulation, and paradigm of VCM are given first. Meanwhile, we systematically review state-of-the-art techniques in video compression and feature compression from the unique perspective of MPEG standardization, which provides the academic and industrial evidence to realize the collaborative compression of video and feature streams in a broad range of AI applications. Finally, we come up with potential VCM solutions, and the preliminary results have demonstrated the performance and efficiency gains. Further direction is discussed as well.

A Deep Quadruplet Network for Local Descriptor Learning

Generating a distinguishable feature for local patch is a main task in computer vision which aims at matching local patches. Recently, local patch descriptors from deep convolutional neural network (CNN) with a triplet loss have achieved promising performance. In this paper, we design a quadruplet loss, which can achieve a better result than other pairwise loss and triplet loss methods. Our loss is inspired by the thoughts of uniform distribution. It separates non-matching examples by using the hard sampled non-matching pairs in a batch, and simultaneously uses the random sampled non-matching examples to keep non-matching pairs to obey uniform distribution. A compact descriptor named QuadrupletNet is generated by combining the proposed quadruplet loss and L2Net CNN architecture. From our experiment, QuadrupletNet shows better performance on the Brown dataset and Hpatches dataset than Triplet loss methods on the same training set. The pre-trained QuadrupletNet is publicly available.