Hand-crafted Descriptors Research Articles

Just Dance: detection of human body reenactment fake videos

In the last few years, research on the detection of AI-generated videos has focused exclusively on detecting facial manipulations known as deepfakes. Much less attention has been paid to the detection of artificial non-facial fake videos. In this paper, we address a new forensic task, namely, the detection of fake videos of human body reenactment. To this purpose, we consider videos generated by the “Everybody Dance Now” framework. To accomplish our task, we have constructed and released a novel dataset of fake videos of this kind, referred to as FakeDance dataset. Additionally, we propose two forgery detectors to study the detectability of FakeDance kind of videos. The first one exploits spatial–temporal clues of a given video by means of hand-crafted descriptors, whereas the second detector is an end-to-end detector based on Convolutional Neural Networks (CNNs) trained on purpose. Both detectors have their peculiarities and strengths, working well in different operative scenarios. We believe that our proposed dataset together with the two detectors will contribute to the research on the detection of non-facial fake videos generated by means of AI.

Complementary pseudo multimodal feature for point cloud anomaly detection

Point cloud anomaly detection is steadily emerging as a promising research area. Recognizing the importance of feature descriptiveness in this task, this study introduces the Complementary Pseudo Multimodal Feature (CPMF), which combines local geometrical information extracted by 3D handcrafted descriptors with global semantic information extracted from 2D pre-trained neural networks. Specifically, to leverage 2D pre-trained neural networks for point-wise feature extraction, this study projects original point clouds into multi-view images. These images are then fed into a pre-trained 2D neural network for informative 2D modality feature extraction. Following the 2D–3D correspondence, the multi-view 2D modality features are projected back to 3D space and aggregated to obtain point-wise 2D modality features. Finally, the point-wise 3D and 2D modality features are fused to derive the CPMF for point cloud anomaly detection. Extensive experiments conducted on MVTec 3D and Real3D datasets demonstrate the complementary capacity between 2D and 3D modality features and the effectiveness of CPMF. Notably, CPMF achieves a significantly higher object-level AUROC of 95.15% compared to other methods on the MVTec 3D benchmark. Code is available at https://github.com/caoyunkang/CPMF.

Atoms as words: A novel approach to deciphering material properties using NLP-inspired machine learning on crystallographic information files (CIFs)

In condensed matter physics and materials science, predicting material properties necessitates understanding intricate many-body interactions. Conventional methods such as density functional theory and molecular dynamics often resort to simplifying approximations and are computationally expensive. Meanwhile, recent machine learning methods use handcrafted descriptors for material representation, which sometimes neglect vital crystallographic information and are often limited to single property prediction or a sub-class of crystal structures. In this study, we pioneer an unsupervised strategy, drawing inspiration from natural language processing to harness the underutilized potential of Crystallographic Information Files (CIFs). We conceptualize atoms and atomic positions within a crystallographic information file similarly to words in textual content. Using a Word2Vec-inspired technique, we produce atomic embeddings that capture intricate atomic relationships. Our model, CIFSemantics, trained on the extensive material project dataset, adeptly predicts 15 distinct material properties from the CIFs. Its performance rivals that of specialized models, marking a significant step forward in material property predictions.

Classification of Multiple H&E Images via an Ensemble Computational Scheme.

In this work, a computational scheme is proposed to identify the main combinations of handcrafted descriptors and deep-learned features capable of classifying histological images stained with hematoxylin and eosin. The handcrafted descriptors were those representatives of multiscale and multidimensional fractal techniques (fractal dimension, lacunarity and percolation) applied to quantify the histological images with the corresponding representations via explainable artificial intelligence (xAI) approaches. The deep-learned features were obtained from different convolutional neural networks (DenseNet-121, EfficientNet-b2, Inception-V3, ResNet-50 and VGG-19). The descriptors were investigated through different associations. The most relevant combinations, defined through a ranking algorithm, were analyzed via a heterogeneous ensemble of classifiers with the support vector machine, naive Bayes, random forest and K-nearest neighbors algorithms. The proposed scheme was applied to histological samples representative of breast cancer, colorectal cancer, oral dysplasia and liver tissue. The best results were accuracy rates of 94.83% to 100%, with the identification of pattern ensembles for classifying multiple histological images. The computational scheme indicated solutions exploring a reduced number of features (a maximum of 25 descriptors) and with better performance values than those observed in the literature. The presented information in this study is useful to complement and improve the development of computer-aided diagnosis focused on histological images.

A deep learning-based local feature extraction method for improved image matching and surface reconstruction from Yutu-2 PCAM images on the Moon

Topographic reconstruction of the lunar surface or other planets is important to engineering applications and scientific research in a planetary exploration mission. The typical methods of terrain reconstruction are usually based on photogrammetry techniques. Structure-from-Motion (SfM) is one of the most effective and commonly used photogrammetric technologies that estimate three-dimensional structures from two-dimensional image sequences. To find correspondences and align photos, SfM approaches require invariant features, such as Scale Invariant Feature Transform (SIFT). The hand-crafted features, however, seriously degrade performance due to weak texture and low light conditions causing image matching failure, which will directly affect the accuracy and robustness of reconstruction. Robust local descriptors based on deep learning outperform hand-crafted descriptors since convolutional neural networks are more robust than hand-engineered representations. Therefore, a novel and robust deep learning-based local feature extraction method is proposed, comprising two branch networks integrated with attention mechanisms for generating reliable keypoints and descriptors, respectively. Furthermore, a 3D terrain surface reconstruction workflow is constructed by combining it with the modern advanced image matching method and SfM system. The effectiveness of the proposed method and the workflow were verified in experiments using Panoramic Camera (PCAM) images acquired from three waypoints explored by the Yutu-2 lunar rover during the Chang’e-4 mission. We also illustrate how our approach supports other applications, such as creating panoramic mosaics of surface imagery. This provides a new and powerful method for planetary terrain reconstruction at a high spatial resolution that can meet the requirements for rover navigation and positioning, as well as geological analysis of the Moon and other planets. The source codes developed in this study are openly available at https://github.com/Atypical-Programmer/Deep_Reconstruction_Workflow.

High-order knowledge-based Discriminant features for kinship verification

This research work aims to propose an effective and robust face kinship verification system by leveraging several axes, including advanced learning techniques, deep learning, and CNN networks. The contributions of this work include the use of a preprocessing method known as Multiscale Retinex with Chromaticity Preservation (MSRCP) and the Gradientfaces technique (GRF) to improve image quality and contrast enhancement. Additionally, a novel discriminative handcrafted descriptor called Histograms of dual-tree complex wavelet transform (Hist-DTCWT) is proposed. Moreover, a logistic regression for score-level fusion is used to enhance the matching process. Furthermore, a high-order knowledge-based feature using a tensor subspace model was proposed to combine multiple discriminative features. Tests were conducted out using three datasets, where the proposed method has outperformed the state the art. Typically, verification accuracies of 96.62%, 95.54% and 92.94% have been reached under Cornell KinFace, UB KinFace and TS KinFace datasets, respectively.

Learning spatio-temporal discriminative model for affine subspace based visual object tracking

Discriminative correlation filters (DCF) with powerful feature descriptors have proven to be very effective for advanced visual object tracking approaches. However, due to the fixed capacity in achieving discriminative learning, existing DCF trackers perform the filter training on a single template extracted by convolutional neural networks (CNN) or hand-crafted descriptors. Such single template learning cannot provide powerful discriminative filters with guaranteed validity under appearance variation. To pinpoint the structural relevance of spatio-temporal appearance to the filtering system, we propose a new tracking algorithm that incorporates the construction of the Grassmannian manifold learning in the DCF formulation. Our method constructs the model appearance within an online updated affine subspace. It enables joint discriminative learning in the origin and basis of the subspace, achieving enhanced discrimination and interpretability of the learned filters. In addition, to improve tracking efficiency, we adaptively integrate online incremental learning to update the obtained manifold. To this end, specific spatio-temporal appearance patterns are dynamically learned during tracking, highlighting relevant variations and alleviating the performance degrading impact of less discriminative representations from a single template. The experimental results obtained on several well-known datasets, i.e., OTB2013, OTB2015, UAV123, and VOT2018, demonstrate the merits of the proposed method and its superiority over the state-of-the-art trackers.

Fringe pattern analysis to evaluate light sources and sensors in digital photoelasticity.

When experimental photoelasticity images are acquired, the spectral interaction between the light source and the sensor used affect the visual information of the fringe patterns in the produced images. Such interaction can lead to fringe patterns with an overall high quality, but also can lead to images with indistinguishable fringes, and bad stress field reconstruction. We introduce a strategy to evaluate such interaction that relies on measuring the value of four handcrafted descriptors: contrast, an image descriptor that accounts simultaneously for blur and noise, a Fourier-based descriptor to measure image quality, and image entropy. The utility of the proposed strategy was validated by measuring the selected descriptors on computational photoelasticity images, and the fringe orders achieved when evaluating the stress field, from 240 spectral configurations: 24 light sources and 10 sensors. We found that high values of the selected descriptors can be related to spectral configurations that lead to better stress field reconstruction. Overall, the results show that the selected descriptors can be useful to identify bad and good spectral interactions, which could help to design better protocols for acquiring photoelasticity images.

An anisotropic Chebyshev descriptor and its optimization for deformable shape correspondence

Shape descriptors have recently gained popularity in shape matching, statistical shape modeling, etc. Their discriminative ability and efficiency play a decisive role in these tasks. In this paper, we first propose a novel handcrafted anisotropic spectral descriptor using Chebyshev polynomials, called the anisotropic Chebyshev descriptor (ACD); it can effectively capture shape features in multiple directions. The ACD inherits many good characteristics of spectral descriptors, such as being intrinsic, robust to changes in surface discretization, etc. Furthermore, due to the orthogonality of Chebyshev polynomials, the ACD is compact and can disambiguate intrinsic symmetry since several directions are considered. To improve the ACD’s discrimination ability, we construct a Chebyshev spectral manifold convolutional neural network (CSMCNN) that optimizes the ACD and produces a learned ACD. Our experimental results show that the ACD outperforms existing state-of-the-art handcrafted descriptors. The combination of the ACD and the CSMCNN is better than other state-of-the-art learned descriptors in terms of discrimination, efficiency, and robustness to changes in shape resolution and discretization.

Deep feature based cross-slide registration.

Registration of multiple sections in a tissue block is an important pre-requisite task before any cross-slide image analysis. Non-rigid registration methods are capable of finding correspondence by locally transforming a moving image. These methods often rely on an initial guess to roughly align an image pair linearly and globally. This is essential to prevent convergence to a non-optimal minimum. We explore a deep feature based registration (DFBR) method which utilises data-driven descriptors to estimate the global transformation. A multi-stage strategy is adopted for improving the quality of registration. A visualisation tool is developed to view registered pairs of WSIs at different magnifications. With the help of this tool, one can apply a transformation on the fly without the need to generate a transformed moving WSI in a pyramidal form. We compare the performance on our dataset of data-driven descriptors with that of hand-crafted descriptors. Our approach can align the images with only small registration errors. The efficacy of our proposed method is evaluated for a subsequent non-rigid registration step. To this end, the first two steps of the ANHIR winner's framework are replaced with DFBR to register image pairs provided by the challenge. The modified framework produce comparable results to those of the challenge winning team.

Cervical Cancer Diagnosis Based on Multi-Domain Features Using Deep Learning Enhanced by Handcrafted Descriptors

Cervical cancer, among the most frequent adverse cancers in women, could be avoided through routine checks. The Pap smear check is a widespread screening methodology for the timely identification of cervical cancer, but it is susceptible to human mistakes. Artificial Intelligence-reliant computer-aided diagnostic (CAD) methods have been extensively explored to identify cervical cancer in order to enhance the conventional testing procedure. In order to attain remarkable classification results, most current CAD systems require pre-segmentation steps for the extraction of cervical cells from a pap smear slide, which is a complicated task. Furthermore, some CAD models use only hand-crafted feature extraction methods which cannot guarantee the sufficiency of classification phases. In addition, if there are few data samples, such as in cervical cell datasets, the use of deep learning (DL) alone is not the perfect choice. In addition, most existing CAD systems obtain attributes from one domain, but the integration of features from multiple domains usually increases performance. Hence, this article presents a CAD model based on extracting features from multiple domains not only one domain. It does not require a pre-segmentation process thus it is less complex than existing methods. It employs three compact DL models to obtain high-level spatial deep features rather than utilizing an individual DL model with large number of parameters and layers as used in current CADs. Moreover, it retrieves several statistical and textural descriptors from multiple domains including spatial and time–frequency domains instead of employing features from a single domain to demonstrate a clearer representation of cervical cancer features, which is not the case in most existing CADs. It examines the influence of each set of handcrafted attributes on diagnostic accuracy independently and hybrid. It then examines the consequences of combining each DL feature set obtained from each CNN with the combined handcrafted features. Finally, it uses principal component analysis to merge the entire DL features with the combined handcrafted features to investigate the effect of merging numerous DL features with various handcrafted features on classification results. With only 35 principal components, the accuracy achieved by the quatric SVM of the proposed CAD reached 100%. The performance of the described CAD proves that combining several DL features with numerous handcrafted descriptors from multiple domains is able to boost diagnostic accuracy. Additionally, the comparative performance analysis, along with other present studies, shows the competing capacity of the proposed CAD.

Nucleus segmentation and classification using residual SE-UNet and feature concatenation approach incervical cytopathology cell images.

Introduction: Pap smear is considered to be the primary examination for the diagnosis of cervical cancer. But the analysis of pap smear slides is a time-consuming task and tedious as it requires manual intervention. The diagnostic efficiency depends on the medical expertise of the pathologist, and human error often hinders the diagnosis. Automated segmentation and classification of cervical nuclei will help diagnose cervical cancer in earlier stages. Materials and Methods: The proposed methodology includes three models: a Residual-Squeeze-and-Excitation-module based segmentation model, a fusion-based feature extraction model, and a Multi-layer Perceptron classification model. In the fusion-based feature extraction model, three sets of deep features are extracted from these segmented nuclei using the pre-trained and fine-tuned VGG19, VGG-F, and CaffeNet models, and two hand-crafted descriptors, Bag-of-Features and Linear-Binary-Patterns, are extracted for each image. For this work, Herlev, SIPaKMeD, and ISBI2014 datasets are used for evaluation. The Herlev datasetis used for evaluating both segmentation and classification models. Whereas the SIPaKMeD and ISBI2014 are used for evaluating the classification model, and the segmentation model respectively. Results: The segmentation network enhanced the precision and ZSI by 2.04%, and 2.00% on the Herlev dataset, and the precision and recall by 0.68%, and 2.59% on the ISBI2014 dataset. The classification approach enhanced the accuracy, recall, and specificity by 0.59%, 0.47%, and 1.15% on the Herlev dataset, and by 0.02%, 0.15%, and 0.22% on the SIPaKMed dataset. Conclusion: The experiments demonstrate that the proposed work achieves promising performance on segmentation and classification in cervical cytopathology cell images..

Dense correspondence of deformable volumetric images via deep spectral embedding and descriptor learning.

Deformable image correspondence plays an essential role in a variety of medical image analysis tasks. Most existing deep learning-based registration and correspondence techniques exploit metric space alignments in the spatial domain and learn a nonlinear voxel-wise mapping function between volumetric images and displacement fields, agnostic to intrinsic structure correspondence. When confronted with high-frequency perturbations of patients' poses and anatomical structural variations, they relied on prior rigid and affine transformations, as well as additional segmentation masks and landmark annotations for reliable registration. This paper presents a data-driven spectral mapping-based correspondence framework to handle the intrinsic correspondence of anatomical structures. At the core of our approach lies a deep convolutional framework that approximates spectral bases and optimizes volumetric descriptors. The multi-path graph convolutional network-based spectral embedding approximation module relieves the computationally expensive eigendecomposition-based embedding of volumetric images. The deep descriptor learning module surpasses the prior hand-crafted descriptors and the descriptor selection. We showcase the efficacy of the core modules, i.e., the spectral embedding approximation and descriptor learning, for volumetric image correspondence and the atlas-based registration on two volumetric image datasets. The proposed method achieves comparable correspondence accuracy with the state-of-the-art deep registration models, resilient to pose and shape perturbations.

A Self-Supervised Gait Encoding Approach With Locality-Awareness for 3D Skeleton Based Person Re-Identification.

Person re-identification (Re-ID) via gait features within 3D skeleton sequences is a newly-emerging topic with several advantages. Existing solutions either rely on hand-crafted descriptors or supervised gait representation learning. This paper proposes a self-supervised gait encoding approach that can leverage unlabeled skeleton data to learn gait representations for person Re-ID. Specifically, we first create self-supervision by learning to reconstruct unlabeled skeleton sequences reversely, which involves richer high-level semantics to obtain better gait representations. Other pretext tasks are also explored to further improve self-supervised learning. Second, inspired by the fact that motion's continuity endows adjacent skeletons in one skeleton sequence and temporally consecutive skeleton sequences with higher correlations (referred as locality in 3D skeleton data), we propose a locality-aware attention mechanism and a locality-aware contrastive learning scheme, which aim to preserve locality-awareness on intra-sequence level and inter-sequence level respectively during self-supervised learning. Last, with context vectors learned by our locality-aware attention mechanism and contrastive learning scheme, a novel feature named Constrastive Attention-based Gait Encodings (CAGEs) is designed to represent gait effectively. Empirical evaluations show that our approach significantly outperforms skeleton-based counterparts by 15-40 percent Rank-1 accuracy, and it even achieves superior performance to numerous multi-modal methods with extra RGB or depth information. Our codes are available at https://github.com/Kali-Hac/Locality-Awareness-SGE.

Feature Extraction of V–I Trajectory Using 2-D Fourier Series for Electrical Load Classification

Nonintrusive load monitoring (NILM) techniques allow the individual consumption of devices in an installation to be reported to the user, collaborating in the awareness and managing consumers’ energy use. One of the most critical steps in NILM procedures is feature extraction, which involves the quantitative description of the load signature. Most state-of-the-art methods, particularly those involving voltage–current ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V–I</i> ) trajectories, are based on features that specialists design when manually analyzing signals—the so-called hand-crafted features. On the other hand, this work presents a mathematical method that describes the transient and steady-state (SS) load signature using a 2-D Fourier series, avoiding hand-crafted descriptors. The proposed approach evaluates load identification results with different classifiers and publicly available datasets. In addition to the evaluation of the proposed method’s computational resources, such as execution time and memory footprint in an embedded system, their robustness to noise insertion is also assessed. The results of the proposed approach indicate an average per-class accuracy higher than 90%, equivalent or superior to hand-crafted <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V–I</i> trajectory-based methods in most cases and even deep learning approaches recently presented in the literature. Such results reinforce the main contributions of this work: 1) a mathematical method to extract interpretable features; 2) results comparable or superior to other state-of-the-art methods; and 3) promising results in computational resources in an embedded system.

LiDAR Odometry by Deep Learning-Based Feature Points with Two-Step Pose Estimation

An accurate ego-motion estimation solution is vital for autonomous vehicles. LiDAR is widely adopted in self-driving systems to obtain depth information directly and eliminate the influence of changing illumination in the environment. In LiDAR odometry, the lack of descriptions of feature points as well as the failure of the assumption of uniform motion may cause mismatches or dilution of precision in navigation. In this study, a method to perform LiDAR odometry utilizing a bird’s eye view of LiDAR data combined with a deep learning-based feature point is proposed. Orthographic projection is applied to generate a bird’s eye view image of a 3D point cloud. Thereafter, an R2D2 neural network is employed to extract keypoints and compute their descriptors. Based on those keypoints and descriptors, a two-step matching and pose estimation is designed to keep these feature points tracked over a long distance with a lower mismatch ratio compared to the conventional strategy. In the experiment, the evaluation of the proposed algorithm on the KITTI training dataset demonstrates that the proposed LiDAR odometry can provide more accurate trajectories compared with the handcrafted feature-based SLAM (Simultaneous Localization and Mapping) algorithm. In detail, a comparison of the handcrafted descriptors is demonstrated. The difference between the RANSAC (Random Sample Consensus) algorithm and the two-step pose estimation is also demonstrated experimentally. In addition, the data collected by Velodyne VLP-16 is also evaluated by the proposed solution. The low-drift positioning RMSE (Root Mean Square Error) of 4.70 m from approximately 5 km mileage shown in the result indicates that the proposed algorithm has generalization performance on low-resolution LiDAR.

MinkLoc3D-SI: 3D LiDAR Place Recognition With Sparse Convolutions, Spherical Coordinates, and Intensity

The 3D LiDAR place recognition aims to estimate a coarse localization in a previously seen environment based on a single scan from a rotating 3D LiDAR sensor. The existing solutions to this problem include hand-crafted point cloud descriptors (e.g., ScanContext, M2DP, LiDAR IRIS) and deep learning-based solutions (e.g., PointNetVLAD, PCAN, LPD-Net, DAGC, MinkLoc3D), which are often only evaluated on accumulated 2D scans from the Oxford RobotCar dataset. We introduce MinkLoc3D-SI, a sparse convolution-based solution that utilizes spherical coordinates of 3D points and processes the intensity of 3D LiDAR measurements, improving the performance when a single 3D LiDAR scan is used. Our method integrates the improvements typical for hand-crafted descriptors (like ScanContext) with the most efficient 3D sparse convolutions (MinkLoc3D). Our experiments show improved results on single scans from 3D LiDARs (USyd Campus dataset) and great generalization ability (KITTI dataset). Using intensity information on accumulated 2D scans (RobotCar Intensity dataset) improves the performance, even though spherical representation doesn’t produce a noticeable improvement. As a result, MinkLoc3D-SI is suited for single scans obtained from a 3D LiDAR, making it applicable in autonomous vehicles.

Re-Identification of Giant Sunfish using Keypoint Matching

We present the first work where re-identification ofthe Giant Sunfish (Mola alexandrini) is automated using computer vision and deep learning. We propose a pipeline that scores an mAP of 60.34% on a full rank of the novel TinyMola dataset which includes 31 IDs and 91 images. The method requires no domain-adaptation or training which makes it especially suited for low-budget or volunteer-based projects, like Match My Mola, as part of a human-in-the-loop model.&#x0D; The pipeline includes segmentation, keypoint detection and description, keypoint matching, and ranking. The choice of feature descriptor has the largest impact on the performance and we show that the deep learning based SuperPoint descriptor greatly outperforms handcrafted descriptors like SIFT and RootSIFT independent of the segmentation level and matching method. Combining SuperPoint and the graph neural network based SuperGlue matching method produces the best results.

Appearance-based loop closure detection combining lines and learned points for low-textured environments

Hand-crafted point descriptors have been traditionally used for visual loop closure detection. However, in low-textured environments, it is usually difficult to find enough point features and, hence, the performance of such algorithms degrade. Under this context, this paper proposes a loop closure detection method that combines lines and learned points to work, particularly, in scenarios where hand-crafted points fail. To index previous images, we adopt separate incremental binary Bag-of-Words (BoW) schemes for points and lines. Moreover, we adopt a binarization procedure for features’ descriptors to benefit from the advantages of learned features into a binary BoW model. Furthermore, image candidates from each BoW instance are merged using a novel query-adaptive late fusion approach. Finally, a spatial verification stage, which integrates appearance and geometry perspectives, allows us to enhance the global performance of the method. Our approach is validated using several public datasets, outperforming other state-of-the-art solutions in most cases, especially in low-textured scenarios.

Optical Flow with Learning Feature for Deformable Medical Image Registration

Deformable medical image registration plays a vital role in medical image applications, such as placing different temporal images at the same time point or different modality images into the same coordinate system. Various strategies have been developed to satisfy the increasing needs of deformable medical image registration. One popular registration method is estimating the displacement field by computing the optical flow between two images. The motion field (flow field) is computed based on either gray-value or handcrafted descriptors such as the scale-invariant feature transform (SIFT). These methods assume that illumination is constant between images. However, medical images may not always satisfy this assumption. In this study, we propose a metric learning-based motion estimation method called Siamese Flow for deformable medical image registration. We train metric learners using a Siamese network, which produces an image patch descriptor that guarantees a smaller feature distance in two similar anatomical structures and a larger feature distance in two dissimilar anatomical structures. In the proposed registration framework, the flow field is computed based on such features and is close to the real deformation field due to the excellent feature representation ability of the Siamese network. Experimental results demonstrate that the proposed method outperforms the Demons, SIFT Flow, Elastix, and VoxelMorph networks regarding registration accuracy and robustness, particularly with large deformations.