Fisher Vector Research Articles

Composite descriptor based on contour and appearance for plant species identification

In this paper, we propose an effective composite descriptor combining contour and appearance features for plant species identification. The composite descriptor uses two different fusion schemes: the first is a method based on the fusion of low-level local contour and appearance features; that is, we use local triangle features and local speeded-up robust features (SURF) to describe leaf images, respectively. Then, we utilize the Hausdorff distance to measure the leaf discrepancy to complete the plant identification task. The second is a method of global feature fusion based on high-level shape and appearance features. Specifically, the Fisher vector encodes the local triangle, and SURF features into high-level shape and appearance features. Then, the Euclidean distance is used to compute the dissimilarity of leaves to realize plant leaf recognition. The composite descriptor based on these two fusion schemes can well describe the characteristics of leaf images and improve the ability to identify plant leaves. This composite descriptor is extensively tested on several standard plant databases, including simple, compound, and natural scene plant leaf datasets. Experimental results demonstrate that the composite descriptor achieves high recognition accuracy, outperforming existing state-of-the-art benchmarks. In addition, we further applied our method directly to shape recognition and achieved good recognition results, demonstrating the wide range of applications of our method.

Can using a pre-trained deep learning model as the feature extractor in the bag-of-deep-visual-words model always improve image classification accuracy?

This article investigates whether higher classification accuracy can always be achieved by utilizing a pre-trained deep learning model as the feature extractor in the Bag-of-Deep-Visual-Words (BoDVW) classification model, as opposed to directly using the new classification layer of the pre-trained model for classification. Considering the multiple factors related to the feature extractor -such as model architecture, fine-tuning strategy, number of training samples, feature extraction method, and feature encoding method-we investigate these factors through experiments and then provide detailed answers to the question. In our experiments, we use five feature encoding methods: hard-voting, soft-voting, locally constrained linear coding, super vector coding, and fisher vector (FV). We also employ two popular feature extraction methods: one (denoted as Ext-DFs(CP)) uses a convolutional or non-global pooling layer, and another (denoted as Ext-DFs(FC)) uses a fully-connected or global pooling layer. Three pre-trained models-VGGNet-16, ResNext-50(32×4d), and Swin-B-are utilized as feature extractors. Experimental results on six datasets (15-Scenes, TF-Flowers, MIT Indoor-67, COVID-19 CXR, NWPU-RESISC45, and Caltech-101) reveal that compared to using the pre-trained model with only the new classification layer re-trained for classification, employing it as the feature extractor in the BoDVW model improves the accuracy in 35 out of 36 experiments when using FV. With Ext-DFs(CP), the accuracy increases by 0.13% to 8.43% (averaged at 3.11%), and with Ext-DFs(FC), it increases by 1.06% to 14.63% (averaged at 5.66%). Furthermore, when all layers of the pre-trained model are fine-tuned and used as the feature extractor, the results vary depending on the methods used. If FV and Ext-DFs(FC) are used, the accuracy increases by 0.21% to 5.65% (averaged at 1.58%) in 14 out of 18 experiments. Our results suggest that while using a pre-trained deep learning model as the feature extractor does not always improve classification accuracy, it holds great potential as an accuracy improvement technique.

Transfer Learning Models for CNN Fusion With Fisher Vector for Codebook Optimization of Foreground Features

Transfer Learning Models for CNN Fusion With Fisher Vector for Codebook Optimization of Foreground Features

Employing Deep Learning Techniques for the Identification and Assessment of Skin Cancer

These days, skin cancer is a prominent cause of death for people. Skin cancer is the name given to the abnormal development of skin cells that are exposed to the sun. These skin cells can develop anywhere on the human body. The majority of malignancies are treatable in the early stages. Thus, early detection of skin cancer is anticipated in order to preserve patient life. With cutting edge innovation, it is possible to detect skin cancer early on. Here, we provide a novel framework for the recognition of dermo duplication pictures that makes use of a neighbouring descriptor encoding method and deep learning technique. Specifically, the deep representations of a rescaled dermo duplication image that were initially removed through training an extraordinarily deep residual neural network on a big dataset of normal images. Subsequently, the neighbourhood deep descriptors are obtained by request-less visual measurement highlights, which rely on fisher vector encoding to create an international image representation. Lastly, a convolution neural network (CNN) was utilised to orchestrate melanoma images employing the Fisher vector encoded depictions. This proposed technique can give more discriminative parts to oversee huge contrasts inside melanoma classes and little varieties among melanoma and non-melanoma classes with least readiness information.

Moving scene object tracking method based on deep convolutional neural network

The effect of target tracking is not ideal when facing various complex tracking scenarios such as non-rigid deformation of target, frequent occlusion, clutter of target background and interference of similar objects. In this paper, the feature based on deep convolutional neural network is used for target tracking in moving scenes, and a sliding window target segmentation method is proposed to study the impact of data normalization and data set expansion on the final result. In order to select more distinguishing features, principal component analysis is used to process the features of Deep Convolution Neural Network (DCNN), and the features of different network layers of DCNN are compared. The feature coding algorithm is studied, and the extracted DCNN features are encoded by Fisher Vectors algorithm, and compared with the locality-constrained linear encoding technique. Experiments show that the feature based on deep convolutional neural network in this paper can obtain higher accuracy than the traditional feature fusion method. According to the result analysis, the tracking accuracy of deep convolutional neural network algorithm is improved under the condition of illumination variation. In the case of local occlusion, the tracking accuracy is also improved.

Alabib-65: A Realistic Dataset for Algerian Sign Language Recognition

Sign language recognition (SLR) is a promising research field that aims to blur boundaries between Deaf and hearing people by creating a system that can transcribe signs into a written or vocal language. There is a growing body of literature that investigates the recognition of different sign languages, especially American sign language. So far, to the best of our knowledge, no study has considered the Algerian SLR. This is mainly due to the lack of datasets. To address this issue, we created the Alabib-65, the first Algerian Sign Language dataset. It consists of up to 6,238 Videos recorded from 41 native signers under realistic settings. This dataset is challenging due to a variety of reasons. First, there is a little inter-class variability. The 65 sign classes are similar in terms of hands’ configuration, placement, or movement and can share the same sub-parts. Second, there is a large intra-class variability. Furthermore, compared to other SL datasets that were collected from an indoor environment with a static and simple background, our videos were recorded from both indoor and outdoor environments with 22 backgrounds varying from simple to cluttered, and from static to dynamic. To underpin future research, we provided baseline results on this new dataset using state-of-the-art machine learning methods, namely: IDTFs with Fisher vector and SVM-classifier, VGG16-GRU, I3D, I3D-GRU, and I3D-GRU-Attention. The results show the validity and the challenges of our dataset.

Regional-Based Object Detection Using Polarization and Fisher Vectors in Passive Millimeter-Wave Imaging

Passive millimeter-wave (PMMW) imaging is a powerful approach for detecting hidden objects underneath clothing. The theoretical basis of object detection methods is the contrast of brightness temperature (TB) image. TB differences may be caused by the diversity of material, physical temperature, surface structure, and so on. Existing methods are mainly based on single-polarization and single-pixel processing, which usually generate many discrete pixels of false alarms or missing detections. In this article, we present a regional-based method for hidden object detection using polarization and Fisher vector (FV) features. The necessity of polarization averaging for detection is revealed by theoretical simulation and experimental analyses. Based on the superpixel segmentation of polarization mean image, a modified FV, regional mean FV (RMFV), is created to extract concealed object features. Various imaging experimental data of typical security inspection scenarios are applied to verify the proposed method. The robustness and effectiveness are proved by comparing with several state-of-the-art methods.

Learning binary and sparse permutation-invariant representations for fast and memory efficient whole slide image search

Considering their gigapixel sizes, the representation of whole slide images (WSIs) for classification and retrieval systems is a non-trivial task. Patch processing and multi-Instance Learning (MIL) are common approaches to analyze WSIs. However, in end-to-end training, these methods require high GPU memory consumption due to the simultaneous processing of multiple sets of patches. Furthermore, compact WSI representations through binary and/or sparse representations are urgently needed for real-time image retrieval within large medical archives. To address these challenges, we propose a novel framework for learning compact WSI representations utilizing deep conditional generative modeling and the Fisher Vector Theory. The training of our method is instance-based, achieving better memory and computational efficiency during the training. To achieve efficient large-scale WSI search, we introduce new loss functions, namely gradient sparsity and gradient quantization losses, for learning sparse and binary permutation-invariant WSI representations called Conditioned Sparse Fisher Vector (C-Deep-SFV), and Conditioned Binary Fisher Vector (C-Deep-BFV). The learned WSI representations are validated on the largest public WSI archive, The Cancer Genomic Atlas (TCGA) and also Liver-Kidney-Stomach (LKS) dataset. For WSI search, the proposed method outperforms Yottixel and Gaussian Mixture Model (GMM)-based Fisher Vector both in terms of retrieval accuracy and speed. For WSI classification, we achieve competitive performance against state-of-art on lung cancer data from TCGA and the public benchmark LKS dataset.

Acoustic Scene Classification using Deep Fisher network

Acoustic Scene Classification (ASC) is the task of assigning a semantic label to an audio recording, based on the surrounding environment. In this work, a Fisher network is introduced for ASC. The proposed method mimics the working mechanism of a feed-forward Convolutional Neural Network (CNN) where, output of a layer is fed as an input to the succeeding layer. The Fisher network consists of a feature extraction step followed by a Fisher layer. The Fisher layer has three sub-layers, namely, Fisher Vector (FV) encoder, temporal pyramid and normalization layers along with feature reduction layer. Gammatone Time Cepstral Coefficients (GTCCs) and Mel-spectrograms are the features encoded as Fisher vector representation in FV encoder sub-layer. Temporal information of the Fisher vectors is retained using temporal pyramid sub-layer. After temporal pyramids are extracted from Fisher vectors, they are available as a feature vector. Irrelevant dimensions of the temporal pyramids are reduced further using Principal Component Analysis (PCA) in normalization and PCA sub-layers. The proposed model is evaluated on five DCASE datasets, TUT Urban Acoustic Scenes 2018 and Mobile, DCASE 2019 Acoustic Scene Classification Task 1(a) and Task 1(b), TAU Urban Acoustic Scenes 2020 datasets. The overall classification accuracy is 93%, 91%, 92%, 91% and 89% for TUT 2018, TUT Mobile 2018, DCASE Task 1(a) 2019, DCASE Task 1(b) 2019, and TAU Urban Acoustic Scenes 2020 datasets, respectively. The proposed model performed much better than the state-of-the-art ASC systems.

Looking at the Body: Automatic Analysis of Body Gestures and Self-Adaptors in Psychological Distress

Psychological distress is a significant and growing issue in society. Automatic detection, assessment, and analysis of such distress is an active area of research. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse. This is, in part, due to the limited available datasets and difficulty in automatically extracting useful body features. Recent advances in pose estimation and deep learning have enabled new approaches to this modality and domain. To enable this research, we have collected and analyzed a new dataset containing full body videos for short interviews and self-reported distress labels. We propose a novel method to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to be correlated with psychological distress. We perform analysis on statistical body gestures and fidgeting features to explore how distress levels affect participants' behaviors. We then propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector Encoding. We demonstrate that our proposed model, combining audio-visual features with automatically detected fidgeting behavioral cues, can successfully predict distress levels in a dataset labeled with self-reported anxiety and depression levels.

Multi-level contour combination features for shape recognition

We present a novel multi-level contour combination feature for shape recognition. This combination feature effectively solves large intra-class changes and nonlinear deformations of object shapes, thereby enhancing the performance of shape recognition. First, we divide the shape contour into two levels: the sampling points and the contour fragments, where sampling points are used to describe the detailed information of a shape and contour fragments are used to represent the global feature of a shape. Second, we employ the Fisher vector (FV) approach to encode the local sampling point feature and contour fragment feature as high-level characteristics. Finally, we combine the high-level characteristics after FV encoding and perform shape recognition through a linear support vector machine (SVM) model. The proposed method has been assessed on three benchmark shape datasets, including the Animal, MPEG-7,and ETH-80 datasets. Our method achieves 92.70%, 99.26% and 98.32% classification accuracy on the Animal, MPEG-7, and ETH-80 datasets, respectively. In addition, our method can also be applied to the classification of objects in real-word scenes. We combine the Weizmann Horse and the ETHZ Cow real-world scene datasets, and our method achieves 99.25% classification accuracy on the combined dataset. The recognition results of our approach are better than prior state-of-the-art shape recognition methods, which demonstrate the effectiveness and superiority of our approach.

BoFF: A bag of fuzzy deep features for texture recognition

Here, we propose a novel method for texture recognition that employs fuzzy modeling over deep learning features. Specifically, the well-established pipeline of deep filter banks for texture description is followed, but using fuzzy equivalence measures for aggregating the deep features. This solution is more robust than a simple “all-or-nothing” assignment used on bag-of-visual-words, and it is less expensive than complex statistical representations such as Fisher vectors. Additionally, it avoids dependence on strong assumptions about specific distributions. The proposed method is evaluated on texture classification tasks, including both benchmark databases and a practical task in botany. In both cases, the results were competitive with state-of-the-art methods and suggest the potential of this combination for texture analysis in general.

Handsight: DeCAF & Improved Fisher Vectors to Classify Clothing Color and Texture With a Finger-Mounted Camera

Handsight: DeCAF & Improved Fisher Vectors to Classify Clothing Color and Texture With a Finger-Mounted Camera

A Learning Robust and Discriminative Shape Descriptor for Plant Species Identification.

Plant identification based on leaf images is a widely concerned application field in artificial intelligence and botany. The key problem is extracting robust discriminative features from leaf images and assigning a measure of similarity. This study proposes an effective, robust shape descriptor to identify plant species from images of their leaves, which we call the high-level triangle shape descriptor (HTSD). First, we extract a leaf image's external contour and internal salient point information. We then use triangle features to describe the leaf contour, which we call the contour point based on triangle features (CPTFs). The internal information of the leaf image is based on salient point triangle features (SPTFs). The third step is to apply the Fisher vector to encode the two kinds of point-based local triangle features into the HTSD. Finally, we employ the simple euclidean distance to calculate the dissimilarities between the HTSD characteristics of leaf images. We have extensively evaluated the proposed approach on several public leaf datasets successfully. Experimental results show that our method has superior recognition accuracy, outperforming current state-of-the-art shape-based and deep-learning plant identification approaches.

Two-stream temporal enhanced Fisher vector encoding for skeleton-based action recognition

The key to skeleton-based action recognition is how to extract discriminative features from skeleton data. Recently, graph convolutional networks (GCNs) are proven to be highly successful for skeleton-based action recognition. However, existing GCN-based methods focus on extracting robust features while neglecting the information of feature distributions. In this work, we aim to introduce Fisher vector (FV) encoding into GCN to effectively utilize the information of feature distributions. However, since the Gaussian Mixture Model (GMM) is employed to fit the global distribution of features, Fisher vector encoding inevitably leads to losing temporal information of actions, which is demonstrated by our analysis. To tackle this problem, we propose a temporal enhanced Fisher vector encoding algorithm (TEFV) to provide more discriminative visual representation. Compared with FV, our TEFV model can not only preserve the temporal information of the entire action but also capture fine-grained spatial configurations and temporal dynamics. Moreover, we propose a two-stream framework (2sTEFV-GCN) by combining the TEFV model with the GCN model to further improve the performance. On two large-scale datasets for skeleton-based action recognition, NTU-RGB+D 60 and NTU-RGB+D 120, our model achieves state-of-the-art performance.

Detection of Neurogenic Voice Disorders Using the Fisher Vector Representation of Cepstral Features

Neurogenic voice disorders (NVDs) are caused by damage or malfunction of the central or peripheral nervous system that controls vocal fold movement. In this paper, we investigate the potential of the Fisher vector (FV) encoding in automatic detection of people with NVDs. FVs are used to convert features from frame level (local descriptors) to utterance level (global descriptors). At the frame level, we extract two popular cepstral representations, namely, Mel-frequency cepstral coefficients (MFCCs) and perceptual linear prediction cepstral coefficients (PLPCCs), from acoustic voice signals. In addition, the MFCC features are also extracted from every frame of the glottal source signal computed using a glottal inverse filtering (GIF) technique. The global descriptors derived from the local descriptors are used to train a support vector machine (SVM) classifier. Experiments are conducted using voice signals from 80 healthy speakers and 80 patients with NVDs (40 with spasmodic dysphonia (SD) and 40 with recurrent laryngeal nerve palsy (RLNP)) taken from the Saarbruecken voice disorder (SVD) database. The overall results indicate that the use of the FV encoding leads to better identification of people with NVDs, compared to the defacto temporal encoding. Furthermore, the SVM trained using the combination of FVs derived from the cepstral and glottal features provides the overall best detection performance.

Robust bag classification approach for multi-instance learning via subspace fuzzy clustering

Multi-instance learning (MIL) allows predictive algorithms to use complex data representation. The data in MIL is organized in the form of labeled bags of instances, and the labels of instances are not available in the training phase. The processing and classification of complex bag representation result in a complicated hypothesis space. Moreover, identifying essential instances inside is also important, as these instances trigger positive labels for the bag and play a vital role in the model interpretation and bag classification. The recent MIL algorithms are not robust to tackle hypothesis space complication. Additionally, the existing instance selection algorithms are based on explicit assumptions regarding the relationship of instances to the bag label. However, these assumptions may hold for a specific bag in the dataset but may not apply to the whole dataset. To deal with the hypothesis space complication and robust instance selection without any prior assumption, this paper proposes a fuzzy subspace clustering approach for robust instance selection and ensemble-based variant for Fisher vector (FV) encoding to solve MIL problems, named (FCBE-miFV). Specifically, the proposed algorithm uses a subspace fuzzy clustering approach to compute instance selection probabilities, selects essential instances from the bag, transforms the input bag into FV, and classifies the generated bag encodings using a stacking-based ensemble approach to obtain improved bag level classification performance. The proposed FCBE-miFV improves model performance by incorporating essential instances in the bag encoding process. The experimental results show that the FCBE-miFV obtained comparable performance to the state-of-the-art MIL problems.

Discriminative Multi-View Dynamic Image Fusion for Cross-View 3-D Action Recognition.

Dramatic imaging viewpoint variation is the critical challenge toward action recognition for depth video. To address this, one feasible way is to enhance view-tolerance of visual feature, while still maintaining strong discriminative capacity. Multi-view dynamic image (MVDI) is the most recently proposed 3-D action representation manner that is able to compactly encode human motion information and 3-D visual clue well. However, it is still view-sensitive. To leverage its performance, a discriminative MVDI fusion method is proposed by us via multi-instance learning (MIL). Specifically, the dynamic images (DIs) from different observation viewpoints are regarded as the instances for 3-D action characterization. After being encoded using Fisher vector (FV), they are then aggregated by sum-pooling to yield the representative 3-D action signature. Our insight is that viewpoint aggregation helps to enhance view-tolerance. And, FV can map the raw DI feature to the higher dimensional feature space to promote the discriminative power. Meanwhile, a discriminative viewpoint instance discovery method is also proposed to discard the viewpoint instances unfavorable for action characterization. The wide-range experiments on five data sets demonstrate that our proposition can significantly enhance the performance of cross-view 3-D action recognition. And, it is also applicable to cross-view 3-D object recognition. The source code is available at https://github.com/3huo/ActionView.

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition.

Increasingly, the task of detecting and recognizing the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors, like light detection and ranging (LiDAR), can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply convolutional neural networks to 3-D data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This article proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-D modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behavior, surveillance, and perhaps most notably by autonomous vehicles (AVs) to data-driven decision-making policies in urban areas and indoor environments.

Defect classification on semiconductor wafers using Fisher vector and visual vocabularies coding

The main trend in the classification of defects in semiconductor wafers from scanning electron microscopy images is the use of convolutional neural networks. These methods allow automatic feature extraction and supply high accuracy. However, they have limitations since they need a large number of samples in the training phase and are computationally expensive in general. Methods based on visual vocabularies are an efficient alternative to this type of networks that can be implemented with a reduced number of training samples. Although visual vocabularies have been successfully used in different image classification problems, to our knowledge they have never been tested in the classification of defects in semiconductor material wafers from scanning electron microscopy images. On the rectangular region holding the defect, the scale-invariant feature transform algorithm is applied to determine the points of interest and to calculate the descriptors of the patches containing these points. By training a clustering algorithm with these features, a visual vocabulary is created to describe each image as a bag of visual words. Two approaches for coding the descriptors are studied: (i) The more traditional one performs a count of the occurrence of each visual word, and (ii) a modern approach known as Fisher vector which measures the deviation between each descriptor and the generative Gaussian mixture model. Comparing both variants using support vector machines it is seen that Fisher vector coding is computationally more efficient and reports better classification results. The usefulness of visual vocabularies for defect classification on semiconductor wafers is showed by the 91.89% and 93.13% accuracy obtained for bag of visual words and Fisher vector coding, respectively. The training and testing times are lower than 5 min for both approaches.