Learned Feature Space Research Articles

Distance-based loss function for deep feature space learning of convolutional neural networks

Convolutional Neural Networks (CNNs) have been on the forefront of neural network research in recent years. Their breakthrough performance in fields such as image classification has gathered efforts in the development of new CNN-based architectures, but recently more attention has been directed to the study of new loss functions. Softmax loss remains the most popular loss function due mainly to its efficiency in class separation, but the function is unsatisfactory in terms of intra-class compactness. While some studies have addressed this problem, most solutions attempt to refine softmax loss or combine it with other approaches. We present a novel loss function based on distance matrices (LDMAT), softmax independent, that maximizes interclass distance and minimizes intraclass distance. The loss function operates directly on deep features, allowing their use on arbitrary classifiers. LDMAT minimizes the distance between two distance matrices, one constructed with the model’s deep features and the other calculated from the labels. The use of a distance matrix in the loss function allows a two-dimensional representation of features and imposes a fixed distance between classes, while improving intra-class compactness. A regularization method applied to the distance matrix of labels is also presented, that allows a degree of relaxation of the solution and leads to a better spreading of features in the separation space. Efficient feature extraction was observed on datasets such as MNIST, CIFAR10 and CIFAR100.

Adaptive-propagating heterophilous graph convolutional network

Graph convolutional networks have significant advantages in dealing with graph-structured data, but most existing methods usually potentially assume that nodes belonging to the same class in a graph tend to form edges, yet inter-class edges exist in many real-world graph-structured data. Due to the propagation mechanism of graph convolutional networks, it is challenging to prevent the interference aggregation from nodes of different classes, which may result in the incorporation of noise and irrelevant data in the outcome, ultimately decreasing the performance of the model. In this paper, we propose a new framework to address this issue on heterophilous graph-structured data. The proposed method comprises two main components. On one hand, the homophily of the graph-structured data is modeled so that the method can adaptively adjust the information propagation process according to the homophily of the edges, and mitigate the influence of inter-class information. On the other hand, the implicit node interaction is captured through the learned feature space, which is then fused with the original interaction to aggregate sufficient intra-class knowledge. Extensive experiments on real-world datasets demonstrate the superiority of the proposed method against current state-of-the-art approaches. 22Code to reproduce our experiments is available at https://github.com/rebridger/APHGCN.

Deep Learning to Authenticate Traditional Handloom Textile

Handloom textile products play an essential role in both the financial and cultural landscape of natives, necessitating accurate and efficient methods for authenticating against replicated powerloom textiles for the protection of heritage and indigenous weavers’ economic viability. This paper presents a new approach to the automated identification of handloom textiles leveraging a deep metric learning technique. A labeled handloom textile dataset of 25,166 images was created by collecting handloom textile samples of six unique types, working with indigenous weavers in Assam, Northeast India. The proposed method achieved remarkable success by acquiring biased feature representations that facilitate the effective separation of different fiber types in a learned feature space. Through extensive experimentation and comparison with baseline models, our approach demonstrated superior efficiency in classifying handloom textiles with an accuracy of 97.8%. Our approach not only contributes to the preservation and promotion of traditional textile craftsmanship in the region but also highlights its significance.

Slow and steady: auditory features for discriminating animal vocalizations.

We propose that listeners can use temporal regularities - spectro-temporal correlations that change smoothly over time - to discriminate animal vocalizations within and between species. To test this idea, we used Slow Feature Analysis (SFA) to find the most temporally regular components of vocalizations from birds (blue jay, house finch, American yellow warbler, and great blue heron), humans (English speakers), and rhesus macaques. We projected vocalizations into the learned feature space and tested intra-class (same speaker/species) and inter-class (different speakers/species) auditory discrimination by a trained classifier. We found that: 1) Vocalization discrimination was excellent (> 95%) in all cases; 2) Performance depended primarily on the ~10 most temporally regular features; 3) Most vocalizations are dominated by ~10 features with high temporal regularity; and 4) These regular features are highly correlated with the most predictable components of animal sounds.

Video-Based Sign Language Recognition via ResNet and LSTM Network.

Sign language recognition technology can help people with hearing impairments to communicate with non-hearing-impaired people. At present, with the rapid development of society, deep learning also provides certain technical support for sign language recognition work. In sign language recognition tasks, traditional convolutional neural networks used to extract spatio-temporal features from sign language videos suffer from insufficient feature extraction, resulting in low recognition rates. Nevertheless, a large number of video-based sign language datasets require a significant amount of computing resources for training while ensuring the generalization of the network, which poses a challenge for recognition. In this paper, we present a video-based sign language recognition method based on Residual Network (ResNet) and Long Short-Term Memory (LSTM). As the number of network layers increases, the ResNet network can effectively solve the granularity explosion problem and obtain better time series features. We use the ResNet convolutional network as the backbone model. LSTM utilizes the concept of gates to control unit states and update the output feature values of sequences. ResNet extracts the sign language features. Then, the learned feature space is used as the input of the LSTM network to obtain long sequence features. It can effectively extract the spatio-temporal features in sign language videos and improve the recognition rate of sign language actions. An extensive experimental evaluation demonstrates the effectiveness and superior performance of the proposed method, with an accuracy of 85.26%, F1-score of 84.98%, and precision of 87.77% on Argentine Sign Language (LSA64).

ScTPC: a novel semisupervised deep clustering model for scRNA-seq data.

Continuous advancements in single-cell RNA sequencing (scRNA-seq) technology have enabled researchers to further explore the study of cell heterogeneity, trajectory inference, identification of rare cell types, and neurology. Accurate scRNA-seq data clustering is crucial in single-cell sequencing data analysis. However, the high dimensionality, sparsity, and presence of "false" zero values in the data can pose challenges to clustering. Furthermore, current unsupervised clustering algorithms have not effectively leveraged prior biological knowledge, making cell clustering even more challenging. This study investigates a semisupervised clustering model called scTPC, which integrates the triplet constraint, pairwise constraint, and cross-entropy constraint based on deep learning. Specifically, the model begins by pretraining a denoising autoencoder based on a zero-inflated negative binomial distribution. Deep clustering is then performed in the learned latent feature space using triplet constraints and pairwise constraints generated from partial labeled cells. Finally, to address imbalanced cell-type datasets, a weighted cross-entropy loss is introduced to optimize the model. A series of experimental results on 10 real scRNA-seq datasets and five simulated datasets demonstrate that scTPC achieves accurate clustering with a well-designed framework. scTPC is a Python-based algorithm, and the code is available from https://github.com/LF-Yang/Code or https://zenodo.org/records/10951780.

Feature space separation by conformity loss driven training of CNN

Convolutional neural networks (CNNs) have enabled tremendous achievements in image classification, as the model can automatically extract image features and assign a proper classification. Nevertheless, the classification is lacking robustness to — for humans’ invisible perturbations on the input. To improve the robustness of the CNN model, it is necessary to understand the decision-making procedure of CNN models. By inspecting the learned feature space, we found that the classification regions are not always clearly separated by the CNN model. The overlap of classification regions increases the possibility to less perturbation induced input changes on classification results. Therefore, the clear separation of feature spaces of the CNN model should support decision robustness. In this paper, we propose to use a novel loss function called “conformity loss” to strengthen disjoint feature spaces during learning at different layers of the CNN, in order to improve the intra-class compactness and inter-class differences in trained representations. The same function was used as an evaluation metric to measure the feature space separation during the testing process. In conclusion, the conformity loss driven trained model has shown better feature space separation at comparable output performance.

HQ3DAvatar: High-quality Implicit 3D Head Avatar

Multi-view volumetric rendering techniques have recently shown great potential in modeling and synthesizing high-quality head avatars. A common approach to capture full head dynamic performances is to track the underlying geometry using a mesh-based template or 3D cube-based graphics primitives. While these model-based approaches achieve promising results, they often fail to learn complex geometric details such as the mouth interior, hair, and topological changes over time. This article presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. It leverages multiresolution hash encoding in the learned feature space, allowing for high quality, faster training, and high-resolution rendering. At test time, our method is driven by a monocular RGB video. Here, an image encoder extracts face-specific features that also condition the learnable canonical space. This encourages deformation-dependent texture variations during training. We also propose a novel optical flow-based loss that ensures correspondences in the learned canonical space, thus encouraging artifact-free and temporally consistent renderings. We show results on challenging facial expressions and show free-viewpoint renderings at interactive real-time rates for a resolution of 480 x 270. Our method outperforms related approaches both visually and numerically. We will release our multiple-identity dataset to encourage further research.

Robust ensemble person reidentification via orthogonal fusion with occlusion handling

Occlusion remains one of the major challenges in person reidentification (ReID) due to the diversity of poses and the variation of appearances. Developing novel architectures to improve the robustness of occlusion-aware person Re-ID requires new insights, especially on low-resolution edge cameras. We propose a deep ensemble model that harnesses both CNN and Transformer architectures to generate robust feature representations. To achieve robust Re-ID without manually labeling occluded regions, we propose to take an ensemble learning-based approach derived from the analogy between arbitrarily shaped occluded regions and robust feature representation. Using the orthogonality principle, our developed deep CNN model uses masked autoencoder (MAE) and global–local feature fusion for robust person identification. Furthermore, we present a part occlusion-aware transformer capable of learning feature space that is robust to occluded regions. Experimental results are reported on several Re-ID datasets to show the effectiveness of our developed ensemble model named orthogonal fusion with occlusion handling (OFOH). Compared to competing methods, the proposed OFOH approach has achieved competent rank-1 and mAP performance.

High-Order Structure Based Middle-Feature Learning for Visible-Infrared Person Re-identification

Visible-infrared person re-identification (VI-ReID) aims to retrieve images of the same persons captured by visible (VIS) and infrared (IR) cameras. Existing VI-ReID methods ignore high-order structure information of features while being relatively difficult to learn a reasonable common feature space due to the large modality discrepancy between VIS and IR images. To address the above problems, we propose a novel high-order structure based middle-feature learning network (HOS-Net) for effective VI-ReID. Specifically, we first leverage a short- and long-range feature extraction (SLE) module to effectively exploit both short-range and long-range features. Then, we propose a high-order structure learning (HSL) module to successfully model the high-order relationship across different local features of each person image based on a whitened hypergraph network. This greatly alleviates model collapse and enhances feature representations. Finally, we develop a common feature space learning (CFL) module to learn a discriminative and reasonable common feature space based on middle features generated by aligning features from different modalities and ranges. In particular, a modality-range identity-center contrastive (MRIC) loss is proposed to reduce the distances between the VIS, IR, and middle features, smoothing the training process. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets show that our HOS-Net achieves superior state-of-the-art performance. Our code is available at https://github.com/Jaulaucoeng/HOS-Net.

Echoes of images: multi-loss network for image retrieval in vision transformers.

This paper introduces a novel approach to enhance content-based image retrieval, validated on two benchmark datasets: ISIC-2017 and ISIC-2018. These datasets comprise skin lesion images that are crucial for innovations in skin cancer diagnosis and treatment. We advocate the use of pre-trained Vision Transformer (ViT), a relatively uncharted concept in the realm of image retrieval, particularly in medical scenarios. In contrast to the traditionally employed Convolutional Neural Networks (CNNs), our findings suggest that ViT offers a more comprehensive understanding of the image context, essential in medical imaging. We further incorporate a weighted multi-loss function, delving into various losses such as triplet loss, distillation loss, contrastive loss, and cross-entropy loss. Our exploration investigates the most resilient combination of these losses to create a robust multi-loss function, thus enhancing the robustness of the learned feature space and ameliorating the precision and recall in the retrieval process. Instead of using all the loss functions, the proposed multi-loss function utilizes the combination of only cross-entropy loss, triplet loss, and distillation loss and gains improvement of 6.52% and 3.45% for mean average precision over ISIC-2017 and ISIC-2018. Another innovation in our methodology is a two-branch network strategy, which concurrently boosts image retrieval and classification. Through our experiments, we underscore the effectiveness and the pitfalls of diverse loss configurations in image retrieval. Furthermore, our approach underlines the advantages of retrieval-based classification through majority voting rather than relying solely on the classification head, leading to enhanced prediction for melanoma - the most lethal type of skin cancer. Our results surpass existing state-of-the-art techniques on the ISIC-2017 and ISIC-2018 datasets by improving mean average precision by 1.01% and 4.36% respectively, emphasizing the efficacy and promise of Vision Transformers paired with our tailor-made weighted loss function, especially in medical contexts. The proposed approach's effectiveness is substantiated through thorough ablation studies and an array of quantitative and qualitative outcomes. To promote reproducibility and support forthcoming research, our source code will be accessible on GitHub.

Profiles of visual perceptual learning in feature space

Visual perceptual learning (VPL), experience-induced gains in discriminating visual features, has been studied extensively and intensively for many years, its profile in feature space, however, remains unclear. Here, human subjects were trained to perform either a simple low-level feature (grating orientation) or a complex high-level object (face view) discrimination task over a long-time course. During, immediately after, and one month after training, all results showed that in feature space VPL in grating orientation discrimination was a center-surround profile; VPL in face view discrimination, however, was a monotonic gradient profile. Importantly, these two profiles can be emerged by a deep convolutional neural network with a modified AlexNet consisted of 7 and 12 layers, respectively. Altogether, our study reveals for the first time a feature hierarchy-dependent profile of VPL in feature space, placing a necessary constraint on our understanding of the neural computation of VPL.

Multimodal Brain Tumor Segmentation Boosted by Monomodal Normal Brain Images.

Many deep learning based methods have been proposed for brain tumor segmentation. Most studies focus on deep network internal structure to improve the segmentation accuracy, while valuable external information, such as normal brain appearance, is often ignored. Inspired by the fact that radiologists often screen lesion regions with normal appearance as reference in mind, in this paper, we propose a novel deep framework for brain tumor segmentation, where normal brain images are adopted as reference to compare with tumor brain images in a learned feature space. In this way, features at tumor regions, i.e., tumor-related features, can be highlighted and enhanced for accurate tumor segmentation. It is known that routine tumor brain images are multimodal, while normal brain images are often monomodal. This causes the feature comparison a big issue, i.e., multimodal vs. monomodal. To this end, we present a new feature alignment module (FAM) to make the feature distribution of monomodal normal brain images consistent/inconsistent with multimodal tumor brain images at normal/tumor regions, making the feature comparison effective. Both public (BraTS2022) and in-house tumor brain image datasets are used to evaluate our framework. Experimental results demonstrate that for both datasets, our framework can effectively improve the segmentation accuracy and outperforms the state-of-the-art segmentation methods. Codes are available at https://github.com/hb-liu/Normal-Brain-Boost-Tumor-Segmentation.

Contrastive Learning of View-invariant Representations for Facial Expressions Recognition

Although there has been much progress in the area of facial expression recognition (FER), most existing methods suffer when presented with images that have been captured from viewing angles that are non-frontal and substantially different from those used in the training process. In this article, we propose ViewFX, a novel view-invariant FER framework based on contrastive learning, capable of accurately classifying facial expressions regardless of the input viewing angles during inference. ViewFX learns view-invariant features of expression using a proposed self-supervised contrastive loss, which brings together different views of the same subject with a particular expression in the embedding space. We also introduce a supervised contrastive loss to push the learned view-invariant features of each expression away from other expressions. Since facial expressions are often distinguished with very subtle differences in the learned feature space, we incorporate the Barlow twins loss to reduce the redundancy and correlations of the representations in the learned representations. The proposed method is a substantial extension of our previously proposed CL-MEx, which only had a self-supervised loss. We test the proposed framework on two public multi-view facial expression recognition datasets, KDEF and DDCF. The experiments demonstrate that our approach outperforms previous works in the area and sets a new state-of-the-art for both datasets while showing considerably less sensitivity to challenging angles and the number of output labels used for training. We also perform detailed sensitivity and ablation experiments to evaluate the impact of different components of our model as well as its sensitivity to different parameters.

Robust Label and Feature Space Co-Learning for Multi-Label Classification

Multi-label classification remains a challenging task for high-dimensional data samples and their labels both increase the complexity of training models. In this paper, we propose a Robust Label and Feature Space Co-Learning method, referred to as RLFSCL, for multi-label classification. Different from traditional multi-label classification methods which focus on feature space learning through regression directly between data samples and labels, our proposed method can further learn robust low rank label space from this traditional regression method. Therefore, our RLFSCL can learn better low rank feature and label representations simultaneously in original noisy and high dimensional spaces. Experimental comparison on five benchmark datasets, including Rcv1s5, Cal500, and Corel16k4 shows that the proposed RLFSCL algorithm outperforms state-of-the-art multi-label classification methods. The code of RLFSCL is made available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JingChuanTang/RLFSCL.</uri>

Improving metric-based few-shot learning with dynamically scaled softmax loss

The metric-based learning framework has been widely used in data-scarce few-shot visual classification. However, the current loss function limits the effectiveness of metric learning. One issue is that the nearest neighbor classification technique used greatly narrows the value range of similarity between the query and class prototypes, which limits the guiding ability of the loss function. The other issue is that the episode-based training setting randomizes the class combination in each iteration, which reduces the perception of the traditional softmax losses for effective learning from episodes with various data distributions.To solve these problems, we first review some variants of the softmax loss from a unified perspective, and then propose a novel Dynamically-Scaled Softmax Loss (DSSL). By adding a probability regulator (for scaling probabilities) and a loss regulator (for scaling losses), the loss function can adaptively adjust the prediction distribution and the training weights of the samples, which forces the model to focus on more informative samples. Finally, we found the proposed DSSL strategy for few-shot classifiers can achieve competitive results on four generic benchmarks and a fine-grained benchmark, demonstrating the effectiveness of improving the distinguishability (for base classes) and generalizability (for novel classes) of the learned feature space.

Automatic Loss Function Search for Adversarial Unsupervised Domain Adaptation

Unsupervised domain adaption (UDA) aims to reduce the domain gap between labeled source and unlabeled target domains. Many prior works exploit adversarial learning that leverages pre-designed discriminators to drive the network for aligning distributions between domains. However, most of them do not consider the degeneration of the domain discriminators caused by the gradually dominating gradients of aligned target samples during training, and they still suffer from the cross-domain semantic mismatch problem in the learned feature space. Hence, this paper attempts to understand and solve both issues from the lens of optimization loss and propose an automatic loss function search for adversarial domain adaptation (ALSDA). First, we extend the common adversarial loss by adding an adjustable hyper-parameter that can re-weight the gradients assigned to target samples, so that the domain discriminator can impose consecutive and influential driving forces for domain alignment. Meanwhile, we upgrade the traditional orthogonality loss with class-wisely adjustable hyper-parameters that can strengthen the cross-domain feature separation. Since manually determining the optimal loss functions requires expensive expert efforts, we leverage the popular AutoML to automatically search for the optimal loss functions from a pre-defined novel and unique search space for UDA. Further, to enable the loss function search when the target domain is unlabeled, we introduce a simple-but-effective entropy-guided search strategy with the aid of REINFORCE learning. Extensive experiments on various typical baselines and benchmark datasets such as Office-Home, Office-31, and Birds-31 have been conducted, and the results validate the generalization and superiority of the proposed ALSDA.

Semantic Knowledge Guided Class-Incremental Learning

Driven by practical needs, research on Class-Incremental Learning (CIL) has received more and more attentions in recent years. A technical challenge to be conquered by CIL methods is the catastrophic forgetting problem, where the model’s performance improves rapidly on new classes while deteriorates drastically on old ones. The main causes behind catastrophic forgetting include network drifts, inter-class confusions, etc. In this paper, we propose a novel CIL method that solves the catastrophic forgetting problem from two aspects. First, to solve the inter-class confusion problem, we propose a novel Semantic knOwledge gUided ciL framework (SOUL) that consists of a CNN feature extractor and a Bi-GCN (Graph Convolutional Network) classifier. In each CIL session, we use the semantic knowledge extracted from the class labels to build two inter-class relation graphs among all the encountered old and new classes. Using these two relation graphs, we develop a Bi-GCN classifier to fuse two kinds of semantic relations in a balanced way, and then to transfer the inter-class relations from semantic modality to image classification weights. The entire SOUL framework is trained end-to-end by the standard BP algorithm, which optimizes the Bi-GCN classifier and the CNN feature extractor jointly. Second, to prevent the network drift, we develop the local topology preserving strategy that divides the global topological structure of the learned feature space into a set of local topological relations, and maintains these local relations at CIL session. Experimental evaluations demonstrate the state-of-the-art performance accuracies on benchmark image classification datasets.

Synthetic‐to‐realistic domain adaptation for cold‐start of rail inspection systems

AbstractRail surface defects are potential danger factors for railway systems, and visual inspection of surface defects plays a vital role in rail maintenance. Recently, the methods based on deep learning have been widely used in rail inspection systems, but such systems often face the problem of a lack of defect samples for training deep learning models at start‐up, which is called the cold‐start problem. It is challenging to obtain sufficient defect samples since defects are sparse and even non‐existent for a running railway system. Therefore, a synthetic‐to‐realistic domain adaptation (SRDA) method is proposed for real‐world rail inspection. SRDA adapts synthetic images to look more realistic for reducing the domain gap between synthetic images and realistic rail surface images and obtains translated images that consist of synthetic defect information and realistic rail background. After that, the translated images are used to train a detector for rail inspection of rail surface defects. In order to make the detector more robust to complex backgrounds, SRDA generates images with the same defect‐level semantics but with different texture appearances and makes the detector align these images in the learned feature space. In addition, the synthetic and realistic rail surface defects (SRRSD) dataset containing 20,662 images is built. The experimental results on SRRSD show that SRDA achieves higher detection performance than other established domain adaption methods with 19.0% for and 26.7% for average precision.

Generative models of birdsong learning link circadian fluctuations in song variability to changes in performance.

Learning skilled behaviors requires intensive practice over days, months, or years. Behavioral hallmarks of practice include exploratory variation and long-term improvements, both of which can be impacted by circadian processes. During weeks of vocal practice, the juvenile male zebra finch transforms highly variable and simple song into a stable and precise copy of an adult tutor's complex song. Song variability and performance in juvenile finches also exhibit circadian structure that could influence this long-term learning process. In fact, one influential study reported juvenile song regresses towards immature performance overnight, while another suggested a more complex pattern of overnight change. However, neither of these studies thoroughly examined how circadian patterns of variability may structure the production of more or less mature songs. Here we relate the circadian dynamics of song maturation to circadian patterns of song variation, leveraging a combination of data-driven approaches. In particular we analyze juvenile singing in learned feature space that supports both data-driven measures of song maturity and generative developmental models of song production. These models reveal that circadian fluctuations in variability lead to especially regressive morning variants even without overall overnight regression, and highlight the utility of data-driven generative models for untangling these contributions.