Image Classification Framework Research Articles

A Unified Asymmetric Knowledge Distillation Framework for Image Classification

Knowledge distillation is a model compression technique that transfers knowledge learned by teacher networks to student networks. Existing knowledge distillation methods greatly expand the forms of knowledge, but also make the distillation models complex and symmetric. However, few studies have explored the commonalities among these methods. In this study, we propose a concise distillation framework to unify these methods and a method to construct asymmetric knowledge distillation under the framework. Asymmetric distillation aims to enable differentiated knowledge transfers for different distillation objects. We designed a multi-stage shallow-wide branch bifurcation method to distill different knowledge representations and a grouping ensemble strategy to supervise the network to teach and learn selectively. Consequently, we conducted experiments using image classification benchmarks to verify the proposed method. Experimental results show that our implementation can achieve considerable improvements over existing methods, demonstrating the effectiveness of the method and the potential of the framework.

A multimodal transfer learning framework for the classification of disaster-related social media images

Rapidly classifying disaster-related social media (SM) images during a catastrophe event is critical for enhancing disaster response efforts. However, the biggest challenge lies in acquiring labeled data for an ongoing (target) disaster to train supervised learning-based models, given that the labeling process is both time-consuming and costly. In this study, we address this challenge by proposing a new multimodal transfer learning framework for the real-time classification of SM images of the target disaster. The proposed framework is based on Contrastive Language-Image Pretraining (CLIP) model, jointly pretrained on a dataset of image-text pairs via contrastive learning. We propose two distinct methods to design our classification framework (1) Zero-Shot CLIP: it learns visual representations from images paired with natural language descriptions of classes. By utilizing the vision and language capabilities of CLIP, we extract meaningful features from unlabeled target disaster images and map them to semantically related textual class descriptions, enabling image classification without training on disaster-specific data. (2) Linear-Probe CLIP: it further enhances the performance and involves training a linear classifier on top of the pretrained CLIP model’s features, specifically tailored to the disaster image classification task. By optimizing the linear-probe classifier, we improve the model’s ability to discriminate between different classes and achieve higher performance without the need for labeled data of the target disaster. Both methods are evaluated on a benchmark X (formerly Twitter) dataset comprising images of seven real-world disaster events. The experimental outcomes showcase the efficacy of the proposed methods, with Linear-Probe CLIP achieving a remarkable 7% improvement in average F1-score relative to the state-of-the-art methods.

Smartphone Camera for Angiographic Computer Vision in Vascular Medicine

Aim: This study aimed to develop a TensorFlow Lite algorithm for angiography classification and to deploy it on a basic mobile smartphone device, thereby verifying the proof of concept for creating a comprehensive end-to-end mobile computer vision application for vascular medicine. Materials and Methods: After ethical approval by the local ethics committee, we collected institutional and open source peripheral angiograms of lower limbs. The angiograms were labeled by a researcher with more than 10 years of experience in vascular surgery. The labeling included dividing the angiograms according to their anatomical pattern into the Global Limb Anatomic Staging System (GLASS). The model was developed using the open-source TensorFlow framework for general image classification and deployed as an Android application. Results: The model utilized 700 angiograms, distributed as follows within the femoropoliteal GLASS disease (fp) categories: fp0 – 187 images, fp1 – 136 images, fp2 – 128 images, fp3 – 97 images, fp4 – 152 images. The reference dataset included 372 non-angiographic images (not_angio). Consequently, the entire model included 1,072 images. After training and deployment, the model demonstrated the following performance: a mean accuracy of 0.72. The best self-reported accuracy per class was for fp0 0.72, fp4 0.83 and not_angio 1.0 classes. Conclusion: We discovered that a smartphone camera could be utilized for angiographic computer vision through end-to-end applications accessible to every healthcare professional. However, the predictive abilities of the model are limited and require improvement. The development of a robust angiographic computer vision smartphone application should incorporate an upload function, undergo validation through head-to-head human-machine comparisons, potentially include segmentation, and feature a prospective design with explicit consent for using collected data in the development of AI models.

A novel Fourier neural operator framework for classification of multi-sized images: Application to three dimensional digital porous media

Fourier neural operators (FNOs) are invariant with respect to the size of input images, and thus images with any size can be fed into FNO-based frameworks without any modification of network architectures, in contrast to traditional convolutional neural networks. Leveraging the advantage of FNOs, we propose a novel deep-learning framework for classifying images with varying sizes. Particularly, we simultaneously train the proposed network on multi-sized images. As a practical application, we consider the problem of predicting the label (e.g., permeability) of three-dimensional digital porous media. To construct the framework, an intuitive approach is to connect FNO layers to a classifier using adaptive max pooling. First, we show that this approach is only effective for porous media with fixed sizes, whereas it fails for porous media of varying sizes. To overcome this limitation, we introduce our approach: instead of using adaptive max pooling, we use static max pooling with the size of channel width of FNO layers. Since the channel width of the FNO layers is independent of the input image size, the introduced framework can handle multi-sized images during training. We show the effectiveness of the introduced framework and compare its performance with the intuitive approach through the example of the classification of three-dimensional digital porous media of varying sizes.

Multi-module attention-guided deep learning framework for precise gastrointestinal disease identification in endoscopic imagery

The automated classification of gastrointestinal endoscopy images holds immense importance in modern health care. It streamlines the diagnostic process by enabling faster and more accurate identification of gastrointestinal diseases. While the existing automated methods have demonstrated promising performance, there still remains a gap in consistently achieving high accuracy. This is due to reason that endoscopy images suffer from inter-class similarities and intra-class differences, which complicates the classification task. To address these problems, we propose a framework for endoscopy image classification. In general, the proposed framework comprises three essential modules. The first module is the Local-Global Convolutional Neural Network (LG-CNN) which aims to extract both local fine-grained features and captures global context, second module is the Endoscopy-Lesion Attention Module (ELA) that enables the framework to emphasize more crucial regions and filter out noises and other irreverent information. Finally, the last module, Gastrointestinal Endoscopy CNN (GE-CNN) leverages the above two modules in a effective way to classify the input image into various categories. We evaluate the performance of proposed framework on two publicly available challenging datasets, namely, Kvasir, and HyperKvasir. Based on the experimental results, we illustrate the efficacy of the proposed framework in effectively classifying endoscopy images.

LBCapsNet: a lightweight balanced capsule framework for image classification of porcelain fragments

The image classification task of porcelain fragments is of great significance for the digital preservation of cultural heritage. However, common issues are encountered in the image processing of porcelain fragments, including the low computation speed, decreased accuracy due to the uneven distribution of sample categories, and model instability. This study proposes a novel Capsule Network model, referred to as LBCapsNet, which is suitable for the extraction of features from images of porcelain artifacts fragments. A bottleneck-like channel transformation module denoted by ChannelTrans, which resides between the convolutional layer and the PrimaryCaps layer, was first designed. This module is used to reduce the computational complexity and enhance the processing speed when dealing with intricate porcelain images. The MF-R loss function was then proposed by incorporating focal loss into the original loss function. This allows to address the issue of imbalanced distribution of ceramic shard samples and reduce the classification errors, which leads to faster convergence with smoother trend. Finally, an adaptive dynamic routing mechanism is designed with a dynamic learning rate to enhance the overall stability of the classification process. The experimental results obtained on public datasets, such as MNIST, Fashion- MNIST, CIFAR10, FMD and DTD as well as porcelain fragments dataset, demonstrate that LBCapsNet achieves high classification accuracy with faster and more stable computation compared with existing methods. Furthermore, the ability of LBCapsNet to process special textures can provide technical support for the digital preservation and restoration of cultural heritage.

Polymorphic Clustering and Approximate Masking Framework for Fine-Grained Insect Image Classification

Insect diversity monitoring is crucial for biological pest control in agriculture and forestry. Modern monitoring of insect species relies heavily on fine-grained image classification models. Fine-grained image classification faces challenges such as small inter-class differences and large intra-class variances, which are even more pronounced in insect scenes where insect species often exhibit significant morphological differences across multiple life stages. To address these challenges, we introduce segmentation and clustering operations into the image classification task and design a novel network model training framework for fine-grained classification of insect images using multi-modality clustering and approximate mask methods, named PCAM-Frame. In the first stage of the framework, we adopt the Polymorphic Clustering Module, and segmentation and clustering operations are employed to distinguish various morphologies of insects at different life stages, allowing the model to differentiate between samples at different life stages during training. The second stage consists of a feature extraction network, called Basenet, which can be any mainstream network that performs well in fine-grained image classification tasks, aiming to provide pre-classification confidence for the next stage. In the third stage, we apply the Approximate Masking Module to mask the common attention regions of the most likely classes and continuously adjust the convergence direction of the model during training using a Deviation Loss function. We apply PCAM-Frame with multiple classification networks as the Basenet in the second stage and conduct extensive experiments on the Insecta dataset of iNaturalist 2017 and IP102 dataset, achieving improvements of 2.2% and 1.4%, respectively. Generalization experiments on other fine-grained image classification datasets such as CUB200-2011 and Stanford Dogs also demonstrate positive effects. These experiments validate the pertinence and effectiveness of our framework PCAM-Frame in fine-grained image classification tasks under complex conditions, particularly in insect scenes.

A novel graph-based multiple kernel learning framework for hyperspectral image classification

ABSTRACT Multiple kernel learning (MKL) is an efficient way to improve hyperspectral image classification with few training samples by integrating spectral and spatial features. Nonetheless, presenting a MKL algorithm to capture significant information and eliminate redundancy is indispensable. In this paper, a novel graph-based MKL method (GMKL) is introduced to enhance the performance of the MKL algorithm and achieve high discriminative features, resulting in maximum separability. To do so, an informative low-rank graph is determined via a new algorithm called block diagonal representation that takes into account both local and global data structures. Then, in lieu of a time-consuming search for finding the optimal combination of the basic kernels, unlike the prior works, an optimal projective direction is acquired using graph theory. Various basic kernels generated from superpixels and attribute profiles are projected to produce a discriminative combination of kernels. Eventually, the optimum kernel is given to the support vector machine classifier. Experiments have been conducted on two popular hyperspectral datasets (Indian Pines and Pavia University). For the datasets, the proposed algorithm gained 90.43% and 99.28% overall accuracy, respectively, using only 1% training samples, which is much better than the state-of-the-art MKL algorithm and deep learning-based approaches with lower computational time.

Action Recognition of Taekwondo Unit Actions Using Action Images Constructed with Time-Warped Motion Profiles.

Taekwondo has evolved from a traditional martial art into an official Olympic sport. This study introduces a novel action recognition model tailored for Taekwondo unit actions, utilizing joint-motion data acquired via wearable inertial measurement unit (IMU) sensors. The utilization of IMU sensor-measured motion data facilitates the capture of the intricate and rapid movements characteristic of Taekwondo techniques. The model, underpinned by a conventional convolutional neural network (CNN)-based image classification framework, synthesizes action images to represent individual Taekwondo unit actions. These action images are generated by mapping joint-motion profiles onto the RGB color space, thus encapsulating the motion dynamics of a single unit action within a solitary image. To further refine the representation of rapid movements within these images, a time-warping technique was applied, adjusting motion profiles in relation to the velocity of the action. The effectiveness of the proposed model was assessed using a dataset compiled from 40 Taekwondo experts, yielding remarkable outcomes: an accuracy of 0.998, a precision of 0.983, a recall of 0.982, and an F1 score of 0.982. These results underscore this time-warping technique's contribution to enhancing feature representation, as well as the proposed method's scalability and effectiveness in recognizing Taekwondo unit actions.

Introspective Deep Metric Learning.

This paper proposes an introspective deep metric learning (IDML) framework for uncertainty-aware comparisons of images. Conventional deep metric learning methods focus on learning a discriminative embedding to describe the semantic features of images, which ignore the existence of uncertainty in each image resulting from noise or semantic ambiguity. Training without awareness of these uncertainties causes the model to overfit the annotated labels during training and produce overconfident judgments during inference. Motivated by this, we argue that a good similarity model should consider the semantic discrepancies with awareness of the uncertainty to better deal with ambiguous images for more robust training. To achieve this, we propose to represent an image using not only a semantic embedding but also an accompanying uncertainty embedding, which describes the semantic characteristics and ambiguity of an image, respectively. We further propose an introspective similarity metric to make similarity judgments between images considering both their semantic differences and ambiguities. The gradient analysis of the proposed metric shows that it enables the model to learn at an adaptive and slower pace to deal with the uncertainty during training. Our framework attains state-of-the-art performance on the widely used CUB-200-2011, Cars196, and Stanford Online Products datasets for image retrieval. We further evaluate our framework for image classification on the ImageNet-1 K, CIFAR-10, and CIFAR-100 datasets, which shows that equipping existing data mixing methods with the proposed introspective metric consistently achieves better results (e.g., +0.44% for CutMix on ImageNet-1 K).

Transformer Driven Matching Selection Mechanism for Multi-Label Image Classification

Graph Matching has recently emerged as an attractive technique applied to various computer vision tasks. Graph Matching based multi-label image classification, in particular, treats each image as a bag of instances and reformulates the classification task as an instance-label matching selection problem, achieving state-of-the-art results on diverse benchmarks. However, the generalization and scalability of such learned model cannot be well guaranteed due to its manually predetermined graph structure and high-dimension embedding of dense connections between instances and labels. To address these limitations, in this work, we propose a novel Transformer Driven Matching Selection framework for Multi-Label Image Classification (C-TMS), where instance structural relationships, class-wise global dependencies, and the co-occurrence possibility of varying instance-label assignments are simultaneously taken into consideration in a unified and adaptive manner. Moreover, the parallelization capability of the Transformer enables efficient computation, making our model scalable to large-scale datasets. Specifically, we first represent instances and labels as nodes in the visual space and label space respectively, and then compute the hidden representation of each node in its individual space, by attending a self-attention strategy over its entire neighborhood. Subsequently, the cross-attention is adopted to excavate the correct assignments between instances and labels, and further interprets how classifying each label depends on the instances within an image and its interaction with other labels. Finally, an asymmetric focal loss is designed to optimize the instance-label correspondence, and read out image-level category confidences. Extensive experiments conducted on various multi-label image datasets demonstrate the superiority of our proposed method.

Automatic Identification and Severity Classification of Retinal Biomarkers in SD-OCT Using Dilated Depthwise Separable Convolution ResNet with SVM Classifier

Purpose Diagnosis of Uveitic Macular Edema (UME) using Spectral Domain OCT (SD-OCT) is a promising method for early detection and monitoring of sight-threatening visual impairment. Viewing multiple B-scans and identifying biomarkers is challenging and time-consuming for clinical practitioners. To overcome these challenges, this paper proposes an image classification hybrid framework for predicting the presence of biomarkers such as intraretinal cysts (IRC), hyperreflective foci (HRF), hard exudates (HE) and neurosensory detachment (NSD) in OCT B-scans along with their severity. Methods A dataset of 10880 B-scans from 85 Uveitic patients is collected and graded by two board-certified ophthalmologists for the presence of biomarkers. A novel image classification framework, Dilated Depthwise Separable Convolution ResNet (DDSC-RN) with SVM classifier, is developed to achieve network compression with a larger receptive field that captures both low and high-level features of the biomarkers without loss of classification accuracy. The severity level of each biomarker is predicted from the feature map, extracted by the proposed DDSC-RN network. Results The proposed hybrid model is evaluated using ground truth labels from the hospital. The deep learning model initially, identified the presence of biomarkers in B-scans. It achieved an overall accuracy of 98.64%, which is comparable to the performance of other state-of-the-art models, such as DRN-C-42 and ResNet-34. The SVM classifier then predicted the severity of each biomarker, achieving an overall accuracy of 89.3%. Conclusions A new hybrid model accurately identifies four retinal biomarkers on a tissue map and predicts their severity. The model outperforms other methods for identifying multiple biomarkers in complex OCT B-scans. This helps clinicians to screen multiple B-scans of UME more effectively, leading to better treatment outcomes.

CMAF: Cross-Modal Augmentation via Fusion for Underwater Acoustic Image Recognition

Underwater image recognition is crucial for underwater detection applications. Fish classification has been one of the emerging research areas in recent years. Existing image classification models usually classify data collected from terrestrial environments. However, existing image classification models trained with terrestrial data are unsuitable for underwater images, as identifying underwater data is challenging due to their incomplete and noisy features. To address this, we propose a cross-modal augmentation via fusion ( CMAF ) framework for acoustic-based fish image classification. Our approach involves separating the process into two branches: visual modality and sonar signal modality, where the latter provides a complementary character feature. We augment the visual modality, design an attention-based fusion module, and adopt a masking-based training strategy with a mask-based focal loss to improve the learning of local features and address the class imbalance problem. Our proposed method outperforms the state-of-the-art methods. Our source code is available at https://github.com/WilkinsYang/CMAF .

DDCNN-F: double decker convolutional neural network 'F' feature fusion as a medical image classification framework

Melanoma is a severe skin cancer that involves abnormal cell development. This study aims to provide a new feature fusion framework for melanoma classification that includes a novel ‘F’ Flag feature for early detection. This novel ‘F’ indicator efficiently distinguishes benign skin lesions from malignant ones known as melanoma. The article proposes an architecture that is built in a Double Decker Convolutional Neural Network called DDCNN future fusion. The network's deck one, known as a Convolutional Neural Network (CNN), finds difficult-to-classify hairy images using a confidence factor termed the intra-class variance score. These hirsute image samples are combined to form a Baseline Separated Channel (BSC). By eliminating hair and using data augmentation techniques, the BSC is ready for analysis. The network's second deck trains the pre-processed BSC and generates bottleneck features. The bottleneck features are merged with features generated from the ABCDE clinical bio indicators to promote classification accuracy. Different types of classifiers are fed to the resulting hybrid fused features with the novel 'F' Flag feature. The proposed system was trained using the ISIC 2019 and ISIC 2020 datasets to assess its performance. The empirical findings expose that the DDCNN feature fusion strategy for exposing malignant melanoma achieved a specificity of 98.4%, accuracy of 93.75%, precision of 98.56%, and Area Under Curve (AUC) value of 0.98. This study proposes a novel approach that can accurately identify and diagnose fatal skin cancer and outperform other state-of-the-art techniques, which is attributed to the DDCNN ‘F’ Feature fusion framework. Also, this research ascertained improvements in several classifiers when utilising the ‘F’ indicator, resulting in the highest specificity of + 7.34%.

S2HM2: A Spectral–Spatial Hierarchical Masked Modeling Framework for Self-Supervised Feature Learning and Classification of Large-Scale Hyperspectral Images

S²HM²: A Spectral–Spatial Hierarchical Masked Modeling Framework for Self-Supervised Feature Learning and Classification of Large-Scale Hyperspectral Images

CMAAC: Combining Multiattention and Asymmetric Convolution Global Learning Framework for Hyperspectral Image Classification

CMAAC: Combining Multiattention and Asymmetric Convolution Global Learning Framework for Hyperspectral Image Classification

PolSAR Image Classification Framework With POA Align and Cyclic Channel Attention

PolSAR Image Classification Framework With POA Align and Cyclic Channel Attention

Hyperspectral Image Classification Framework Based on Multichannel Graph Convolutional Networks and Class-Guided Attention Mechanism

Hyperspectral Image Classification Framework Based on Multichannel Graph Convolutional Networks and Class-Guided Attention Mechanism

Dual Discriminative Low-Rank Projection Learning for Robust Image Classification

Numerous methods have exploited projection learning to extract low-dimensional features for image classification. Some projection learning methods integrate low-rank matrix recovery into classification models to equip the projection subspace with discrimination and robustness against corruption. However, these methods cannot directly recover “clean” components from the new datum in a low-dimensional subspace. Additionally, they are sensitive to the selection of projection dimensions. To overcome these shortcomings, we propose a dual discriminative low-rank projection learning framework for robust image classification. Specifically, the proposed method learns a low-rank projection and a semi-orthogonal projection to recover “clean” components from the original data and simultaneously obtain a low-dimensional subspace. Thereafter, to preserve discriminative information in the low-dimensional subspace, an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,1</sub> -norm term is constructed by concentrating the projected intra-class samples around their adaptive class centroids. Regression-based terms are appended using the low-dimensional features extracted from the recovered clean data and the class centroids for more accurate classification. Experiments on five public databases with various corruptions demonstrate that the proposed method can robustly classify image data despite a small training sample sizes and gross corruption. The superiority of the proposed method is further verified on the large-scale PubFig83 database, on which it achieves an 87.58% classification accuracy.

A Dual-Attention Deep Discriminative Domain Generalization Model for Hyperspectral Image Classification

Recently, hyperspectral image classification has made great progress with the development of convolutional neural networks. However, due to the challenges of distribution shifts and data redundancies, the classification accuracy is low. Some existing domain adaptation methods try to mitigate the distribution shifts by training source samples and some labeled target samples. However, in practice, labeled target domain samples are difficult or even impossible to obtain. To solve the above challenges, we propose a novel dual-attention deep discriminative domain generalization framework (DAD3GM) for cross-scene hyperspectral image classification without training the labeled target samples. In DAD3GM, we mainly design two blocks: dual-attention feature learning (DAFL) and deep discriminative feature learning (DDFL). DAFL is designed to extract spatial features by multi-scale self-attention and extract spectral features by multi-head external attention. DDFL is further designed to extract deep discriminative features by contrastive regularization and class discrimination regularization. The combination of DAFL and DDFL can effectively reduce the computational time and improve the generalization performance of DAD3GM. The proposed model achieves 84.25%, 83.53%, and 80.63% overall accuracy on the public Houston, Pavia, and GID benchmarks, respectively. Compared with some classical and state-of-the-art methods, the proposed model achieves optimal results, which reveals its effectiveness and feasibility.