Multi-label Image Research Articles

MHA-DGCLN: multi-head attention-driven dynamic graph convolutional lightweight network for multi-label image classification of kitchen waste

MHA-DGCLN: multi-head attention-driven dynamic graph convolutional lightweight network for multi-label image classification of kitchen waste

A multi-class fundus disease classification system based on an adaptive scale discriminator and hybrid loss

Fundus images are crucial in the observation and detection of ophthalmic diseases. However, detecting multiple ophthalmic diseases from fundus images using deep learning techniques is a complex and challenging task One challenge is the complexity of fundus disease structures, which leads to low detection accuracy. Another challenge is the class imbalance problem common in multi-label image classification, which increases the difficulty of algorithm training and evaluation. To address these issues, this study leverages deep learning to propose an ophthalmic disease classification system. We first employ ResNet50 as the backbone network to extract image features, and then use our designed multi-dimensional attention module and adaptive scale discriminator to enhance the network's ability to detect disease features. During training, we innovatively propose a hybrid loss function method to improve the detection capability on imbalanced data. Finally, we conducted experiments on the ODRI-5K dataset with the proposed classification system. In the test set, our method achieved an AUC of 98.53 and an F1-score of 89.73. This result fully demonstrates the excellent disease classification capability of our method. In summary, the multi-label fundus image disease classification system we proposed exhibits outstanding recognition capability, providing an effective solution for the diagnosis of multi-label fundus image diseases.

Multi-Label Image Recognition Based on Attention and Multi-Scale Dynamic Graph Convolutional Network

Multi-Label Image Recognition Based on Attention and Multi-Scale Dynamic Graph Convolutional Network

Orbital angular momentum superimposed mode recognition based on multi-label image classification

Orbital angular momentum (OAM) multiplexing technology has great potential in high capacity optical communication. OAM superimposed mode can extend communication channels and thus enhance the capacity, and accurate recognition of multi-OAM superimposed mode at the receiver is very crucial. However, traditional methods are inefficient and complex for the recognition task. Machine learning and deep learning can offer fast, accurate and adaptable recognition, but they also face challenges. At present, the OAM mode recognition mainly focus on single OAM mode and ±l superimposed dual-OAM mode, while few researches on multi-OAM superimposed mode, due to the limitations of single-object image classification techniques and the diversity of features to recognize. To this end, we develop a recognition method combined with multi-label image classification to accurately recognize multi-OAM superimposed mode vortex beams. Firstly, we create datasets of intensity distribution map of three-OAM and four-OAM superimposed mode vortex beams based on numerical simulations and experimental acqusitions. Then we design a progressive channel-spatial attention (PCSA) model, which incorporates a progressive training strategy and two weighted attention modules. For the numerical simulation datasets, our model achieves the highest average recognition accuracy of 94.9% and 91.2% for three-OAM and four-OAM superimposed mode vortex beams with different transmission distances and noise strengths respectively. The highest experimental average recognition accuracy for three-OAM superimposed mode achieves 92.7%, which agrees with the numerical result very well. Furthermore, our model significantly outperforms in most metrics compared with ConvNeXt, and all experiments are within the affordable range of computational cost.

Dual-stream multi-label image classification model enhanced by feature reconstruction

Dual-stream multi-label image classification model enhanced by feature reconstruction

Advanced Multi-Label Fire Scene Image Classification via BiFormer, Domain-Adversarial Network and GCN

Detecting wildfires presents significant challenges due to the presence of various potential targets in fire imagery, such as smoke, vehicles, and people. To address these challenges, we propose a novel multi-label classification model based on BiFormer’s feature extraction method, which constructs sparse region-indexing relations and performs feature extraction only in key regions, thereby facilitating more effective capture of flame characteristics. Additionally, we introduce a feature screening method based on a domain-adversarial neural network (DANN) to minimize misclassification by accurately determining feature domains. Furthermore, a feature discrimination method utilizing a Graph Convolutional Network (GCN) is proposed, enabling the model to capture label correlations more effectively and improve performance by constructing a label correlation matrix. This model enhances cross-domain generalization capability and improves recognition performance in fire scenarios. In the experimental phase, we developed a comprehensive dataset by integrating multiple fire-related public datasets, and conducted detailed comparison and ablation experiments. Results from the tenfold cross-validation demonstrate that the proposed model significantly improves recognition of multi-labeled images in fire scenarios. Compared with the baseline model, the mAP increased by 4.426%, CP by 4.14% and CF1 by 7.04%.

Three-color single-molecule localization microscopy in chromatin.

Super-resolution microscopy has revolutionized our ability to visualize structures below the diffraction limit of conventional optical microscopy and is particularly useful for investigating complex biological targets like chromatin. Chromatin exhibits a hierarchical organization with structural compartments and domains at different length scales, from nanometers to micrometers. Single molecule localization microscopy (SMLM) methods, such as STORM, are essential for studying chromatin at the supra-nucleosome level due to their ability to target epigenetic marks that determine chromatin organization. Multi-label imaging of chromatin is necessary to unpack its structural complexity. However, these efforts are challenged by the high-density nuclear environment, which can affect antibody binding affinities, diffusivity and non-specific interactions. Optimizing buffer conditions, fluorophore stability, and antibody specificity is crucial for achieving effective antibody conjugates. Here, we demonstrate a sequential immunolabeling protocol that reliably enables three-label studies within the dense nuclear environment. This protocol couples multiplexed localization datasets with a robust analysis algorithm, which utilizes localizations from one target as seed points for distance, density and multi-label joint affinity measurements to explore complex organization of all three targets. Applying this multi-plexed algorithm to analyze distance and joint density reveals that heterochromatin and euchromatin are not-distinct territories, but that localization of transcription and euchromatin couple with the periphery of heterochromatic clusters. This work is a crucial step in molecular imaging of the dense nuclear environment as multi-label capacity enables for investigation of complex multi-component systems like chromatin with enhanced accuracy.

A semantic guidance-based fusion network for multi-label image classification

Multi-label image classification (MLIC), a fundamental task assigning multiple labels to each image, has been seen notable progress in recent years. Considering simultaneous appearances of objects in the physical world, modeling object correlations is crucial for enhancing classification accuracy. This involves accounting for spatial image feature correlation and label semantic correlation. However, existing methods struggle to establish these correlations due to complex spatial location and label semantic relationships. On the other hand, regarding the fusion of image feature relevance and label semantic relevance, existing methods typically learn a semantic representation in the final CNN layer to combine spatial and label semantic correlations. However, different CNN layers capture features at diverse scales and possess distinct discriminative abilities. To address these issues, in this paper we introduce the Semantic Guidance-Based Fusion Network (SGFN) for MLIC. To model spatial image feature correlation, we leverage the advanced TResNet architecture as the backbone network and employ the Feature Aggregation Module for capturing global spatial correlation. For label semantic correlation, we establish both local and global semantic correlation. We further enrich model features by learning semantic representations across multiple convolutional layers. Our method outperforms current state-of-the-art techniques on PASCAL VOC (2007, 2012) and MS-COCO datasets.

A consistency-aware deep capsule network for hierarchical multi-label image classification

Hierarchical classification is a significant challenge in computer vision due to the logical order and interconnectedness of multiple labels. This paper presents HD-CapsNet, a novel neural network architecture based on deep capsule networks, specifically designed for hierarchical multi-label classification(HMC). By incorporating a tree-like hierarchical structure, HD-CapsNet is designed to leverage the inherent ontological order within the hierarchical label tree, thereby ensuring classification consistency across different levels. Additionally, we introduce a specialized loss function that promotes accurate hierarchical relationships while penalizing inconsistencies. This not only enhances classification performance but also strengthens the network’s robustness. We rigorously evaluate HD-CapsNet’s efficacy by benchmarking it against existing HMC methods across six diverse datasets: Fashion-MNIST, Marine-Tree, CIFAR-10, CIFAR-100, Caltech-UCSD Birds-200-2011, and Stanford Cars. Our results conclusively demonstrate that HD-CapsNet excels in learning hierarchical relationships and significantly outperforms the competition in various image classification tasks. Our implementation is available at https://github.com/tasrif-khondaker/HD-CapsNet.

Identification of multimodal vortex optical orbital angular momentum in multimode fiber speckle patterns

Orbital angular momentum (OAM) multiplexing technology is a highly promising approach that can significantly increase the data capacity of optical communication systems. However, traditional optical communication systems are prone to atmospheric disturbances such as turbulence, which can degrade transmission quality. To mitigate this issue, multimode fibers (MMFs) have been introduced to reduce the impact of turbulence on signals. Moreover, previous studies have primarily focused on the identification of single vortex beams, facing challenges in accurately recognizing multiplexed modes. To address this challenge, this chapter proposes a multi-label image classification optimization algorithm based on transfer learning. By utilizing the pre-trained MobileNet V2 model as a feature extractor, this network structure can accurately identify 8-bit, 16-bit, and 24-bit multiplexed OAM from speckle patterns in multimode fibers, even with small sample datasets, achieving classification accuracies exceeding 95%. This method overcomes the limitations of traditional optical communication systems that are susceptible to atmospheric disturbances, providing new possibilities for long-distance transmission and increased data capacity in optical communication systems.

Multi-label recognition in open driving scenarios based on bipartite-driven superimposed dynamic graph

The multi-label image recognition task is widely prevalent in real-world scenarios. Overcoming the issue of overlapping and densely packed objects in complex scenes is crucial. For instance, in traffic scenarios, there are overlaps and close proximity among pedestrians, various types of vehicles, and signage. However, a primary obstacle in leveraging label relationships to enhance image classification lies in effectively integrating label semantic topology information with the image data itself. In this paper, we propose a novel framework, the Bipartite-driven Superimposed Dynamic Graph Convolutional Network (Bi-SDNet), augmented with Mapping Alignment Module (MAM) and Semantic Decoupling Module(SDM). Our approach initially decomposes input features into representations capable of discerning category label semantics at multiple scales, facilitated by MAM and SDM modules. Furthermore, through the meticulously designed Superimposed Dynamic Graph, we adeptly capture content-aware category relationships for each image, effectively modeling the relationships between these representations for the final recognition task. We conducted extensive experiments on publicly available benchmark datasets and the traffic scene dataset WZ-traffic. The model achieved an impressive 87.5% mean average precision (mAP) on the MS-COCO dataset and a commendable 91% mAP on the WZ-traffic dataset. Our research introduces novel techniques and significant breakthroughs in this field, furnishing powerful tools for enhancing model performance.

A Lightweight Multi-Label Classification Method for Urban Green Space in High-Resolution Remote Sensing Imagery

Urban green spaces are an indispensable part of the ecology of cities, serving as the city’s “purifier” and playing a crucial role in promoting sustainable urban development. Therefore, the refined classification of urban green spaces is an important task in urban planning and management. Traditional methods for the refined classification of urban green spaces heavily rely on expert knowledge, often requiring substantial time and cost. Hence, our study presents a multi-label image classification model based on MobileViT. This model integrates the Triplet Attention module, along with the LSTM module, to enhance its label prediction capabilities while maintaining its lightweight characteristic for standalone operation on mobile devices. Trial outcomes in our UGS dataset in this study demonstrate that the approach we used outperforms the baseline by 1.64%, 3.25%, 3.67%, and 2.71% in mAP,F1,precision, and recall, respectively. This indicates that the model can uncover the latent dependencies among labels to enhance the multi-label image classification device’s performance. This study provides a practical solution for the intelligent and detailed classification of urban green spaces, which holds significant importance for the management and planning of urban green spaces.

Multi-label image classification using adaptive graph convolutional networks: From a single domain to multiple domains

This paper proposes an adaptive graph-based approach for multi-label image classification. Graph-based methods have been largely exploited in the field of multi-label classification, given their ability to model label correlations. Specifically, their effectiveness has been proven not only when considering a single domain but also when taking into account multiple domains. However, the topology of the used graph is not optimal as it is pre-defined heuristically. In addition, consecutive Graph Convolutional Network (GCN) aggregations tend to destroy the feature similarity. To overcome these issues, an architecture for learning the graph connectivity in an end-to-end fashion is introduced. This is done by integrating an attention-based mechanism and a similarity-preserving strategy. The proposed framework is then extended to multiple domains using an adversarial training scheme. Numerous experiments are reported on well-known single-domain and multi-domain benchmarks. The results demonstrate that our approach achieves competitive results in terms of mean Average Precision (mAP) and model size as compared to the state-of-the-art. The code will be made publicly available.

ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset

Automated medical image analysis systems often require large amounts of training data with high quality labels, which are difficult and time consuming to generate. This paper introduces Radiology Object in COntext version 2 (ROCOv2), a multimodal dataset consisting of radiological images and associated medical concepts and captions extracted from the PMC Open Access subset. It is an updated version of the ROCO dataset published in 2018, and adds 35,705 new images added to PMC since 2018. It further provides manually curated concepts for imaging modalities with additional anatomical and directional concepts for X-rays. The dataset consists of 79,789 images and has been used, with minor modifications, in the concept detection and caption prediction tasks of ImageCLEFmedical Caption 2023. The dataset is suitable for training image annotation models based on image-caption pairs, or for multi-label image classification using Unified Medical Language System (UMLS) concepts provided with each image. In addition, it can serve for pre-training of medical domain models, and evaluation of deep learning models for multi-task learning.

A multi-label dataset and its evaluation for automated scoring system for cleanliness assessment in video capsule endoscopy.

An automated scoring system for cleanliness assessment during video capsule endoscopy (VCE) is presently lacking. The present study focused on developing an approach to automatically assess the cleanliness in VCE frames as per the latest scoring i.e., Korea-Canada (KODA). Initially, an easy-to-use mobile application called artificial intelligence-KODA (AI-KODA) score was developed to collect a multi-label image dataset of twenty-eight patient capsule videos. Three readers (gastroenterology fellows), who had been trained in reading VCE, rated this dataset in a duplicate manner. The labels were saved automatically in real-time. Inter-rater and intra-rater reliability were checked. The developed dataset was then randomly split into train:validate:test ratio of 70:20:10 and 60:20:20. It was followed by a comprehensive benchmarking and evaluation of three multi-label classification tasks using ten machine learning and two deep learning algorithms. Reliability estimation was found to be overall good among the three readers. Overall, random forest classifier achieved the best evaluation metrics, followed by Adaboost, KNeighbours, and Gaussian naive bayes in the machine learning-based classification tasks. Deep learning algorithms outperformed the machine learning-based classification tasks for only VM labels. Thorough analysis indicates that the proposed approach has the potential to save time in cleanliness assessment and is user-friendly for research and clinical use. Further research is required for the improvement of intra-rater reliability of KODA, and the development of automated multi-task classification in this field.

Cross-modality semantic guidance for multi-label image classification

Multi-label image classification aims to predict a set of labels that are present in an image. The key challenge of multi-label image classification lies in two aspects: modeling label correlations and utilizing spatial information. However, the existing approaches mainly calculate the correlation between labels according to co-occurrence among them. While the result is easily affected by the label noise and occasional co-occurrences. In addition, some works try to model the correlation between labels and spatial features, but the correlation among labels is not fully considered to model the spatial relationships among features. To address the above issues, we propose a novel cross-modality semantic guidance-based framework for multi-label image classification, namely CMSG. First, we design a semantic-guided attention (SGA) module, which applies the label correlation matrix to guide the learning of class-specific features, which implicitly models semantic correlations among labels. Second, we design a spatial-aware attention (SAA) module to extract high-level semantic-aware spatial features based on class-specific features obtained from the SGA module. The experiments carried out on three benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art algorithms on multi-label image classification.

Multi-labeled image data-based generative topology optimization of primary mirror with conditional designable generative adversarial network and reinforcement learning

In this study, topology optimization based on multi-labeled image data was conducted for a multi-objective primary mirror to produce novel designs with varying design variables. The primary mirror utilized in this application possessed a delicate structure where both the wavefront error and weight were subject to optimization. However, it was observed that the wavefront error and weight did not exhibit an inverse relationship, necessitating the development of a new multi-objective optimization approach for the primary mirror. Initially, 93 data points were gathered using the finite element method and categorized based on their wavefront error and weight. Topology optimization and the generation of novel designs were accomplished through iterative utilization of both the conditional designable generative adversarial network (CDGAN) and reinforcement learning (RL). To address the limitations inherent in each model and to ensure the effective implementation of the primary mirror, the structure of the CDGAN + RL model underwent modifications and optimizations employing multi-labeled images and considerations of boundary conditions. The application of CDGAN + RL successfully yielded multiple design solutions for unseen-optimized primary mirrors contingent upon the wavefront error and weight. Three-dimensional design variables (rib thickness, face-sheet thickness, cutting depth, and double arch) were optimized and validated based on the labels of the image data and the corresponding generated designs, revealing a minimum error rate of 1.73% and a maximum error rate of 9.37%. Comparative analysis against the initial design demonstrated performance enhancements of 41.84% and 5.41% for the wavefront error and weight, respectively.

Graph relationship-driven label coded mapping and compensation for multi-label textile fiber recognition

Recently, neural network-based methods for multi-label textile fiber recognition have achieved considerable success. However, a significant limitation of most current approaches is their disregard for the valuable dependencies that exist among different fiber categories. The universal multi-label image recognition methods that consider label relationships often fall short when applied to the challenge posed by the mixture of fibers in textile images. And these relationship modeling manners have not yet been used in the field of multi-label textile fiber recognition. In this work, a graph relationship-driven method is proposed for the recognition of multi-label textile fibers. Based on the graph attention network, the proposed method introduces global relation graph, label coded mapping, and label compensation to equip the feature learning backbone with the relationship modeling ability. First, feature learning backbone extracts the semantic features from images, and global relation graph mines shallow representations of global label relationships from label embeddings. Then, label coded mapping combines these semantic features and global features to model joint relationships. The obtained joint representations constitute the label code, which is used to map the fiber class indices. Finally, label compensation further extracts the deep representations of global label relationships and utilizes them to compensate the fiber class indices to obtain final prediction scores. By employing experiments on a textile fiber dataset and a rearranged dataset, the effectiveness and superiority of the proposed method are validated. Further experiments on PASCAL VOC 2007 showcase the potential applications of the proposed method beyond textile fiber recognition.

Artificial Intelligence Assistance for the Measurement of Full Alignment Parameters in Whole-Spine Lateral Radiographs

BackgroundMeasuring spinal alignment with radiological parameters is essential in patients with spinal conditions likely to be treated surgically. These evaluations are not usually included in the radiological report. As a result, spinal surgeons commonly perform the measurement, which is time-consuming and subject to errors. We aim to develop a fully automated artificial intelligence tool to assist in measuring alignment parameters in whole-spine lateral radiograph (WSL X-rays). Material and MethodsWe developed a tool called Vertebrai that automatically calculates the global spinal parameters (GSP): Pelvic incidence (PI), sacral slope (SS), pelvic tilt (PT), L1-L4 angle, L4-S1 lumbo-pelvic angle, T1 pelvic angle (TPA), sagittal vertical axis (SVA), cervical lordosis (CL), C1-C2 lordosis, lumbar lordosis (LL), mid-thoracic kyphosis (MTK), proximal thoracic kyphosis (PTK), global thoracic kyphosis (GTK), T1 slope (T1S), C2-C7 plummet (cSVA), spino-sacral angle (SSA), C7 tilt (C7T), global tilt (GT), spinopelvic tilt (T1SPi) and hip odontoid axis (OD-HA). We assessed human-AI interaction instead of AI performance alone. We compared the time to measure GSP and inter-rater agreement with and without AI assistance. Two institutional datasets were created with 2267 multilabel images for classification, and 784 WSL X-rays with reference standard landmark labeled by spinal surgeons. ResultsVertebrai significantly reduced the measurement time comparing spine surgeons with AI assistance and the AI algorithm alone, without human intervention (3 minutes vs 0.26 minutes; p < 0.05). Vertebrai achieved an average accuracy of 83% in detecting abnormal alignment values, with the SS parameter exhibiting the lowest accuracy at 61.5% and T1SPi demonstrating the highest accuracy at 100%. Intra-class correlation analysis revealed a high level of correlation and consistency in the global alignment parameters. ConclusionsVertebrai's measurements can accurately detect alignment parameters, making it a promising tool for measuring GSP automatically.

A multi-label image classification method combining multi-stage image semantic information and label relevance

A multi-label image classification method combining multi-stage image semantic information and label relevance