Fine-grained Attributes Research Articles

Fine-Grained Fault Diagnosis of Photovoltaic Systems Based on DMAD-GAN and IFD-FGIF

Abstract In the field of photovoltaic system monitoring, fault detection faces two critical challenges: data imbalance and fault diversity, as well as incomplete complex fault information. To address these issues, this paper proposes a Dual-Mechanism Anomaly Detection with
Generative Adversarial Network (DMAD-GAN) and an Integrated Fault Diagnosis with Fine-Grained Information Fusion (IFD-FGIF) method. DMAD-GAN utilizes GAN to integrate dual mechanisms for anomaly detection in photovoltaic datasets, with Coordinate-Space
Attention (CSA) enhancing the perception of subtle features and differences in photovoltaic panels.The anomaly scoring mechanism utilizes an improved loss function to compute anomaly scores, assessing the degree of anomaly for each sample. In the IFD-FGIF method, t-SNE is used to visualize features for fault pre-classification to determine the presence of new faults. A Fine-Grained Information Fusion Module (FGIFM) is designed, leveraging
ResNet50 to extract features from fault key areas and original images. This module integrates fine-grained features, original features, and fine-grained attributes. Fault attributes and categories are determined using an attribute classifier and Euclidean distance. If a new fault is identified during pre-classification, the network undergoes transfer learning to recognize and adapt to the new fault. The experimental results demonstrate that the proposed method outperforms other networks, achieving an anomaly detection accuracy of 95.86%. The fault fine-grained recognition accuracy is 95.62%. Particularly, there is a 4% improvement in fine-grained information fusion accuracy. Furthermore, unsupervised learning for new faults is successfully implemented.

FEditNet++: Few-Shot Editing of Latent Semantics in GAN Spaces With Correlated Attribute Disentanglement.

Generative Adversarial Networks have achieved significant advancements in generating and editing high-resolution images. However, most methods suffer from either requiring extensive labeled datasets or strong prior knowledge. It is also challenging for them to disentangle correlated attributes with few-shot data. In this paper, we propose FEditNet++, a GAN-based approach to explore latent semantics. It aims to enable attribute editing with limited labeled data and disentangle the correlated attributes. We propose a layer-wise feature contrastive objective, which takes into consideration content consistency and facilitates the invariance of the unrelated attributes before and after editing. Furthermore, we harness the knowledge from the pretrained discriminative model to prevent overfitting. In particular, to solve the entanglement problem between the correlated attributes from data and semantic latent correlation, we extend our model to jointly optimize multiple attributes and propose a novel decoupling loss and cross-assessment loss to disentangle them from both latent and image space. We further propose a novel-attribute disentanglement strategy to enable editing of novel attributes with unknown entanglements. Finally, we extend our model to accurately edit the fine-grained attributes. Qualitative and quantitative assessments demonstrate that our method outperforms state-of-the-art approaches across various datasets, including CelebA-HQ, RaFD, Danbooru2018 and LSUN Church.

SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing

Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics. Our code is available at https://bestzzhang.github.io/SGEdit.

Vision language model for interpretable and fine-grained detection of safety compliance in diverse workplaces

Workplace accidents due to personal protective equipment (PPE) non-compliance raise serious safety concerns and lead to legal liabilities, financial penalties, and reputational damage. While object detection models have shown the capability to address this issue by identifying safety gears, most existing models, such as YOLO, Faster R-CNN, and SSD, are limited in verifying the fine-grained attributes of PPE across diverse workplace scenarios. Vision language models (VLMs) are gaining traction for detection tasks by leveraging the synergy between visual and textual information, offering a promising solution to traditional object detection limitations in PPE recognition. Nonetheless, VLMs face challenges in consistently verifying PPE attributes due to the complexity and variability of workplace environments, requiring them to interpret context-specific language and visual cues simultaneously. We introduce Clip2Safety, an interpretable detection framework for diverse workplace safety compliance, which comprises four main modules: scene recognition, the visual prompt, safety gears detection, and fine-grained verification. The scene recognition identifies the current scenario to determine the necessary safety gear. The visual prompt formulates the specific visual prompts needed for the detection process. The safety gears detection identifies whether the required safety gear is being worn according to the specified scenario. Lastly, the fine-grained verification assesses whether the worn safety equipment meets the fine-grained attribute requirements. We conduct real-world case studies across six different scenarios. The results show that Clip2Safety not only demonstrates an accuracy improvement over state-of-the-art question-answering based VLMs but also achieves inference times 21× faster.

ThyFusion: A lightweight attribute enhancement module for thyroid nodule diagnosis using gradient and frequency-domain awareness

Accurate classification of thyroid nodules is a critical step in clinical diagnosis and plays a crucial role in guiding subsequent treatment planning. However, current deep learning methods lack the ability to effectively perceive the attributes of thyroid nodules. Specifically, convolutional neural network-based methods struggle to capture fine-grained attributes, such as echogenic foci, due to repeated downsampling operations. On the other hand, Transformer-based methods partition the image into blocks, which hinders the capture of continuous attributes such as margins, and require a large number of parameters. To overcome these limitations, this paper proposes a novel lightweight attribute enhancement module called ThyFusion. Firstly, a multi-scale Gaussian filtering attention module is developed to accurately capture fine-grained information. Secondly, a frequency-domain global regulation module is proposed to regulate global features and capture coarse-grained attributes. Finally, a cross domain mutual learning module is presented to fuse fine-grained and coarse-grained attributes. Extensive experiments were conducted to demonstrate the effectiveness of ThyFusion, and it was compared against 15 benchmark methods. ThyFusion achieves higher accuracy, F1 score, recall, and precision (87.18%, 87.12%, 87.36% and 87.10%). ThyFusion is a plug-and-play lightweight module that can be embedded behind any layer of a backbone network to enhance the accuracy of thyroid nodule classification. The advantages of ThyFusion in thyroid nodule classification can greatly streamline the process of automated diagnosis and improve the accuracy of thyroid diagnosis.

Imbalanced image classification algorithm based on fine-grained analysis

Fine-grained attribute analysis and data imbalance have always been research hotspots in the field of computer vision. Due to the complexity and diversity of fine-grained attribute images, traditional image classification methods have shortcomings in paying attention to fine-grained attributes of images and perform poorly when dealing with imbalanced data sets. To overcome these problems, this study proposes a fine-grained image threshold classification algorithm based on deep metric learning. By introducing a metric learning method, the focus on fine-grained attributes of images is enhanced. At the same time, by applying pairwise loss and proxy loss, the classification accuracy of the model is improved and the model convergence speed is accelerated. In order to deal with the problem of data imbalance, a classifier based on threshold analysis is designed. The classifier uses threshold analysis technology to achieve multi-level classification of fine-grained images, thereby improving the problem of low classification accuracy of a few categories in imbalanced data sets. Experimental results show that the proposed fine-grained image threshold classification algorithm based on deep metric learning is significantly superior to other methods in terms of classification accuracy.

A mutual reconstruction network model for few-shot classification of histological images: addressing interclass similarity and intraclass diversity.

The automated classification of histological images is crucial for the diagnosis of cancer. The limited availability of well-annotated datasets, especially for rare cancers, poses a significant challenge for deep learning methods due to the small number of relevant images. This has led to the development of few-shot learning approaches, which bear considerable clinical importance, as they are designed to overcome the challenges of data scarcity in deep learning for histological image classification. Traditional methods often ignore the challenges of intraclass diversity and interclass similarities in histological images. To address this, we propose a novel mutual reconstruction network model, aimed at meeting these challenges and improving the few-shot classification performance of histological images. The key to our approach is the extraction of subtle and discriminative features. We introduce a feature enhancement module (FEM) and a mutual reconstruction module to increase differences between classes while reducing variance within classes. First, we extract features of support and query images using a feature extractor. These features are then processed by the FEM, which uses a self-attention mechanism for self-reconstruction of features, enhancing the learning of detailed features. These enhanced features are then input into the mutual reconstruction module. This module uses enhanced support features to reconstruct enhanced query features and vice versa. The classification of query samples is based on weighted calculations of the distances between query features and reconstructed query features and between support features and reconstructed support features. We extensively evaluated our model using a specially created few-shot histological image dataset. The results showed that in a 5-way 10-shot setup, our model achieved an impressive accuracy of 92.09%. This is a 23.59% improvement in accuracy compared to the model-agnostic meta-learning (MAML) method, which does not focus on fine-grained attributes. In the more challenging, 5-way 1-shot setting, our model also performed well, demonstrating a 18.52% improvement over the ProtoNet, which does not address this challenge. Additional ablation studies indicated the effectiveness and complementary nature of each module and confirmed our method's ability to parse small differences between classes and large variations within classes in histological images. These findings strongly support the superiority of our proposed method in the few-shot classification of histological images. The mutual reconstruction network provides outstanding performance in the few-shot classification of histological images, successfully overcoming the challenges of similarities between classes and diversity within classes. This marks a significant advancement in the automated classification of histological images.

Novel item selection strategies for cognitive diagnostic computerized adaptive testing: A heuristic search framework.

The computerized adaptive form of cognitive diagnostic testing, CD-CAT, has gained increasing attention in the domain of personalized measurements for its ability to categorize individual mastery status of fine-grained attributes more accurately and efficiently through administering items tailored to one's ability progressively. How to select the next item based on previous response(s) is crucial for the success of CD-CAT. Previous item selection strategies for CD-CAT have often followed a greedy or semi-greedy approach, which makes it difficult to strike a balance between diagnostic performance and item bank utilization. To address this issue, this study takes a graph perspective and transforms the item selection problem in CD-CAT into a path-searching problem, in which paths refer to possible test construction and nodes refer to individual items. A heuristic function is defined to predict the prospect of a path, indicating how well the corresponding test can diagnose the current examinee. Two search mechanisms with different biases towards item exposure control are proposed to approximate the optimal path with the best prospect. The first unused item on the resulting path is selected as the next item. The above components compose a novel CD-CAT item selection framework based on heuristic search. Simulation studies are conducted under a variety of conditions regarding bank designs, bank-quality conditions, and testing scenarios. The results are compared with different types of classic item selection strategies in CD-CAT, showing that the proposed framework can enhance bank utilization at a smaller cost of diagnostic performance.

LFR-GAN: Local Feature Refinement based Generative Adversarial Network for Text-to-Image Generation

Text-to-image generation aims to generate images from text descriptions. Its main challenge lies in two aspects: (1) Semantic consistency, i.e., the generated images should be semantically consistent with the input text; and (2) Visual reality, i.e., the generated images should look like real images. To ensure text-image consistency, existing works mainly learn to establish the cross-modal representations via a text encoder and image encoder. However, due to the limited representation capability of the fixed-length embeddings and the flexibility of the free-form text descriptions, the learned text-to-image model is incapable of maintaining the semantic consistency between image local regions and fine-grained descriptions. As a result, the generated images sometimes miss some fine-grained attributes of the generated object, such as the color or shape of a part of the object. To address this issue, this paper proposes a Local Feature Refinement Based Generative Adversarial Network (LFR-GAN) , which first divides the text into some independent fine-grained attributes and generates an initial image, then refines the image details based on these attributes. The main contributions are three-fold: (1) An attribute modeling approach is proposed to model the fine-grained text descriptions by mapping them into representations of independent attributes, which provides more fine-grained details for image generation. (2) A local feature refinement approach is proposed to enable the generated image to form a complete reflection of the fine-grained attributes contained in the text description. (3) A multi-stage generation approach is proposed to realize the fine-grained manipulation of complex images progressively, which aims to improve the performance of the refinement and generate photo-realistic images. Extensive experiments on the CUB and Oxford102 datasets show the effectiveness of our LFR-GAN approach in both text-to-image generation and text-guided image manipulation tasks. Our LFR-GAN approach shows superior performance compared to the state-of-the-art methods. The codes will be released at https://github.com/PKU-ICST-MIPL/LFR-GAN_TOMM2023 .

Less Is More: Learning from Synthetic Data with Fine-Grained Attributes for Person Re-Identification

Person re-identification (ReID) plays an important role in applications such as public security and video surveillance. Recently, learning from synthetic data [ 9 ], which benefits from the popularity of the synthetic data engine, has attracted great attention from the public. However, existing datasets are limited in quantity, diversity, and realisticity, and cannot be efficiently used for the ReID problem. To address this challenge, we manually construct a large-scale person dataset named FineGPR with fine-grained attribute annotations. Moreover, aiming to fully exploit the potential of FineGPR and promote the efficient training from millions of synthetic data, we propose an attribute analysis pipeline called AOST based on the traditional machine learning algorithm, which dynamically learns attribute distribution in a real domain, then eliminates the gap between synthetic and real-world data and thus is freely deployed to new scenarios. Experiments conducted on benchmarks demonstrate that FineGPR with AOST outperforms (or is on par with) existing real and synthetic datasets, which suggests its feasibility for the ReID task and proves the proverbial less-is-more principle. Our synthetic FineGPR dataset is publicly available at https://github.com/JeremyXSC/FineGPR .

A novel self-boosting dual-branch model for pedestrian attribute recognition

Pedestrian attributes carry broad information of abstract and detailed annotations. Pedestrian attribute recognition (PAR) can generate high-level semantic feature maps and provide auxiliary information in tasks such as person retrieval and re-identification. Since pedestrian attributes have broad categories and appear in complex region combinations in video sequences, enhancing the feature representation of fine-grained attributes is the key to improving pedestrian attribute recognition. This paper proposes an end-to-end feature-enhanced multi-scale dual-branch model for multiple attributes recognition (FEMDAR) with a feature-enhanced block (FEB) and multi-scale modules. The FEB module fuses the contextual features of each level, stacking different rates of dilated convolutions to expand the receptive field of the feature expression effectively. Simultaneously, the feature-enhanced block module utilizes a dual-branch structure to incorporate enhanced feature representation. Moreover, a new residual module is designed with multi-scale modules to extract the inconsistency in various attribute scales. Hence, the proposed model can achieve robust feature representation that can flexibly adapt to multiple pedestrian attributes of different scales. We validate our model on three public large-scale pedestrian attribute datasets. The experimental results show that the FEMDAR model shows prominent advantages in instance-based measurements. Furthermore, the ablation study shows the effectiveness of the proposed FEB and multi-scale modules in feature presentation.

Image Measurement Attribute Extraction Method under Large Error Condition

In recent years, the rapid development of image processing based on deep learning has had a wide impact on industrial production and social life. In the field of information security, image content description as a kind of attribute has been used in authentication, access control and other technologies. In addition to describing what is in an image, fine-grained image content properties should also be able to express the relative size and position relationships between objects. Due to the strong fuzziness of the relative scale between objects in the image, it is not suitable for precise definition. This paper proposes a method to extract measurement attributes under large error to describe the relative size of two parts in an image. Based on image classification and object detection, the error range is estimated by statistical method, and a fuzzy set is constructed as the length unit. Then the contradictory critical points of measurement are used to determine the comparable scale range and similar regions. Finally, the measurement attribute of image content is given. Experiments show that the size description obtained by this method accords with people’s subjective feelings, and can effectively extract the fine-grained attributes of the image.

Extending exploratory diagnostic classification models: Inferring the effect of covariates.

Diagnostic models provide a statistical framework for designing formative assessments by classifying student knowledge profiles according to a collection of fine-grained attributes. The context and ecosystem in which students learn may play an important role in skill mastery, and it is therefore important to develop methods for incorporating student covariates into diagnostic models. Including covariates may provide researchers and practitioners with the ability to evaluate novel interventions or understand the role of background knowledge in attribute mastery. Existing research is designed to include covariates in confirmatory diagnostic models, which are also known as restricted latent class models. We propose new methods for including covariates in exploratory RLCMs that jointly infer the latent structure and evaluate the role of covariates on performance and skill mastery. We present a novel Bayesian formulation and report a Markov chain Monte Carlo algorithm using a Metropolis-within-Gibbs algorithm for approximating the model parameter posterior distribution. We report Monte Carlo simulation evidence regarding the accuracy of our new methods and present results from an application that examines the role of student background knowledge on the mastery of a probability data set.

Using Bayesian Networks for Cognitive Assessment of Student Understanding of Buoyancy: A Granular Hierarchy Model

ABSTRACT Cognitive diagnostic assessment based on Bayesian networks (BN) is developed in this paper to evaluate student understanding of the physical concept of buoyancy. we propose a three-order granular-hierarchy BN model which accounts for both fine-grained attributes and high-level proficiencies. Conditional independence in the BN structure is tested and utilized to validate the proposed model. The proficiency relationships are verified and the initial Q-matrix is refined. Then, an optimized granular hierarchy model is constructed based on the updated Q-matrix. All variants of the constructed models are evaluated on the basis of the prediction accuracy and the goodness-of-fit test. The experimental results demonstrate that the optimized granular-hierarchy model has the best prediction and model-fitting performance. In general, the BN method not only can provide more flexible modeling approach, but also can help validate or refine the proficiency model and the Q-matrix and this method has its unique advantage in cognitive diagnosis.

Using an integrated discourse analysis approach to analyze a group's collaborative argumentation

Collaborative argumentation is widely used in K-12 and higher education to foster students' argumentation skills, facilitate deep learning, and construct collective knowledge. Collaborative argumentation requires students to coordinate their social, cognitive, and metacognitive practices with peers through interactive discourses. Discourse analysis is a traditional analytic method that has been used to understand collaborative discourses. But most previous research has either integrated computational or statistical techniques to analyze frequencies of discourses or analyzed discourse from a qualitative perspective to demonstrate the microlevel, fine-grained attributes. There has been a lack of integrated discourse analysis method to holistically investigate and understand collaborative argumentation. To fill this gap, this case study used the scripted role strategy to support a group's collaborative argumentation and proposed an integrated discourse analysis approach to illustrate the temporal changes of the group's discourse moves, structures, and turn taking processes. The results showed that four participants had different discourse attributes. Based on the results, pedagogical and analytical implications are proposed to facilitate research and practice of collaborative argumentation.

High-Accuracy Clothing and Style Classification via Multi-Feature Fusion

In recent years, the online selection of virtual clothing styles has been used to explore and expand diversified personal aesthetics, and it is also an overall reform and challenge to the clothing industry. Under the condition of the existing clothing style categories, this paper puts forward a style classification method combining fine-grained and coarse-grained techniques. Furthermore, a new deep neural network is proposed, which can improve the robustness of recognition and avoid the interference of image background through the pan learning and the background learning of image features. In order to study the relationship between the fine-grained attributes of clothing and the whole style, firstly, the clothing types are learned to realize the pre-training of model parameters. Secondly, through the transfer learning of the first stage of the pre-training model parameters, the model parameters are fine-tuned to make them more suitable for identifying the coarse-grained style types. Finally, a network structure based on the dual attention mechanism is proposed to improve the accuracy of final identification by adding different attention mechanisms at different stages of the network to enhance the performance of network features. In the experiment, we collected 50,000 images of 10 clothing styles to train and evaluate the models. The results show that the proposed classification method can effectively distinguish clothing styles and types.

WYSIWYG: IoT Device Identification Based on WebUI Login Pages.

With the improvement of intelligence and interconnection, Internet of Things (IoT) devices tend to become more vulnerable and exposed to many threats. Device identification is the foundation of many cybersecurity operations, such as asset management, vulnerability reaction, and situational awareness, which are important for enhancing the security of IoT devices. The more information sources and the more angles of view we have, the more precise identification results we obtain. This study proposes a novel and alternative method for IoT device identification, which introduces commonly available WebUI login pages with distinctive characteristics specific to vendors as the data source and uses an ensemble learning model based on a combination of Convolutional Neural Networks (CNN) and Deep Neural Networks (DNN) for device vendor identification and develops an Optical Character Recognition (OCR) based method for device type and model identification. The experimental results show that the ensemble learning model can achieve 99.1% accuracy and 99.5% F1-Score in the determination of whether a device is from a vendor that appeared in the training dataset, and if the answer is positive, 98% accuracy and 98.3% F1-Score in identifying which vendor it is from. The OCR-based method can identify fine-grained attributes of the device and achieve an accuracy of 99.46% in device model identification, which is higher than the results of the Shodan cyber search engine by a considerable margin of 11.39%.

Attribute-aware heterogeneous graph network for fashion compatibility prediction

Fashion compatibility prediction aims to provide a compatibility score for a set of fashion combinations, making an effort to meet people’s needs for clothing matching in daily life. One difficulty of this problem is that whether the fashion items are compatible depends on a variety of attribute factors, such as design style, material, color. Existing works mostly use the attributes as the item feature representations or integrate attribute features into the compatibility modeling process as auxiliary information in a simple way. These works ignore the fine-grained attributes that can play a role as bridges in establishing the potential compatibility relationship between fashion items. To better take advantage of fashion attributes information, we propose an Attribute-aware Heterogeneous Graph Network (AHGN), which combines the fashion items and their attributes as a heterogeneous graph to integrate the heterogeneous information in the fashion compatibility problem. While fusing the fine-grained attribute information into the item nodes representation, the attributes act as bridges to transfer the characteristics of potential compatible fashion items into the target item node. Previous methods of calculating outfit score by item pairs without considering the effect of outfit compositions. An outfit-based item pairs scoring method is proposed. The feature representation of item pairs is adjusted by integrating the overall outfit information to consider the influence of the outfit environment on the item pairs. We conduct experiments on three tasks (fill-in-the-blank, compatibility prediction, and fashion item retrieval) in two real-world fashion datasets: Fashion32 and IQON3000. Experimental results demonstrate the effectiveness of our proposed method.

AI Ekphrasis: Multi-Modal Learning with Foundation Models for Fine-Grained Poetry Retrieval

Artificial intelligence research in natural language processing in the context of poetry struggles with the recognition of holistic content such as poetic symbolism, metaphor, and other fine-grained attributes. Given these challenges, multi-modal image–poetry reasoning and retrieval remain largely unexplored. Our recent accessibility study indicates that poetry is an effective medium to convey visual artwork attributes for improved artwork appreciation of people with visual impairments. We, therefore, introduce a deep learning approach for the automatic retrieval of poetry suitable to the input images. The recent state-of-the-art CLIP provides a way for multi-modal visual and text features matched using cosine similarity. However, it lacks shared cross-modality attention features to model fine-grained relationships. The proposed approach in this work takes advantage of strong pre-training of the CLIP model and overcomes its limitations by introducing shared attention parameters to better model the fine-grained relationship between both modalities. We test and compare our proposed approach using the expertly annotated MiltiM-Poem dataset, which is considered the largest public image–poetry pair dataset for English poetry. The proposed approach aims to solve the problems of image-based attribute recognition and automatic retrieval for fine-grained poetic verses. The test results reflect that the shared attention parameters alleviate fine-grained attribute recognition, and the proposed approach is a significant step towards automatic multi-modal retrieval for improved artwork appreciation of people with visual impairments.

Interactive Visual Exploration of Longitudinal Historical Career Mobility Data.

The increased availability of quantitative historical datasets has provided new research opportunities for multiple disciplines in social science. In this article, we work closely with the constructors of a new dataset, CGED-Q (China Government Employee Database-Qing), that records the career trajectories of over 340,000 government officials in the Qing bureaucracy in China from 1760 to 1912. We use these data to study career mobility from a historical perspective and understand social mobility and inequality. However, existing statistical approaches are inadequate for analyzing career mobility in this historical dataset with its fine-grained attributes and long time span, since they are mostly hypothesis-driven and require substantial effort. We propose CareerLens, an interactive visual analytics system for assisting experts in exploring, understanding, and reasoning from historical career data. With CareerLens, experts examine mobility patterns in three levels-of-detail, namely, the macro-level providing a summary of overall mobility, the meso-level extracting latent group mobility patterns, and the micro-level revealing social relationships of individuals. We demonstrate the effectiveness and usability of CareerLens through two case studies and receive encouraging feedback from follow-up interviews with domain experts.