Part Annotations Research Articles

Enhancing long-term person re-identification using global, local body part, and head streams

This work addresses the task of long-term person re-identification. Typically, person re-identification assumes that people do not change their clothes, which limits its applications to short-term scenarios. To overcome this limitation, we investigate long-term person re-identification, which considers both clothes-changing and clothes-consistent scenarios. In this paper, we propose a novel framework that effectively learns and utilizes both global and local information. The proposed framework consists of three streams: global, local body part, and head streams. The global and head streams encode identity-relevant information from an entire image and a cropped image of the head region, respectively. Both streams encode the most distinct, less distinct, and average features using the combinations of adversarial erasing, max pooling, and average pooling. The local body part stream extracts identity-related information for each body part, allowing it to be compared with the same body part from another image. Since body part annotations are not available in re-identification datasets, pseudo-labels are generated using clustering. These labels are then utilized to train a body part segmentation head in the local body part stream. The proposed framework is trained by backpropagating the weighted summation of the identity classification loss, the pair-based loss, and the pseudo body part segmentation loss. To demonstrate the effectiveness of the proposed method, we conducted experiments on three publicly available datasets (Celeb-reID, PRCC, and VC-Clothes). The experimental results demonstrate that the proposed method outperforms the previous state-of-the-art method.

Parallel developmental changes in children’s production and recognition of line drawings of visual concepts

Childhood is marked by the rapid accumulation of knowledge and the prolific production of drawings. We conducted a systematic study of how children create and recognize line drawings of visual concepts. We recruited 2-10-year-olds to draw 48 categories via a kiosk at a children’s museum, resulting in >37K drawings. We analyze changes in the category-diagnostic information in these drawings using vision algorithms and annotations of object parts. We find developmental gains in children’s inclusion of category-diagnostic information that are not reducible to variation in visuomotor control or effort. Moreover, even unrecognizable drawings contain information about the animacy and size of the category children tried to draw. Using guessing games at the same kiosk, we find that children improve across childhood at recognizing each other’s line drawings. This work leverages vision algorithms to characterize developmental changes in children’s drawings and suggests that these changes reflect refinements in children’s internal representations.

PartLabeling: A Label Management Framework in 3D Space

In this work, we focus on the label layout problem: specifying the positions of overlaid virtual annotations in Virtual/Augmented Reality scenarios. Designing a layout of labels that does not violate domain-specific design requirements, while at the same time satisfying aesthetic and functional principles of good design, can be a daunting task even for skilled visual designers. Presenting the annotations in 3D object space instead of projection space, allows for the preservation of spatial and depth cues. This results in stable layouts in dynamic environments, since the annotations are anchored in 3D space. In this paper we make two major contributions. First, we propose a technique for managing the layout and rendering of annotations in Virtual/Augmented Reality scenarios by manipulating the annotations directly in 3D space. For this, we make use of Artificial Potential Fields and use 3D geometric constraints to adapt them in 3D space. Second, we introduce PartLabeling: an open source platform in the form of a web application that acts as a much-needed generic framework allowing to easily add labeling algorithms and 3D models. This serves as a catalyst for researchers in this field to make their algorithms and implementations publicly available, as well as ensure research reproducibility. The PartLabeling framework relies on a dataset that we generate as a subset of the original PartNet dataset [17] consisting of models suitable for the label management task. The dataset consists of 1,000 3D models with part annotations.

Supervised spectral feature learning for fine-grained classification in small data set

Fine-grained image classification is a challenging task due to the small inter-class variance, the large intra-class difference, and the small training data. Traditional methods typically rely on large-scale training samples with annotated part annotations, making them costly and severely limiting their application area. In this paper, we propose an effective and weakly supervised fine-grained classification framework. In this framework, a discriminative class-specific spectral feature is learned by intra-class spectral coupling and inter-class spectral decoupling under the weak supervision of image-level category labels, and then the new input images are classified based on the learned class-specific spectral feature. Different from existing strong supervised methods, the proposed technique creatively combines weak supervision of the image-level category labels with unsupervised spectral graph decomposition, not relying on large-scale training samples with dense part annotations, which are heavily labor-consuming. The performance of the proposed methods has been verified on four kinds of typical datasets: the JAFFE dataset, the Yale database, the UCI-CMU face database, and the neural foramina dataset. The satisfactory classification results have been achieved by the proposed method in expression recognition on the JAFFE dataset (with a mean accuracy of 95.31%), face recognition on the Yale database (with a mean accuracy of 98.79%), object recognition on the UCI-CMU face database (with a mean accuracy of 96.96%), and disease grading on the neural foramina dataset (with a mean accuracy of 92.09%). Compared with most state-of-the-art methods, the proposed method has superior classification performance in the small data set.

An efficient fine-grained vehicle recognition method based on part-level feature optimization

This paper presents an effective method for strengthening the discriminative ability of high-level deep features by enhancing and aggregating discriminative part-level features for the fine-grained vehicle recognition task. In general, the task of visual recognition concentrates more on the visual differences at the object level. However, for fine-grained object recognition, the visual differences between target objects typically exist in local discriminative areas, so it is more concerned about extracting fine-grained features from these part regions. In this context, we propose solving this issue with a novel feature extraction method from two perspectives: the generation of more feature descriptors of part regions through the learning process of deep networks and the aggregation of part-level discriminative features. This approach is designed to improve the backbone networks to generate finer-level part features through a part-level feature enhancement module and to investigate the intrinsic part-level features of the backbone networks with the help of a feature aggregation module. The enhancement module efficiently finds the finer features highly correlated to the part regions. Then the feature aggregation module builds correlations of similar part features through feature grouping and fusion. Moreover, our proposed method does not require additional parts annotations and achieves comparable performance on two widely-used benchmarks for recognizing fine-grained vehicle types. Experimental results and explainable visualizations demonstrate the effectiveness of the proposed method.

Feature relocation network for fine-grained image classification

In fine-grained image classification, there are only very subtle differences between classes. It is challenging to learn local discriminative features and remove distractive features in fine-grained image classification. Existing fine-grained image classification methods learn discriminative feature mainly via manual part annotation or attention mechanisms. However, due to the large intraclass variance and interclass similarity, the discriminative information and distractive information still are not distinguished effectively. To address this problem, we propose a feature relocation network (FRe-Net) which takes advantage of the different natures of features learned from different stages of the network. Our network consists of a distractive feature learning module and a relocated high-level feature learning module. In the distractive feature learning module, we propose to exploit the difference between low-level features and high-level features to design a distractive loss Ldistractive, which guides the attention to locate distractive regions more accurately. In the relocated high-level feature learning module, we enhance the representing capacity of the middle-level feature via the attention module and subtract the distractive feature learned from the distractive feature learning module in order to learn more local discriminative features. In end-to-end model training, the distractive feature learning module and the relocated high-level feature learning module are beneficial to each other via joint optimization. We conducted comprehensive experiments on three benchmark datasets widely used in fine-grained image classification. The experimental results show that FRe-Net achieves state-of-the-art performance, which validates the effectiveness of FRe-Net.

ქართველურ ენათა ლინგვისტური ანოტაცია: I. სახელები (პრობლემები და მათი გადაჭრის გზები)

The international standards of morphemic analysis of texts have outlined completely new prospects of research for the languages with rich morphological structurebecause typological linguistics systematizes and classifies language facts and phenomena based on the structural content and the number of structural markers. The unity of these markers is relevant for the definition of the essence of language as such, the very phenomenon of language. Morphologically, of great interest with regard to any language are the noun and the verb, because they are structural units of ample syntagmatic data. This proves the importance of the outcomes of morphemic analysis, as well as the vast grammatical capacity of these parts of speech. Kartvelian languages embrace the empirical material of four languages (Georgian, Megrelian, Laz and Svan) with diverse and multifunctional morphemic data. Their systemic research is urgent, important and vital for the definition of the typology of Kartvelian languages as well as for the general typological research. Modern challenges have created the need for the new,reliable and knowledge-based research product that will meet contemporary international standards. Linguistic morphological analysis is carried out based on the synchronic data. Due to the morphonological changes peculiar of nominal stems, we should take into account the diachronic-etymological factors in order to identify the original forms and meanings of words. The problems of linguistic annotation of the nominal parts of speech in Kartvelian languages are analyzed based on the grammatical categories of case and number:The issues of segmentation and qualification of the nominal parts of speech with postpositions are discussed with reference to all the four Kartvelian languages.In Megrelian and Laz, especially in Megrelian, the definition-identification of the nominal stem is specific with regard to the following:a. The former genitive cases (secondary cases).b. Due to the phonetic insertion in the singular and plural forms (the insertion that separates consonants or vowels).c. Due to the phonetic transformation of the stem and the case-marker.d. Due to the changes in some case-markers caused by phonetic rules.In Megrelian anthroponyms (maiden names) a gender suffix -ხე /xe/ is usually found.It is traditionally considered that Svan nouns are more complex than Svan verbs. This opinion is based on the existence of two/three stems in the Svan declension system and the existence of double declension paradigm. The empirical material is diverse, taking into account the dialectal data as well. Out of Kartvelian languages, only Svan has the so-called “prepositional-postpositional“ cases. Such forms express semantic nuances, and, in this, they are close to double local cases found in the highland dialects of the Svan language. The lexical-grammatical patterns denoting names of clans-families are also extremely diverse in the Svan langauge.In the process of annotation (segmentation-marking) of the nominal lexemes, alongside with formal data, we should take into account the semantic peculiarities. Besides, description-qualification of the formal mechanisms of nominal parts of speech will be complete and accurateonly if it reflects the outcomes of the phonetic transformation of forms. Annotation, including morphological annotation, is conventional (achieved on the basis of agreement). However, the theoretical foundation should be consistent, especially in condition of diverse opinions.

Attribute Prototype Network for Any-Shot Learning

Any-shot image classification allows to recognize novel classes with only a few or even zero samples. For the task of zero-shot learning, visual attributes have been shown to play an important role, while in the few-shot regime, the effect of attributes is under-explored. To better transfer attribute-based knowledge from seen to unseen classes, we argue that an image representation with integrated attribute localization ability would be beneficial for any-shot, i.e. zero-shot and few-shot, image classification tasks. To this end, we propose a novel representation learning framework that jointly learns discriminative global and local features using only class-level attributes. While a visual-semantic embedding layer learns global features, local features are learned through an attribute prototype network that simultaneously regresses and decorrelates attributes from intermediate features. Furthermore, we introduce a zoom-in module that localizes and crops the informative regions to encourage the network to learn informative features explicitly. We show that our locality augmented image representations achieve a new state-of-the-art on challenging benchmarks, i.e. CUB, AWA2, and SUN. As an additional benefit, our model points to the visual evidence of the attributes in an image, confirming the improved attribute localization ability of our image representation. The attribute localization is evaluated quantitatively with ground truth part annotations, qualitatively with visualizations, and through well-designed user studies.

One Sketch for All: One-Shot Personalized Sketch Segmentation.

We present the first one-shot personalized sketch segmentation method. We aim to segment all sketches belonging to the same category provisioned with a single sketch with a given part annotation while (i) preserving the parts semantics embedded in the exemplar, and (ii) being robust to input style and abstraction. We refer to this scenario as personalized. With that, we importantly enable a much-desired personalization capability for downstream fine-grained sketch analysis tasks. To train a robust segmentation module, we deform the exemplar sketch to each of the available sketches of the same category. Our method generalizes to sketches not observed during training. Our central contribution is a sketch-specific hierarchical deformation network. Given a multi-level sketch-strokes encoding obtained via a graph convolutional network, our method estimates rigid-body transformation from the target to the exemplar, on the upper level. Finer deformation from the exemplar to the globally warped target sketch is further obtained through stroke-wise deformations, on the lower-level. Both levels of deformation are guided by mean squared distances between the keypoints learned without supervision, ensuring that the stroke semantics are preserved. We evaluate our method against the state-of-the-art segmentation and perceptual grouping baselines re-purposed for the one-shot setting and against two few-shot 3D shape segmentation methods. We show that our method outperforms all the alternatives by more than 10% on average. Ablation studies further demonstrate that our method is robust to personalization: changes in input part semantics and style differences.

SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization.

Over the past few years, a significant progress has been made in deep convolutional neural networks (CNNs)-based image recognition. This is mainly due to the strong ability of such networks in mining discriminative object pose and parts information from texture and shape. This is often inappropriate for fine-grained visual classification (FGVC) since it exhibits high intra-class and low inter-class variances due to occlusions, deformation, illuminations, etc. Thus, an expressive feature representation describing global structural information is a key to characterize an object/ scene. To this end, we propose a method that effectively captures subtle changes by aggregating context-aware features from most relevant image-regions and their importance in discriminating fine-grained categories avoiding the bounding-box and/or distinguishable part annotations. Our approach is inspired by the recent advancement in self-attention and graph neural networks (GNNs) approaches to include a simple yet effective relation-aware feature transformation and its refinement using a context-aware attention mechanism to boost the discriminability of the transformed feature in an end-to-end learning process. Our model is evaluated on eight benchmark datasets consisting of fine-grained objects and human-object interactions. It outperforms the state-of-the-art approaches by a significant margin in recognition accuracy.

Multiple Instance Learning Convolutional Neural Networks for Fine-Grained Aircraft Recognition

The key to fine-grained aircraft recognition is discovering the subtle traits that can distinguish different subcategories. Early approaches leverage part annotations of fine-grained objects to derive rich representations. However, manual labeling part information is cumbersome. In response to this issue, previous CNN-based methods reuse the backbone network to extract part-discrimination features, the inference process of which consumes much time. Therefore, we introduce generalized multiple instance learning (MIL) into fine-grained recognition. In generalized MIL, an aircraft is assumed to consist of multiple instances (such as head, tail, and body). Firstly, instance-level representations are obtained by the feature extractor and instance conversion component. Secondly, the obtained instance features are scored by an MIL classifier, which can yield high-level part semantics. Finally, a fine-grained object label is inferred by a MIL pooling function that aggregates multiple instance scores. The proposed approach is trained end-to-end without part annotations and complex location networks. Experimental evidence is conducted to prove the feasibility and effectiveness of our approach on combined aircraft images (CAIs).

Pedestrian instance segmentation with prior structure of semantic parts

• We propose a simple, easily interpreted semantic part branch for pedestrian instance segmentation. • We proposed a pedestrian parts annotations generation method. • Our method improves the accuracy of pedestrian instance segmentation on the baseline. • Our method does not need to introduce additional annotations. Existing pedestrian segmentation and detection methods often show a significant drop in performance when heavy occlusion and deformation happen because most approaches rely on holistic modeling. Unlike many previous deep models that directly learn a holistic detector, in this paper, we introduce a pedestrian instance segmentation method with a prior structure of semantic parts named Part Mask R-CNN. Based on pedestrian parts’ proportion structure, process the original dataset annotations and then generate parts annotations as prior. By combining the semantic part branch with other classic detection and segmentation branches, the network learns more about pedestrian instances. Besides, we get such a more accurate pedestrian instance segmentation model without any artificial annotations. By extensive evaluations on the Cityscapes dataset, the results demonstrate that the proposed method can improve approaches such as Mask R-CNN, inaccuracy on pedestrian single class instance segmentation.

High-Order-Interaction for weakly supervised Fine-Grained Visual Categorization

Fine-Grained Visual Categorization (FGVC) is a challenging task due to the large intra-subcategory and small inter-subcategory variances. Recent studies tackle this task through a weakly supervised manner without using the part annotation from the experts. Of those, methods based on bilinear pooling are one of the main categories for computing the interaction between deep features and have shown high effectiveness. However, these methods mainly focus on the correlation within one specific layer but largely ignore the high interactions between multiple layers. In this study, we argue that considering the high interaction between the features from multiple layers can help to learn more distinguishing fine-grained features. To this end, we propose a High-Order-Interaction (HOI) method for FGVC. In our HOI, an efficient cross-layer trilinear pooling is introduced to calculate the third-order interaction between three different layers. Third-order interactions of different combinations are then fused to form the final representation. HOI can produce more discriminative representations and be readily integrated with the two popular techniques, attention mechanism and triplet loss, to obtain superposed improvement. Extensive experiments conducted on four FGVC datasets show the great superiority of our method over bilinear-based methods and demonstrate that the proposed method achieves the state of the art.

Practical fine-grained learning based anomaly classification for ECG image

As a widely used vital sign within cardiology, Electrocardiography (ECG) provides the basis for assessing heart function and diagnosing cardiovascular diseases. Automated anomaly detection for ECG plays an important role in improving patient diagnosis efficiency and reducing healthcare costs. Practically, due to the limits of electronics support or the medical system setting, image is a more common format for large-scale ECG storage in most clinical institutions. To guarantee an automated ECG detection model's scalability and practicality in clinical applications, taking good advantage of ECG images is crucial. However, existing time digital-based discriminative models fail to learn from images effectively for two reasons. First of all, the signals recorded on images have much lower resolution and higher noise, which makes it impractical to extract precise ECG signals following existing techniques. Meanwhile, the differences between abnormal signals are usually subtle, and they may be overwhelmed by the noises in the images as well. Towards this end, we design a novel neural framework that can be directly applied to massive ECG images determining various types of cardiology abnormalities. It classifies fine-grained ECG images based on weakly supervised strategy, in which case only image-level labeling is required. By eliminating the need for part annotations, the proposed method can result in significant savings in annotation time and cost. The effectiveness of the method is demonstrated by experimental results on two real ECG datasets.

SP-GAN

We present SP-GAN, a new unsupervised sphere-guided generative model for direct synthesis of 3D shapes in the form of point clouds. Compared with existing models, SP-GAN is able to synthesize diverse and high-quality shapes with fine details and promote controllability for part-aware shape generation and manipulation, yet trainable without any parts annotations. In SP-GAN, we incorporate a global prior (uniform points on a sphere) to spatially guide the generative process and attach a local prior (a random latent code) to each sphere point to provide local details. The key insight in our design is to disentangle the complex 3D shape generation task into a global shape modeling and a local structure adjustment, to ease the learning process and enhance the shape generation quality. Also, our model forms an implicit dense correspondence between the sphere points and points in every generated shape, enabling various forms of structure-aware shape manipulations such as part editing, part-wise shape interpolation, and multi-shape part composition, etc., beyond the existing generative models. Experimental results, which include both visual and quantitative evaluations, demonstrate that our model is able to synthesize diverse point clouds with fine details and less noise, as compared with the state-of-the-art models.

Unsupervised learning for cuboid shape abstraction via joint segmentation from point clouds

Representing complex 3D objects as simple geometric primitives, known as shape abstraction, is important for geometric modeling, structural analysis, and shape synthesis. In this paper, we propose an unsupervised shape abstraction method to map a point cloud into a compact cuboid representation. We jointly predict cuboid allocation as part segmentation and cuboid shapes and enforce the consistency between the segmentation and shape abstraction for self-learning. For the cuboid abstraction task, we transform the input point cloud into a set of parametric cuboids using a variational auto-encoder network. The segmentation network allocates each point into a cuboid considering the point-cuboid affinity. Without manual annotations of parts in point clouds, we design four novel losses to jointly supervise the two branches in terms of geometric similarity and cuboid compactness. We evaluate our method on multiple shape collections and demonstrate its superiority over existing shape abstraction methods. Moreover, based on our network architecture and learned representations, our approach supports various applications including structured shape generation, shape interpolation, and structural shape clustering.

Construction of text resources for automatic identification of clinical information in unstructured narratives

A significant proportion of the clinical record is in free text format, making it difficult to extract key information and make secondary use of patient data. Automatic detection of information within narratives initially requires humans, following specific protocols and rules, to identify medical entities of interest. To build a linguistic resource of annotated medical entities on texts produced in Chilean hospitals. A clinical corpus was constructed using 150 referrals in public hospitals. Three annotators identified six medical entities: clinical findings, diagnoses, body parts, medications, abbreviations, and family members. An annotation scheme was designed, and an iterative approach to train the annotators was applied. The F1-Score metric was used to assess the progress of the annotator's agreement during their training. An average F1-Score of 0.73 was observed at the beginning of the project. After the training period, it increased to 0.87. Annotation of clinical findings and body parts showed significant discrepancy, while abbreviations, medications, and family members showed high agreement. A linguistic resource with annotated medical entities on texts produced in Chilean hospitals was built and made available, working with annotators related to medicine. The iterative annotation approach allowed us to improve performance metrics. The corpus and annotation protocols will be released to the research community.

Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification

Deep convolutional neural networks (CNNs) have shown a strong ability in mining discriminative object pose and parts information for image recognition. For fine-grained recognition, context-aware rich feature representation of object/scene plays a key role since it exhibits a significant variance in the same subcategory and subtle variance among different subcategories. Finding the subtle variance that fully characterizes the object/scene is not straightforward. To address this, we propose a novel context-aware attentional pooling (CAP) that effectively captures subtle changes via sub-pixel gradients, and learns to attend informative integral regions and their importance in discriminating different subcategories without requiring the bounding-box and/or distinguishable part annotations. We also introduce a novel feature encoding by considering the intrinsic consistency between the informativeness of the integral regions and their spatial structures to capture the semantic correlation among them. Our approach is simple yet extremely effective and can be easily applied on top of a standard classification backbone network. We evaluate our approach using six state-of-the-art (SotA) backbone networks and eight benchmark datasets. Our method significantly outperforms the SotA approaches on six datasets and is very competitive with the remaining two.

CA‐PMG: Channel attention and progressive multi‐granularity training network for fine‐grained visual classification

Fine-grained visual classification is challenging due to the inherently subtle intra-class object variations. To solve this issue, a novel framework named channel attention and progressive multi-granularity training network, is proposed. It first exploits meaningful feature maps through the channel attention module and captures multi-granularity features by the progressive multi-granularity training module. For each feature map, the channel attention module is proposed to explore channel-wise correlation. This allows the model to re-weight the channels of the feature map according to the impact of their semantic information on performance. Furthermore, the progressive multi-granularity training module is introduced to fuse features cross multi-granularity. And the fused features pay more attention to the subtle differences between images. The model can be trained efficiently in an end-to-end manner without bounding box or part annotations. Finally, comprehensive experiments are conducted to show that the method achieves state-of-the-art performances on the CUB-200-2011, Stanford Cars, and FGVC-Aircraft datasets. Ablation studies demonstrate the effectiveness of each part in our module.

Learning to locate for fine-grained image recognition

In this paper, we propose an end-to-end weakly supervised method for fine-grained image recognition called bounding box-part location method(BBPL), which can locate the object and part precisely without part annotations. The proposed method includes three modules: object detection, ObjectMask, and classification. Firstly, the object detection module predicts the bounding boxes, and the predicted bounding boxes are employed to generate a mask through ObjectMask module. The generated mask can suppress the background interference during recognition. Secondly, the classification module can be further divided into two branches, which are global feature classification and local feature classification. In global feature classification branch, global feature is extracted to get global classification result. While in local feature classification branch, salient point is first detected through our novel salient point detection module, which can greatly reduce the consuming-time compared with the most existing local feature extraction methods. Further, the local feature is extracted in these detected salient points, and local classification result is obtained by local feature classification branch. Finally, we get the final result by fusing the results of two classification branches together. With experiments on three widely used fine-grained image recognition datasets (CUB-200-2011, Stanford Cars, Stanford Dogs), our method can achieve the state-of-the-art performance.