Visual Attributes Research Articles

WYTIWYR: A User Intent‐Aware Framework with Multi‐modal Inputs for Visualization Retrieval

AbstractRetrieving charts from a large corpus is a fundamental task that can benefit numerous applications such as visualization recommendations. The retrieved results are expected to conform to both explicit visual attributes (e.g., chart type, colormap) and implicit user intents (e.g., design style, context information) that vary upon application scenarios. However, existing example‐based chart retrieval methods are built upon non‐decoupled and low‐level visual features that are hard to interpret, while definition‐based ones are constrained to pre‐defined attributes that are hard to extend. In this work, we propose a new framework, namely WYTIWYR (What‐You‐Think‐Is‐What‐You‐Retrieve), that integrates user intents into the chart retrieval process. The framework consists of two stages: first, the Annotation stage disentangles the visual attributes within the query chart; and second, the Retrieval stage embeds the user's intent with customized text prompt as well as bitmap query chart, to recall targeted retrieval result. We develop aprototype WYTIWYR system leveraging a contrastive language‐image pre‐training (CLIP) model to achieve zero‐shot classification as well as multi‐modal input encoding, and test the prototype on a large corpus with charts crawled from the Internet. Quantitative experiments, case studies, and qualitative interviews are conducted. The results demonstrate the usability and effectiveness of our proposed framework.

Dynamic learning for imbalanced data in learning chest X-ray and CT images

Massive annotated datasets are necessary for networks of deep learning. When a topic is being researched for the first time, as in the situation of the viral epidemic, handling it with limited annotated datasets might be difficult. Additionally, the datasets are quite unbalanced in this situation, with limited findings coming from significant instances of the novel illness. We offer a technique that allows a class balancing algorithm to understand and detect lung disease signs from chest X-ray and CT images. Deep learning techniques are used to train and evaluate images, enabling the extraction of basic visual attributes. The training objects' characteristics, instances, categories, and relative data modeling are all represented probabilistically. It is possible to identify a minority category in the classification process by using an imbalance-based sample analyzer. In order to address the imbalance problem, learning samples from the minority class are examined. The Support Vector Machine (SVM) is used to categorize images in clustering. Physicians and medical professionals can use the CNN model to validate their initial assessments of malignant and benign categorization. The proposed technique for class imbalance (3-Phase Dynamic Learning (3PDL)) and parallel CNN model (Hybrid Feature Fusion (HFF)) for multiple modalities achieve a high F1 score of 96.83 and precision is 96.87, its outstanding accuracy and generalization suggest that it may be utilized to create a pathologist's help tool.

Aesthetics++: Refining Graphic Designs by Exploring Design Principles and Human Preference.

During the creation of graphic designs, individuals inevitably spend a lot of time and effort on adjusting visual attributes (e.g., positions, colors, and fonts) of elements to make them more aesthetically pleasing. It is a trial-and-error process, requires repetitive edits, and relies on good design knowledge. In this work, we seek to alleviate such difficulty by automatically suggesting aesthetic improvements, i.e., taking an existing design as the input and generating a refined version with improved aesthetic quality as the output. This goal presents two challenges: proposing a refined design based on the user-given one, and assessing whether the new design is better aesthetically. To cope with these challenges, we propose a design principle-guided candidate generation stage and a data-driven candidate evaluation stage. In the candidate generation stage, we generate candidate designs by leveraging design principles as the guidance to make changes around the existing design. In the candidate evaluation stage, we learn a ranking model upon a dataset that can reflect humans' aesthetic preference, and use it to choose the most aesthetically pleasing one from the generated candidates. We implement a prototype system on presentation slides and demonstrate the effectiveness of our approach through quantitative analysis, sample results, and user studies.

Frequency of adapting events affects face aftereffects but not blur aftereffects

The dynamics of visual adaptation remain poorly understood. Recent studies have found that the strength of adaptation aftereffects in the perception of numerosity depends more strongly on the number of adaptation events than on the duration of the adaptation. We investigated whether such effects can be observed for other visual attributes. We measured blur (perceived focus-sharp vs blurred adapt) and face (perceived race- Asian vs. White adapt) aftereffects by varying the number of adaptation events (4 or 16) and the duration of each adaptation event (0.25 s or 1 s). We found evidence for an effect of event number on face but not on blur adaptation, though the effect for faces was significant for only one of the two face adapt conditions (Asian). Our results suggest that different perceptual dimensions may vary in how adaptation effects accrue, potentially because of differences in factors such as the sites (early or late) of the sensitivity changes or nature of the stimulus. These differences may impact how and how rapidly the visual system can adjust to different visual properties.

RETRACTED ARTICLE: Simulation of video image detection in leisure sports tourism industry based on convolutional neural network

Due to the continuous development of information technology, emerging technologies such as 5G networks, Turing devices and deep learning are widely used in people's lives. At present, the sports and tourism industry has become a pillar industry supporting the national economy. Sports events have attracted many people's attention, and people are willing to devote themselves to sports activities. Participants and visitors are also increasing day by day. The article also conducts research on the rapid detection of human video behavior, obtains more representative features for this demand. This research is of great help to the development of the industry. This allows the network to manipulate the input video behavior at different scales and extract deeper video information. Therefore, this paper proposes another 3D convolutional neural network model that combines a multi-level pyramid network with an attention mechanism. The visual attributes in the video behavior data set are explicitly learned to refine the classification of similar features. This paper also proposes an integrated neural network based on visual attribute enhancement. It also shows good adaptability, and can complete video data processing based on time series. The extraction of time and space functions is realized by the two-stream neural network method, the two networks learn the corresponding pixel ratios, and it is not easy to lose the clues of the action recognition. This paper also proposes a spatiotemporal convolutional neural network video for human body recognition algorithm, which is applied to the leisure tourism industry according to the specific analysis of the above method, and promotes the development of the industry.

Multi-printer learning framework for efficient optical printer characterization.

A high prediction accuracy of optical printer models is a prerequisite for accurately reproducing visual attributes (color, gloss, translucency) in multimaterial 3D printing. Recently, deep-learning-based models have been proposed, requiring only a moderate number of printed and measured training samples to reach a very high prediction accuracy. In this paper, we present a multi-printer deep learning (MPDL) framework that further improves data efficiency utilizing supporting data from other printers. Experiments on eight multi-material 3D printers demonstrate that the proposed framework can significantly reduce the number of training samples thus the overall printing and measurement efforts. This makes it economically feasible to frequently characterize 3D printers to achieve a high optical reproduction accuracy consistent across different printers and over time, which is crucial for color- and translucency-critical applications.

Stabilizing expectations when shifting from analytical to intuitive reasoning: The role of prediction errors in reasoning

As humans, we rely on intuitive reasoning for most of our decisions. However, when there is a novel or atypical decision to be made, we must rely on a slower and more deliberative thought process—analytical reasoning. As we gain experience with these novel or atypical decisions, our reasoning shifts from analytical to intuitive, which parallels a reduction in the need for cognitive control. Here, we sought to confirm this claim by employing electroencephalographic (EEG) measures of cognitive control as participants performed a simple perceptual decision-making task. Specifically, we had participants categorize “blobs” into families based on their visual attributes so we could examine how their reasoning changed with learning. In a key manipulation, halfway through the experiment we introduced novel blob families to categorize, thus temporarily increasing the need for analytical reasoning (i.e., cognitive control). Congruent with past research, we focused our EEG analyses on frontal theta activity as it has been linked to cognitive control and analytical thinking. As hypothesized, we found a transition from analytical to intuitive decision-making systems with learning as indexed by a decrease in frontal theta power. Further, when the novel blobs were introduced at the midpoint of the experiment, we found that decisions about these stimuli recruited analytical reasoning as indicated by increased theta power in comparison to decisions about well-practiced stimuli. We propose our findings to reflect prediction errors to decision demands—a monitoring process that determines whether our expectations of demands are met. Shifting from analytical to intuitive reasoning thus reflects the stabilization of our expectations of decision demands, which can be violated with unexpected demands when encountering novel stimuli.

Hierarchical disentangling network for object representation learning

An object can be described as the combination of primary visual attributes. Disentangling such underlying primitives is the long-term objective of representation learning. It is observed that categories have natural hierarchical characteristics, i.e., any two objects can share some common primitives at a particular category level while possess unique traits at another. However, previous works usually operate in a flat manner (i.e., at a particular level) to disentangle the representations of objects. Even though they may obtain the primitives to constitute objects as the categories at that level, their results are obviously not efficient and complete. In this paper, we propose a Hierarchical Disentangling Network (HDN) to exploit the rich hierarchical characteristics among categories to divide the disentangling process in a coarse-to-fine manner (i.e., level-wise), such that each level only focuses on learning the specific representations and finally the common and unique representations at all levels jointly constitute the raw object. Specifically, HDN is designed based on an encoder-decoder architecture. To simultaneously ensure the level-wise disentanglement and interpretability of the encoded representations, a novel hierarchical Generative Adversarial Network (GAN) is introduced. Quantitative and qualitative evaluations on popular object datasets validate the effectiveness of our method.

Research on Texture Classification Based on Multi-Scale Information Fusion

Texture feature is an important visual cue for an image, which is the unified description of human visual attributes and sensory attributes. The inherent problem of texture image is that the difference of intra-class images is large and the disparity of inter-class images is small. This problem increases the difficulty of texture image recognition. Therefore, improving the relevance embedding of intra-class images can reduce the classification errors caused by this problem. To solve this problem, this paper proposes a multi-scale information fusion network algorithm, which adopts a cascade structure. It combines multi-scale feature information with the corresponding background information. The shallow background information guides the next stage of feature formation and enhances the similarity of intra-class images. The intra-class feature information obtained is more general. The algorithm has been tested on data sets describable texture database (DTD) and Flickr material dataset (FMD), which has achieved good results.

Learning discriminative visual semantic embedding for zero-shot recognition

We present a novel zero-shot learning (ZSL) method that concentrates on strengthening the discriminative visual information of the semantic embedding space for recognizing object classes. To address the ZSL problem, many previous works strive to learn a transformation to bridge the visual features and semantic representations, while ignoring that the discriminative property of the semantic embedding space can benefit zero-shot prediction tasks. Among these existing approaches, human-defined attributes are typically employed to build up the mid-level semantics. However, the discriminative capability and completeness of manually defined attributes are hard to guarantee, which may easily cause semantic ambiguity. To alleviate this issue, we propose a discriminative visual semantic embedding (DVSE) model that formulates the ZSL problem as a supervised dictionary learning framework. The proposed method is capable of exploring a set of discriminative visual attributes and ensures knowledge transfer across categories. Moreover, a unified objective is introduced to generate an augmented semantic embedding space where these learned visual attributes and human-defined attributes are incorporated jointly for consolidating the visual cues of feature representations. Finally, we treat the DVSE model as an optimization problem and further propose an iterative solver. Extensive experiments on several challenging benchmark datasets demonstrate that the proposed method achieves favorable performances compared with state-of-the-art ZSL approaches.

Antibacterial and antifungal activity of zinc oxide nanoparticles produced by Phaeodactylum tricornutum culture supernatants and their potential application to extend the shelf life of sweet cherry (Prunus avium L.)

In the current work, zinc oxide nanoparticles (ZnONPs) synthesized by the cell-free supernatant of Phaeodactylum tricornutum cultures were first examined for their physicochemical and antimicrobial proprieties. They were then applied as spray coating on the surface of fresh sweet cherries to evaluate their efficacy on quality maintenance of the fruit over 10 days of storage at 25 °C. Synthesized nanoparticles showed a large-scale inhibitory effect against Escherichia coli (ATCC 25922), Staphylococcus aureus MRSA (ATCC 43300) and Candida albicans (ATCC 14053) with minimum inhibitory concentrations (MIC) ranged between 0.078 and 0.156 mg/mL. Furthermore, ZnONPs at 100–1000 mg/L effectively inhibited mycelia growth of Mucor heimalis, a causative agent of Mucor rot in sweet cherries. The results also indicated that ZnONPs significantly delayed the changes of visual and biochemical attributes of the fruit. Compared to untreated control, coated cherries with ZnONP at a concentration of 500 mg/L showed very small change in moisture and total sugar content. Therefore, all coated fruits exhibited less decay rates. In addition, spray coating with ZnONPs had a positive effect on maintaining suitable amounts of ascorbic acid (29.74 mg/100 g FW), polyphenols (579.2 μg/g FW) and flavonoids in the fruits (1991.87 μg/g FW). Data from this study showed 5.11% and 11.63%, increase in flavonoid content and antioxidant activity in the fruits coated with 500 mg/L ZnONPs, respectively, at the end of storage. These data together suggest that green ZnONPs-based coating is one of the safest and effective ways to preserve commercial fruits for prolonged storage.

On Selecting, Ranking, and Quantifying Features for Building a Liver CT Diagnosis Aiding Computational Intelligence Method

The liver is one of the most common locations for incidental findings during abdominal CT scans. There are multiple types of disease that can arise within the liver and many of them are nodular. The ultimate goal of our research is to develop an expert knowledge-based system using fuzzy signatures, to support decisions during diagnosis of the most frequent of these nodular lesions. Since the literature contains limited information about the graphical properties of CT images that must be taken into consideration and their relationship to one another, in this paper we focused on selecting and ranking the input parameters using expert knowledge and determining their importance. Six visual attributes of lesions (size, shape, density, homogeneity contour, and other features) were selected based on textbooks of radiology and expert opinion. The importance of these attributes was ranked by radiologist experts using questionnaires and a pairwise comparison technique. The most important feature was found to be the density of the lesion on the various CT phases, and the least important was the size, the order of the other attributes was other features, contour, homogeneity, and shape, with a Kendall concordance coefficient of 0.612. Weights for the attributes, to be used in the future fuzzy signatures, were also determined. As a last step, several statistical parameter-based quantities were generated to represent the above abstract attributes and evaluated by comparing them to expert opinions.

Preparation of Vanillin-Taurine Antioxidant Compound, Characterization, and Evaluation for Improving the Post-Harvest Quality of Litchi

Litchi's post-harvest pericarp browning is one of the main constraints that drastically affect its visual attributes and market potential. Therefore, the vanillin-taurine Schiff base (VTSB) compound prepared from natural compounds of vanillin and taurine exhibited higher DPPH-radical-scavenging invitro antioxidant activity than vanillin. VTSB first-time report to mitigate the postharvest browning of litchi fruit. In this study, litchi fruits were dipped in 0.3 mM (based on pre-experiment) VTSB solution and stored at 25 ± 1 °C for six days to examine their effects on browning and postharvest quality. Fruit treated with VTSB had lower levels of browning degree (BD), browning index (BI), weight loss, soluble quinone (SQ), relative electrolyte leakage (REL), and malondialdehyde (MDA) than control fruit. Additionally, total anthocyanins and phenolic concentrations, Total soluble solids (TSS), and 2,2-diphenyl-1-picrylhydrazyl-free radical scavenging activity (DPPH-RSA) were preserved higher in VTSB-treated litchi fruit. The levels of Ascorbate peroxidase (APX), Superoxide dismutase (SOD), and Catalase (CAT) were higher in treated fruit, whereas polyphenol oxidase (PPO) and Peroxidase (POD) were decreased during the postharvest period. This study suggested that VTSB would be very useful for different post-harvest problems in the fruit and vegetable industry.

Neural Photometry-Guided Visual Attribute Transfer.

We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g., texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following an image analogies approach, the method only requires the training data to contain the same visual structures as the input guidance. Our approach works at interactive rates, making it suitable for material edit applications. We thoroughly evaluate our learning methodology in a controlled setup providing quantitative measures of performance. Last, we demonstrate that training the model on a single material is enough to generalize to materials of the same type without the need for massive datasets.

Body size and the architecture of the visual system in crepuscular and diurnal bees

AbstractCrepuscular bees have larger compound eyes, ommatidia and ocelli than diurnal bees. These visual adaptations allow these bees to forage during twilight. While the sizes of visual organs are positively correlated with body size in diurnal bees, this relationship is as yet unclear in crepuscular bees. Here we measured the sizes of the ocellus; compound eyes; ommatidial density; and dorsal, frontal and ventral facets of 11 phylogenetically related bee species with different temporal patterns. We tested whether and how each of these visual attributes is correlated with body size and/or temporal pattern. Except for ommatidial density and size of the frontal ommatidia, which are similar in crepuscular bees of different sizes, we found that all other visual variables in diurnal and crepuscular bees are correlated with both body size and temporal pattern. The lower variability in ommatidial density among crepuscular bees suggests that their eyes operate on the threshold between light sensitivity and visual acuity. Furthermore, similar frontal ommatidial sizes in bees of different sizes indicate that this eye region is under higher evolutionary pressure in small than in large crepuscular bees to ensure light sensitivity. Finally, those crepuscular bees from different families convergently share the same visual characteristics.

The visible subtitler: Blockchain technology towards right management and minting.

Background: Subtitles are produced through different workflows and technologies: from fully automatic to human in open source web editors or in-house platforms, and increasingly through hybrid human-machine interaction. There is little agreement regarding subtitle copyright beyond the understanding that it is a derivative work. While same language verbatim subtitles may have little room for creativity, interlingual subtitling is heavily dependent on the subtitler skills to translate, prioritise, and condense information. These days creative subtitles are increasingly being used as one more aesthetic element in audiovisual narrative. Though they may be in the same language, the visual attributes that contribute to the narrative development make creative subtitles one more element that should be acknowledged and copyright protected. Methods: The paper will present a short introduction to subtitling copyright. It will then describe centralised and decentralised copyright management - where blockchain technology can be applied to aid subtitler identification. A focus group with expert professional subtitlers was organised, and feedback is reported. Conclusions: Subtitle copyright is country dependent, still subtitling working practices and media asset distribution have no geographical borders. Blockchain technology -as a concept- could aid subtitle traceability. This can be useful beyond financial and moral right management and work towards media sustainability, allowing for reuse and repurpose of existing media assets.

Automated Image Annotation With Novel Features Based on Deep ResNet50-SLT

Due to their vast size, the growing number of digital images found in personal archives and on websites has become unmanageable, making it challenging to accurately retrieve images from these large databases. While these collections are popular due to their convenience, they are often not equipped with proper indexing information, making it difficult for users to find what they need. One of the most significant challenges in the field of computer vision and multimedia is image annotation, which involves labeling images with descriptive keywords. However, computers do not possess the capability to understand the essence of images in the same way that humans do, and people can only identify images based on their visual attributes, not their deeper semantic meaning. Therefore, image annotation requires the use of keywords to effectively communicate the contents of an image to a computer system. However, raw pixels in an image do not provide enough information to generate semantic concepts, making image annotation a complex task. Unlike text annotation, where the dictionary linking words to semantics is well established, image annotation lacks a clear definition of "words" or "sentences" that can be associated with the meaning of the image, known as the semantic gap. To address this challenge, this study aimed to characterize image content meaningfully to make information retrieval easier. An improved automatic image annotation (AIA) system was proposed to bridge the semantic gap between low-level computer features and human interpretation of images by assigning one or multiple labels to images. The proposed AIA system can convert raw image pixels into semantic-level concepts, providing a clearer representation of the image content. The study combined the ResNet50 and slantlet transform with word2vec and principal component analysis with t-distributed stochastic neighbor embedding to balance precision and recall. This allowed the researchers to determine the optimal model for the proposed ResNet50-SLT AIA framework. A Word2vec model with ResNet50-SLT was used with principal component analysis and t-distributed stochastic neighbor embedding to improve IA prediction accuracy. The distributed representation approach involved encoding and storing information about image features. The proposed AIA system utilized seq2seq to generate sentences depending on feature vectors. The system was implemented on the most popular datasets (Flickr8k, Corel-5k, ESP-Game). The results showed that the newly developed AIA scheme overcame the computational time complexity associated with most existing image annotation models during the training phase for large datasets. The performance evaluation of the AIA scheme showed its excellent flexibility of annotation, improved accuracy, and reduced computational costs, thus outperforming the existing state-of-the-art methods. In conclusion, this AIA framework can provide immense benefits in accurately selecting and extracting image features and easily retrieving images from large databases. The extracted features can effectively be used to represent the image, thus accelerating the annotation process and minimizing the computational complexity.

Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting

This work proposes a multimodal Hindi-to-English machine translation on a news corpus by integrating multiple input modalities. The experimental dataset comprises of an image from a news article and its caption in the English-Hindi language. The parallel English-Hindi dataset is constructed by translating the image caption from English-language news articles into Hindi. Unlike standard images used for NLP tasks, images in news articles frequently depict significant events, locations, or individuals. The caption provides a more extensive description of the image, resulting in the inclusion of multiple identified entities. Our methodology consists of encoders for each input modality, allowing for simultaneous consideration of both text and image. We undertake two tests to examine how effectively the image content improves MT systems when 1) the entire image is considered first and 2) the largest object recognized in the image is considered. We employ Byte Pair Encoding (BPE) to represent the text, and the visual attributes are extracted from the image with VGG-19, a pre-trained CNN model. In terms of BLEU and chrF, our Multimodal Machine Translation (MMT) systems are superior than the baseline unimodal NMT system. Our MMT system outperforms the NMT model by +1.8 BLEU and +0.03 chrF.

A Large Visual, Qualitative, and Quantitative Dataset for Web Intelligence Applications

The Web is the communication platform and source of information par excellence. The volume and complexity of its content have grown enormously, with organizing, retrieving, and cleaning Web information becoming a challenge for traditional techniques. Web intelligence is a novel research area to improve Web‐based services and applications using artificial intelligence and automatic learning algorithms, for which a large amount of Web‐related data are essential. Current datasets are, however, limited and do not combine visual representation and attributes of Web pages. Our work provides a large dataset of 49,438 Web pages, composed of webshots, along with qualitative and quantitative attributes. This dataset covers all the countries in the world and a wide range of topics, such as art, entertainment, economics, business, education, government, news, media, science, and the environment, addressing different cultural characteristics and varied design preferences. We use this dataset to develop three Web Intelligence applications: knowledge extraction on Web design using statistical analysis, recognition of error Web pages using a customized convolutional neural network (CNN) to eliminate invalid pages, and Web categorization based solely on screenshots using a CNN with transfer learning to assist search engines, indexers, and Web directories.

Self-Supervised Video-Based Action Recognition With Disturbances.

Self-supervised video-based action recognition is a challenging task, which needs to extract the principal information characterizing the action from content-diversified videos over large unlabeled datasets. However, most existing methods choose to exploit the natural spatio-temporal properties of video to obtain effective action representations from a visual perspective, while ignoring the exploration of the semantic that is closer to human cognition. For that, a self-supervised Video-based Action Recognition method with Disturbances called VARD, which extracts the principal information of the action in terms of the visual and semantic, is proposed. Specifically, according to cognitive neuroscience research, the recognition ability of humans is activated by visual and semantic attributes. An intuitive impression is that minor changes of the actor or scene in video do not affect one person's recognition of the action. On the other hand, different humans always make consistent opinions when they recognize the same action video. In other words, for an action video, the necessary information that remains constant despite the disturbances in the visual video or the semantic encoding process is sufficient to represent the action. Therefore, to learn such information, we construct a positive clip/embedding for each action video. Compared to the original video clip/embedding, the positive clip/embedding is disturbed visually/semantically by Video Disturbance and Embedding Disturbance. Our objective is to pull the positive closer to the original clip/embedding in the latent space. In this way, the network is driven to focus on the principal information of the action while the impact of sophisticated details and inconsequential variations is weakened. It is worthwhile to mention that the proposed VARD does not require optical flow, negative samples, and pretext tasks. Extensive experiments conducted on the UCF101 and HMDB51 datasets demonstrate that the proposed VARD effectively improves the strong baseline and outperforms multiple classical and advanced self-supervised action recognition methods.