Visual Information Extraction Research Articles

Multi-view distillation based on multi-modal fusion for few-shot action recognition (CLIP-MDMF)

In recent years, the field of few-shot action recognition (FSAR) has garnered significant attention. Although many methods primarily rely on mono-modal data, there is a growing trend towards utilizing multi-modal data. However, existing FSAR methods often employ simplistic fusion techniques, such as concatenation or comparison, which may not fully leverage the potential of multi-modal information. We aim to explore multi-modal information from different views and subsequently perform multi-view fusion. Based on the textual and visual modality information extracted by the CLIP backbone, we propose an MDMF method that comprehensively utilizes these modalities at two levels. At the first level, we extract visual information from two views: Local Temporal Context and Global Temporal Context. Within each view, we fuse visual features with textual information through concatenation and Cross-Transformer operations. We then employ metric comparison to derive probability distributions for classification under the meta-learning paradigm for each view. At the second level, we fuse the probability distributions from both views to make the final decision. Concurrently, during training, for each query, we calculate the posterior distributions of the textual and visual modalities within each view using text information distance and visual information distance. Based on these distributions, we group query samples with higher view reliability. Subsequently, we enhance the representation of the less reliable view of specific samples through mutual distillation. By delving deep into multi-modal data through a multi-view approach, our few-shot action recognition model demonstrates the potential for achieving higher accuracy and enhanced robustness. Our code is available at the URL: https://github.com/cofly2014/CLIP-MDMF.

Image manipulation localization via dynamic cross-modality fusion and progressive integration

Image manipulation localization aims at distinguishing forged regions from the whole test image. Though many outstanding prior arts have been proposed for this task, there are still two issues that need to be further studied: (1) how to fuse diverse types of features with forgery clues; (2) how to progressively integrate multistage features for better localization performance. In this paper, we propose a tripartite progressive integration network (TriPINet) for end-to-end image manipulation localization. First, we extract both visual perception information, e.g., RGB input images, and visual imperceptible features, e.g., frequency and noise traces for forensic feature learning. Second, we develop a guided cross-modality dual-attention (gCMDA) module to fuse different types of forged clues. Third, we design a set of progressive integration squeeze-and-excitation (PI-SE) modules to improve localization performance by appropriately incorporating multiscale features in the decoder. Extensive experiments are conducted to compare our method with state-of-the-art image forensics approaches. The proposed TriPINet obtains competitive results on several benchmark datasets.

Language conditioned multi-scale visual attention networks for visual grounding

Visual grounding (VG) is a task that requires to locate a specific region in an image according to a natural language expression. Existing efforts on the VG task are divided into two-stage, one-stage and Transformer-based methods, which have achieved high performance. However, most of the previous methods extract visual information at a single spatial scale and ignore visual information at other spatial scales, which makes these models unable to fully utilize the visual information. Moreover, the insufficient utilization of linguistic information, especially failure to capture global linguistic information, may lead to failure to fully understand language expressions, thus limiting the performance of these models. To better address the task, we propose a language conditioned multi-scale visual attention network (LMSVA) for visual grounding, which can sufficiently utilize visual and linguistic information to perform multimodal reasoning, thus improving performance of model. Specifically, we design a visual feature extractor containing a multi-scale layer to get the required multi-scale visual features by expanding the original backbone. Moreover, we exploit pooling the output of the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to extract sentence-level linguistic features, which can enable the model to capture global linguistic information. Inspired by the Transformer architecture, we present the Visual Attention Layer guided by Language and Multi-Scale Visual Features (VALMS), which is able to better learn the visual context guided by multi-scale visual and linguistic features, and facilitates further multimodal reasoning. Extensive experiments on four large benchmark datasets, including ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg, demonstrate that our proposed model achieves the state-of-the-art performance.

A survey for table recognition based on deep learning

Tables, as an important information interaction media containing much structured information, are popularly adopted in our life. There are a lot of table recognition methods for understanding and applying these information. Among them, table recognition based on deep learning is the mainstream. Therefore, we conducted this survey to summarize the latest research on table recognition based on deep learning in recent years. Table recognition can be divided into three sub-tasks, namely table detection, table structure recognition and table content recognition, so we firstly introduce the importance and significance of table recognition research and analyze the current challenges faced by each sub-task. Then, we list the existing public table recognition datasets and introduce commonly used evaluation metrics. Furthermore, we review deep learning-based solutions proposed in recent years for three different sub-tasks and reclassify them. Especially for table structure recognition, we reclassify them according to their annotation schemes. Finally, we report the performance of some excellent methods, such as CascadeTabNet, TSRFormer, SEM, etc, on public datasets, and describe the future research directions, including table recognition in natural scenes, question and answer about tables, visual information extraction, domain adaptation etc. In summary, this survey reviews the data, evaluations, methods, and development directions of table recognition. Table understanding and domain adaptation need as well to be focused on. The former can predict the relationships between two table elements, which helps us to understand the table. The latter can enhance the generalization ability of table recognition methods, thereby obtaining more robust recognition performance. Both are important development directions for table recognition in the future.

Visual perception of group fatness as a precursor to social identity threat and safety

ABSTRACT Across two studies (N = 190), we evaluate how fat people perceive groups of bodies and how this visual processing predicts social identity threat and safety. We examined how fat people ensemble code – rapidly and simultaneously extract visual information from a group – body size from groups of bodies. Stimuli included sets of 3-d modeled bodies varying in body mass index (BMI; Study 1) and real photographs of groups of people with known BMI (Study 2). As hypothesized, we found that fat people were generally accurate at ensemble coding the average body size of arrays after 500 ms exposure, and that, particularly among fat people with medium and high stigma consciousness, arrays of increasing average BMI were associated with heightened feelings of identity safety. Our work suggests that the representation of fatness in one’s environment may be an important precursor to social identity threat and safety processes.

PGCL: Prompt guidance and self-supervised contrastive learning-based method for Visual Question Answering

Recent works have demonstrated the efficacy of Chain-of-Thought (CoT), which comprises multimodal information, in multiple complex reasoning tasks. CoT, involving multiple stages of reasoning, has also been applied to Visual Question Answering (VQA) for scientific questions. Existing research on CoT in science-oriented VQA primarily concentrates on the extraction and integration of visual and textual information. However, they overlook the fact that image-question pairs, categorized by different attributes (such as subject, topic, category, skill, grade, and difficulty), emphasize distinct text information, visual information, and reasoning capabilities. Therefore, this work proposes a novel VQA method termed PGCL, founded on the prompt guidance strategy and self-supervised contrastive learning. PGCL strategically excavates and integrates text and visual information based on attribute information. Specifically, two prompt templates are first crafted. They are subsequently combined with the attribution information and the interference information of image-question pairs to generate a series of prompt positive and prompt negative samples respectively. The mining of visual and text representations is then guided by constructed prompts. These prompt-guided representations are integrated and enhanced via transformer architecture and self-supervised contrastive learning. The fused features are eventually learned to predict answers for VQA. Sufficient experiments have convincingly substantiated the individual contributions of the components within PGCL, as well as the performance of PGCL.

LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation.

Surgical scene segmentation is a critical task in Robotic-assisted surgery. However, the complexity of the surgical scene, which mainly includes local feature similarity (e.g., between different anatomical tissues), intraoperative complex artifacts, and indistinguishable boundaries, poses significant challenges to accurate segmentation. To tackle these problems, we propose the Long Strip Kernel Attention network (LSKANet), including two well-designed modules named Dual-block Large Kernel Attention module (DLKA) and Multiscale Affinity Feature Fusion module (MAFF), which can implement precise segmentation of surgical images. Specifically, by introducing strip convolutions with different topologies (cascaded and parallel) in two blocks and a large kernel design, DLKA can make full use of region- and strip-like surgical features and extract both visual and structural information to reduce the false segmentation caused by local feature similarity. In MAFF, affinity matrices calculated from multiscale feature maps are applied as feature fusion weights, which helps to address the interference of artifacts by suppressing the activations of irrelevant regions. Besides, the hybrid loss with Boundary Guided Head (BGH) is proposed to help the network segment indistinguishable boundaries effectively. We evaluate the proposed LSKANet on three datasets with different surgical scenes. The experimental results show that our method achieves new state-of-the-art results on all three datasets with improvements of 2.6%, 1.4%, and 3.4% mIoU, respectively. Furthermore, our method is compatible with different backbones and can significantly increase their segmentation accuracy. Code is available at https://github.com/YubinHan73/LSKANet.

Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition

The goal of multimodal named entity recognition (MNER) is to detect entity spans in given image–text pairs and classify them into corresponding entity types. Despite the success of existing works that leverage cross-modal attention mechanisms to integrate textual and visual representations, we observe three key issues. Firstly, models are prone to misguidance when fusing unrelated text and images. Secondly, most existing visual features are not enhanced or filtered. Finally, due to the independent encoding strategies employed for text and images, a noticeable semantic gap exists between them. To address these challenges, we propose a framework called visual clue guidance and consistency matching (GMF). To tackle the first issue, we introduce a visual clue guidance (VCG) module designed to hierarchically extract visual information from multiple scales. This information is utilized as an injectable visual clue guidance sequence to steer text representations for error-insensitive prediction decisions. Furthermore, by incorporating a cross-scale attention (CSA) module, we successfully mitigate interference across scales, enhancing the image’s capability to capture details. To address the third issue of semantic disparity between text and images, we employ a consistency matching (CM) module based on the idea of multimodal contrastive learning, facilitating the collaborative learning of multimodal data. To validate the effectiveness of our proposed framework, we conducted comprehensive experimental studies, including extensive comparative experiments, ablation studies, and case studies, on two widely used benchmark datasets, demonstrating the efficacy of the framework.

“Determining the efficacy of a machine learning model for measuring periodontal bone loss”

BackgroundConsidering the prevalence of Periodontitis, new tools to help improve its diagnostic workflow could be beneficial. Machine Learning (ML) models have already been used in dentistry to automate radiographic analysis.AimsTo determine the efficacy of an ML model for automatically measuring Periodontal Bone Loss (PBL) in panoramic radiographs by comparing it to dentists.MethodsA dataset of 2010 images with and without PBL was segmented using Label Studio. The dataset was split into n = 1970 images for building a training dataset and n = 40 images for building a testing dataset. We propose a model composed of three components. Firstly, statistical inference techniques find probability functions that best describe the segmented dataset. Secondly, Convolutional Neural Networks extract visual information from the training dataset. Thirdly, an algorithm calculates PBL as a percentage and classifies it in stages. Afterwards, a standardized test compared the model to two radiologists, two periodontists and one general dentist. The test was built using the testing dataset, 40 questions long, done in controlled conditions, with radiologists considered as ground truth. Presence or absence, percentage, and stage of PBL were asked, and time to answer the test was measured in seconds. Diagnostic indices, performance metrics and performance averages were calculated for each participant.ResultsThe model had an acceptable performance for diagnosing light to moderate PBL (weighted sensitivity 0.23, weighted F1-score 0.29) and was able to achieve real-time diagnosis. However, it proved incapable of diagnosing severe PBL (sensitivity, precision, and F1-score = 0).ConclusionsWe propose a Machine Learning model that automates the diagnosis of Periodontal Bone Loss in panoramic radiographs with acceptable performance.

Image feature extraction algorithm based on visual information

Abstract Vision is the main sensory organ for human beings to contact and understand the objective world. The results of various statistical data show that more than 60% of all ways for human beings to obtain external information are through the visual system. Vision is of great significance for human beings to obtain all kinds of information needed for survival, which is the most important sense of human beings. The rapid growth of computer technology, image processing, pattern recognition, and other disciplines have been widely applied. Traditional image processing algorithms have some limitations when dealing with complex images. To solve these problems, some scholars have proposed various new methods. Most of these methods are based on statistical models or artificial neural networks. Although they meet the requirements of modern computer vision systems for feature extraction algorithms with high accuracy, high speed, and low complexity, these algorithms still have many shortcomings. For example, many researchers have used different methods for feature extraction and segmentation to get better segmentation results. Scale-invariant feature transform (SIFT) is a description used in the field of image processing. This description has scale invariance and can detect key points in the image. It is a local feature descriptor. A sparse coding algorithm is an unsupervised learning method, which is used to find a set of “super complete” basis vectors to represent sample data more efficiently. Therefore, combining SIFT and sparse coding, this article proposed an image feature extraction algorithm based on visual information to extract image features. The results showed that the feature extraction time of X algorithm for different targets was within 0.5 s when the other conditions were the same. The feature matching time was within 1 s, and the correct matching rate was more than 90%. The feature extraction time of Y algorithm for different targets was within 2 s. The feature matching time was within 3 s, and the correct matching rate was between 80 and 90%, indicating that the recognition effect of X algorithm was better than that of Y algorithm. It indicates the positive relationship between visual information and image feature extraction algorithm.

Extraction of visual information to predict crowdfunding success

Researchers have increasingly turned to crowdfunding platforms to gain insights into entrepreneurial activity and dynamics. While previous studies have explored various factors influencing crowdfunding success, such as technology, communication, and marketing strategies, the role of visual elements that can be automatically extracted from images has received less attention. This is surprising, considering that crowdfunding platforms emphasize the importance of attention‐grabbing and high‐resolution images, and previous research has shown that image characteristics can significantly impact product evaluations. Indeed, a comprehensive review of empirical articles ( n = 202) utilized Kickstarter data, focusing on the incorporation of visual information in their analyses. Our findings reveal that only 29.70% controlled for the number of images, and less than 12% considered any image details. In this manuscript, we contribute to the existing literature by emphasizing the significance of visual characteristics as essential variables in empirical investigations of crowdfunding success. We review the literature on image processing and its relevance to the business domain, highlighting two types of visual variables: visual counts (number of pictures and number of videos) and image details. Building upon previous work that discussed the role of color, composition, and figure–ground relationships, we introduce visual scene elements that have not yet been explored in crowdfunding, including the number of faces, the number of concepts depicted, and the ease of identifying those concepts. To demonstrate the predictive value of visual counts and image details, we analyze Kickstarter data using flexible machine learning models (Lasso, Ridge, Bayesian additive regression trees, and eXtreme Gradient Boosting). Our results highlight that visual count features are two of the top three predictors of success and highlight the ease at which researchers can incorporate some information about visual information. Our results also show that simple image detail features such as color matter a lot, and our proposed measures of visual scene elements can also be useful. By supplementing our article with R and Python codes that help authors extract image details ( https://osf.io/ujnzp/ ), we hope to stimulate scholars in various disciplines to consider visual information data in their empirical research and enhance the impact of visual cues on crowdfunding success.

Invited Session III: Diversity in chromatic processing across the animal kingdom: Color processing in the mouse retina.

Across species, specialized retinal circuits allow animals to extract visual information from their environments. How retinal circuits extract relevant visual information is a major area of inquiry. In the mouse retina, cone photoreceptors possess a gradient of opsin expression leading to uneven detection of colors across visual space. However, at the output of the retina, ganglion cells' color preferences deviate from this gradient, suggesting that circuits in the retina may alter the color information before sending it to the brain. We explored how circuits in the retina shape chromatic information, focusing on the retina's interneurons, amacrine cells and bipolar cells. We found that inhibitory amacrine cells rebalance color preferences, leading to diverse color selectivity throughout retinal space. Since amacrine cells vary widely across species, these cells are poised to tune the chromatic information sent to the brain to each species' environmental niche.

INFORMATION EXTRACTION FROM MULTISPECTRAL SATELLITE IMAGES

The article analyzes various methods and approaches of modern remote sensing that can be used in the processing of multispectral satellite images in order to effectively extract visual information about territories for which preliminary data is not available. Attention is paid to the creation of new derivative images (synthesized and indexed) and to performing pixel-oriented computer non supervised classification. A series of experiments have been made that clearly reveal the advantages and conveniences of remote retrieval of information from multispectral satellite images in a territory for which reference objects and other data acquired in situ are not available.

Anomalous binocular vision in African Harrier-Hawks

An animal's visual field is the three-dimensional space around its head from which it can extract visual information at any instant1. Bird visual fields vary markedly between species, and this variation is likely to be driven primarily by foraging ecology1,2,3. The binocular visual field is the region in which the visual fields of the two eyes overlap; thus, objects in the binocular field are imaged by both eyes simultaneously. The binocular field plays a pivotal role in the detection of symmetrical optic flow-fields, providing almost instantaneous information on the direction of travel and the time to contact a target towards which the head or feet is travelling; thus, information from the binocular field is crucial in guiding key foraging behaviours2,3. Here, we demonstrate an unusual visual field and binocular extent above the head in African Harrier-Hawks, also known as Gymnogenes (Polyboroides typus) compared to 18 other members of the Accipitridae4,5. We argue that the observed visual field can be attributed to the unusual and specific foraging behaviour of African Harrier-Hawks.

Computational assessment of visual coding across mouse brain areas and behavioural states.

Our brain is bombarded by a diverse range of visual stimuli, which are converted into corresponding neuronal responses and processed throughout the visual system. The neural activity patterns that result from these external stimuli vary depending on the object or scene being observed, but they also change as a result of internal or behavioural states. This raises the question of to what extent it is possible to predict the presented visual stimuli from neural activity across behavioural states, and how this varies in different brain regions. To address this question, we assessed the computational capacity of decoders to extract visual information in awake behaving mice, by analysing publicly available standardised datasets from the Allen Brain Institute. We evaluated how natural movie frames can be distinguished based on the activity of units recorded in distinct brain regions and under different behavioural states. This analysis revealed the spectrum of visual information present in different brain regions in response to binary and multiclass classification tasks. Visual cortical areas showed highest classification accuracies, followed by thalamic and midbrain regions, with hippocampal regions showing close to chance accuracy. In addition, we found that behavioural variability led to a decrease in decoding accuracy, whereby large behavioural changes between train and test sessions reduced the classification performance of the decoders. A generalised linear model analysis suggested that this deterioration in classification might be due to an independent modulation of neural activity by stimulus and behaviour. Finally, we reconstructed the natural movie frames from optimal linear classifiers, and observed a strong similarity between reconstructed and actual movie frames. However, the similarity was significantly higher when the decoders were trained and tested on sessions with similar behavioural states. Our analysis provides a systematic assessment of visual coding in the mouse brain, and sheds light on the spectrum of visual information present across brain areas and behavioural states.

ARQUITETURA ESCOLAR SENSÍVEL AO PROCESSAMENTO SENSORIAL DOS AUTISTAS: estudo do conforto visual por meio da APO da E. M. Luiza Terra de Andrade – RJ

This research analyzes E. M. Luiza Terra de Andrade, with the objective of verifying if this school unit has a physical structure, focusing on visual comfort, capable of serving students with Autism Spectrum Disorder (ASD). The aim is to create design parameters for the construction of schools sensitive to the characteristics of autistic people, since the Sensory Discrimination Disorders (SDD) of these people in-terfere in the way they extract visual information from the built environments. Using the correlational method, the relationships between TDS and autistic behaviors with the visual aspects of environments were examined. The qualitative method allowed the appropriation of information on neurodevelopment and environmental psychology. Through the experiential research, it was possible to structure the pos-t-occupancy evaluation (POA) carried out. Thus, it was found that the analyzed School, with regard to aspects of visual comfort, does not meet the particularities of autistic people, impacting the care of these students.

Gaze behavior data in the vitrine of human movement science: considerations on eye-tracking technique

Background: The eyes are the main gateway of visual information input. Moving the eyes is essential to extract visual information from scenes while performing motor actions. This has helped to explain motor behavior, especially in relation to visual attention mechanisms, gaze training and learning, and the relevance of visual information in controlling actions. Thus, collecting data on gaze behavior has become important for explaining motor behavior. Aim: We present the main video-based eye-tracking techniques, briefly describing the anatomy of the eyes, explaining the operation of the eye-tracker (eye capture techniques, calibration, and data analysis), and proposing interpretations of the main variables extracted by the technique. In this way, we carry out considerations (limitations and advantages) on the eye-tracking technique that placed gaze behavior data in the vitrine of human movement science. Interpretation: Eye-tracking has become an excellent tool to assist in the analysis of human movement through gaze behavior. Mainly by combining sensory information, such as visual information, with performance during motor tasks, it is possible to infer about perception, cognition, and human behavior during the most diverse day-to-day activities. Eye-tracker systems have been employed in different majors related to motor behavior, such as medicine, commerce, and game development.

Development of Image Processing Tools

The development of image processing tools has revolutionised various fields, including computer vision, medical imaging, remote sensing, and multimedia applications. These tools utilise algorithms and techniques to analyse and manipulate images, allowing for enhanced visual perception, feature extraction, and information extraction. This abstract highlights the recent advancements and trends in the development of image processing tools. It discusses the importance of image preprocessing techniques such as noise reduction, image enhancement, and geometric transformations in improving image quality and extracting valuable information. Additionally, it explores the role of image segmentation, object detection, and recognition algorithms in identifying and classifying objects within images. Image processing tools have emerged as powerful tools for tasks such as image classification, object detection, and image generation. This article discusses some image processing tools that help achieve the above-mentioned tasks.

Differences Between East Asians and Westerners in the Mental Representations and Visual Information Extraction Involved in the Decoding of Pain Facial Expression Intensity.

The online version contains supplementary material available at 10.1007/s42761-023-00186-1.

Driver eye glance behavior and performance with camera-based visibility systems versus mirrors

Objective Drivers’ ability to extract visual information efficiently from mirrors or camera-based visibility systems impacts driving performance when carrying out maneuvers such as lane changes. The objective of the research was to compare drivers’ eye gaze behavior and driving performance with mirrors versus camera-based visibility systems (i.e., CMS, or camera monitor system) to identify any differences and possible impacts on safety. Methods A test track study was conducted comparing drivers’ eye gaze and lane change behavior when driving a vehicle equipped with outside mirrors versus a prototype CMS. Participants’ opinions regarding usability and comfort in using mirrors versus the tested CMS were also obtained using a post-drive questionnaire. Results Study results were somewhat mixed but did demonstrate that with the tested CMS, participants took longer to pass a slower moving vehicle and maintained a greater resultant distance from the passed vehicle. Additionally, participants had a greater number of fixations to the CMS displays compared to the outside rearview mirrors. Results also found slight perceived advantages for the tested CMS in regard to ease of use, comfortability, and visibility. When asked to choose which rear visibility technology they would prefer to use in everyday driving, most participants preferred the outside rearview mirrors over the tested prototype CMS or having both systems. However, not all lane change and gaze metrics followed the same pattern. Conclusions In this study, participants’ longer time to pass a slower moving vehicle, greater distance when passing a slower moving vehicle, greater number of fixations, and lower subjective ratings with the tested CMS may indicate difficulty in judging distances and focusing on the electronic image. This study provides preliminary findings that suggest differences in driving behavior exist between a single tested prototype CMS and outside rearview mirrors and is a foundational step toward evaluating whether these trends are consistent across different systems and overall implications for safe driving behavior.