Visual Features Research Articles

Event-horizon-scale Imaging of M87* under Different Assumptions via Deep Generative Image Priors

Reconstructing images from the Event Horizon Telescope (EHT) observations of M87*, the supermassive black hole at the center of the galaxy M87, depends on a prior to impose desired image statistics. However, given the impossibility of directly observing black holes, there is no clear choice for a prior. We present a framework for flexibly designing a range of priors, each bringing different biases to the image reconstruction. These priors can be weak (e.g., impose only basic natural-image statistics) or strong (e.g., impose assumptions of black hole structure). Our framework uses Bayesian inference with score-based priors, which are data-driven priors arising from a deep generative model that can learn complicated image distributions. Using our Bayesian imaging approach with sophisticated data-driven priors, we can assess how visual features and uncertainty of reconstructed images change depending on the prior. In addition to simulated data, we image the real EHT M87* data and discuss how recovered features are influenced by the choice of prior.

Graph-enhanced visual representations and question-guided dual attention for visual question answering

Visual Question Answering (VQA) has witnessed significant advancements recently, due to the application of deep learning in the field of vision-language research. Most current VQA models focus on merging visual and text features, but it is essential for these models to also consider the relationships between different parts of an image and use question information to highlight important features. This study proposes a method to enhance neighboring image region features and learn question-aware visual representations. First, we construct a region graph to represent spatial relationships between objects in the image. Then, graph convolutional networks (GCNs) are used to propagate information across neighboring regions, enriching each region’s feature representation by integrating contextual information. To capture long-range dependencies, the graph is enhanced with random walk with restart (RWR), enabling multi-hop reasoning across distant regions. Furthermore, a question-aware dual attention mechanism is introduced to further refine region features at both region and feature levels, ensuring that the model emphasizes key regions that are critical for answering the question. The enhanced region representations are then combined with the encoded question to predict an answer. Through extensive experiments on VQA benchmarks, the study demonstrates state-of-the-art performance by leveraging regional dependencies and question guidance. The integration of GCNs and random walks in the graph helps capture contextual information to focus visual attention selectively, resulting in significant improvements over existing methods on VQA 1.0 and VQA 2.0 benchmark datasets.

Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering

Medical Visual Question Answering (VQA) is a task that aims to provide answers to questions about medical images, which utilizes both visual and textual information in the reasoning process. The absence of large-scale annotated medical VQA datasets presents a formidable obstacle to training a medical VQA model from scratch in an end-to-end manner. Existing works have been using image captioning dataset in the pre-training stage and fine-tuning to downstream VQA tasks. Following the same paradigm, we use a collection of public medical image captioning datasets to pre-train multimodality models in a self-supervised setup, and fine-tune to downstream medical VQA tasks. In the work, we propose a method that featured with Cross-Modal pre-training with Multiple Objectives (CMMO), which includes masked image modelling, masked language modelling, image-text matching, and image-text contrastive learning. The proposed method is designed to associate the visual features of medical images with corresponding medical concepts in captions, for learning aligned vision and language feature representations, and multi-modal interactions. The experimental results reveal that our proposed CMMO method outperforms state-of-the-art methods on three public medical VQA datasets, showing absolute improvements of 2.6%, 0.9%, and 4.0% on the VQA-RAD, PathVQA, and SLAKE dataset, respectively. We also conduct comprehensive ablation studies to validate our method, and visualize the attention maps which show a strong interpretability. The code and pre-trained weights will be released at https://github.com/pengfeiliHEU/CMMO.

LRCN: Layer-residual Co-Attention Networks for Visual Question Answering

Visual Question Answering (VQA) is a multimodal task requiring a collaborative understanding of fine-grained visual concepts and language semantics. The key to VQA research is constructing a framework for modeling inter and intramodal information interactions between intricate modalities. Due to the superior global view and the ability to capture the relationships within multimodal information, the Transformer and its variants have become the prime choice for VQA tasks. Despite this layer-by-layer architecture enabling answer reasoning by optimizing multimodal feature information, the information may still be lost when transferred from lower to higher layers. To solve such an issue, we propose a Layer-Residual Mechanism (LRM), a plug-and-play generic approach that can reduce the computation and memory overhead to almost negligible. By adding a residual straight-through line between adjacent layers that cascades the attention block in-depth, LRM mitigates the vanishing during information transfer, stabilizes training, and endeavors to overcome performance decline in VQA models at deeper layers. To verify the effectiveness and generality of the proposed LRM, we apply it to the Encoder-Decoder structure, the Pure-Stacking structure, and a specifically designed Co-Stacking structure that can simultaneously understand textual features and visual features called Layer-residual Co-Attention Networks (LRCN). Extensive ablation studies and comparative experiments on the benchmark VQA v2 and CLEVR datasets show that LRCN significantly outperforms the original architectures, demonstrating the superior effectiveness and compatibility of LRCN.

Scene-aware Foveated Rendering.

We propose a new scene-aware foveated rendering method, which incorporates the scene awareness and characteristics of the human visual system into the mapping-based foveated rendering framework. First, we generate the conservative visual importance map that encodes the visual features of the scene, visual acuity, and gaze motion. Second, we construct the pixel size control map using a convolution kernel method. Third, we utilize the pixel size control map to guide the foveated rendering. At last, a temporal coherent refinement strategy is used to maintain the smooth foveated rendering for the adjacent frames. Compared to the state-of-the-art mapping-based foveated rendering methods using the same compression ratio, our method achieves smaller MSE, higher PSNR, and SSIM in the fovea, periphery, salient regions, and the whole image. We also conducted user studies, and the results proved that the perceptual quality of our method has a high visual similarity with the around truth rendered with the full resolution.

Memory-enhanced hierarchical transformer for video paragraph captioning

Video paragraph captioning aims to describe a video that contains multiple events with a paragraph of generated coherent sentences. Such a captioning task is full of challenges since the high requirements for visual-textual relevance and semantic coherence across the captioning paragraph of a video. In this work, we introduce a memory-enhanced hierarchical transformer for video paragraph captioning. Our model adopts a hierarchical structure, where the outer layer transformer extracts visual information from a global perspective and captures the relevancy between event segments throughout the entire video, while the inner layer transformer further mines local details within each event segment. By thoroughly exploring both global and local visual information at the video and event levels, our model can provide comprehensive visual feature cues for promising paragraph caption generation. Additionally, we design a memory module to capture similar patterns among event segments within a video, which preserves contextual information across event segments and updates its memory state accordingly. Experimental results on two popular datasets, ActivityNet Captions and YouCook2, demonstrate that our proposed model can achieve superior performance, generating higher quality caption while maintaining consistency in the content of video.

A Picture May Be Worth a Hundred Words for Visual Question Answering

How far can textual representations go in understanding images? In image understanding, effective representations are essential. Deep visual features from object recognition models currently dominate various tasks, especially Visual Question Answering (VQA). However, these conventional features often struggle to capture image details in ways that match human understanding, and their decision processes lack interpretability. Meanwhile, the recent progress in language models suggests that descriptive text could offer a viable alternative. This paper investigated the use of descriptive text as an alternative to deep visual features in VQA. We propose to process description–question pairs rather than visual features, utilizing a language-only Transformer model. We also explored data augmentation strategies to enhance training set diversity and mitigate statistical bias. Extensive evaluation shows that textual representations using approximately a hundred words can effectively compete with deep visual features on both the VQA 2.0 and VQA-CP v2 datasets. Our qualitative experiments further reveal that these textual representations enable clearer investigation of VQA model decision processes, thereby improving interpretability.

Assessment of cervical spine CT by an image quality audit using qualitative and quantitative methods.

To study the feasibility and assess the correlation of qualitative and quantitative methods for an image quality (IQ) audit of a Cervical spine CT. Five radiologists retrospectively performed a blinded visual grading analysis (VGA) on 20 studies (10 from Protocol 1 and 10 from Protocol 2), using the RANZCR CT IQ Self-Audit worksheet. A Visual Grading Analysis Score (VGAS) and Area under the curve using Visual Grading Characteristics (AUCVGC) were the figures of merit. Quantitative metrics for noise and contrast were correlated to the qualitative assessment. No statistically significant difference was observed in the IQ, VGASProtocol1 = 0.65, 95% CI [0.54, 0.75] and VGASProtocol 2 = 0.73, 95% CI [0.67, 0.79] and AUCVGC = 0.548, 95% CI [0.40, 0.69]. Protocol 2 indicated a statistically significant average dose reduction of 35% in CTDIvol (P = 0.020) and a higher noise; however, the difference was statistically insignificant. There was a moderate correlation between the manual noise measurements in soft tissue and air (P = 0.035) and a strong correlation between the manual and automated noise measurements (P < 0.001). The contrast resolution-based quantitative parameter, EdgeGradientSoft, correlated to the qualitative scores (P = 0.031). Validated VGA tools can be used for IQ audits; however, tailoring the image criteria and rating scale to the clinical practice is suggested. The use of contrast-based IQ metrics showed encouraging results, and further larger-scale studies are needed to explore their potential use in quality management.

Modeling the Effect of Greenways’ Multilevel Visual Characteristics on Thermal Perception in Summer Based on Bayesian Network and Computer Vision

The aim of this study is to reveal the effects of multilevel visual characteristics of greenways on thermal perception in hot and humid regions during summer and to explore the potential of visual design to enhance psychological thermal comfort. Data on light (L), color (C), plant richness (PR), space openness (SO), scenic view (SV), thermal sensation (TS), and thermal preference (TP) were collected through questionnaires (n = 546). Computer vision technology was applied to measure the green view index (GVI), sky view index (SVI), paving index (PI), spatial enclosure (SE), and water index (WI). Using the hill climbing algorithm in R to construct a Bayesian network, model validation results indicated prediction accuracies of 0.799 for TS and 0.838 for TP. The results showed that: (1) SE, WI, and SV significantly positively influence TS, while L significantly negatively influences TS (R2 = 0.6805, p-value &lt; 0.05); (2) WI, TS, and SV significantly positively influence TP (R2 = 0.759, p-value &lt; 0.05).

The influence of neighbor selection on self-organized UAV swarm based on finite perception vision.

Recently, vision-based unmanned aerial vehicle (UAV) swarming has emerged as a promising alternative that can overcome the adaptability and scalability limitations of distributed and communication-based UAV swarm systems. While most vision-based control algorithms are predicated on the detection of neighboring objects, they often overlook key perceptual factors such as visual occlusion and the impact of visual sensor limitations on swarm performance. To address the interaction problem of neighbor selection at the core of self-organizing UAV swarm control, a perceptually realistic finite perception visual (FPV) neighbor selection model is proposed, which is based on the lateral visual characteristics of birds, incorporates adjustable lateral visual field widths and orientations, and is able to ignore occluded agents. Based on the FPV model, a neighbor selection method based on the Acute Angle Test (AAT) is proposed,&#xD;which overcomes the limitation that the traditional neighbor selection mechanism can only interact with the nearest neighboring agents. A large number of Monte Carlo simulation comparison experiments show that the proposed FPV+AAT neighborselection mechanism can reduce the redundant communication burden between large scale self-organized UAV swarms, and outperforms the traditional neighbor selection method in terms of order, safety, union, connectivity, and noise resistance.

A Study on the Visual Characteristics of Children's Picture Book Covers from the Perspective of Peircean Semiotics -Focusing on Picture Books Themed on Death-

A Study on the Visual Characteristics of Children's Picture Book Covers from the Perspective of Peircean Semiotics -Focusing on Picture Books Themed on Death-

User emotion recognition and indoor space interaction design: a CNN model optimized by multimodal weighted networks

In interior interaction design, achieving intelligent user-interior interaction is contingent upon understanding the user’s emotional responses. Precise identification of the user’s visual emotions holds paramount importance. Current visual emotion recognition methods rely solely on singular features, predominantly facial expressions, resulting in inadequate coverage of visual characteristics and low recognition rates. This study introduces a deep learning-based multimodal weighting network model to address this challenge. The model initiates with a convolutional attention module, employing a self-attention mechanism within a convolutional neural network (CNN). As a result, the multimodal weighting network model is integrated to optimize weights during training. Finally, a weight network classifier is derived from these optimized weights to facilitate visual emotion recognition. Experimental outcomes reveal a 77.057% correctness rate and a 74.75% accuracy rate in visual emotion recognition. Comparative analysis against existing models demonstrates the superiority of the multimodal weight network model, showcasing its potential to enhance human-centric and intelligent indoor interaction design.

Using Deep Learning and Cbir To Address Copyright Concerns of AI-Generated Art: A Systematic Literature Review

This systematic literature review explores the intersection of deep learning and content-based image retrieval (CBIR) in addressing copyright concerns related to AI-generated art. As artificial intelligence rapidly transforms various artistic domains, it raises critical questions regarding authorship, ownership, and the ethical implications of machine-generated creativity. The review examines the capabilities of CBIR systems in identifying AI-generated images by analyzing visual features such as color, texture, and shape. Additionally, it highlights the role of deep learning models in enhancing the accuracy of these systems through the detection of distinctive patterns characteristic of AI artworks. The findings underscore the importance of developing robust methodologies that leverage AI and CBIR technologies to protect intellectual property rights while fostering innovation in the creative industries. This research contributes to the broader discourse on the legal and ethical challenges posed by AI in art, providing insights for policymakers, artists, and technologists in navigating the evolving landscape of AI-generated content.

Early neural development of social interaction perception: evidence from voxel-wise encoding in young children and adults.

From a young age, children have advanced social perceptual and reasoning abilities. However, the neural development of these abilities is still poorly understood. To address this gap, we used fMRI data collected 122 3-12-year-old children (64 females) and 33 adults (20 females) watched an engaging and socially rich movie to investigate how the cortical basis of social processing changes throughout development. We labeled the movie with visual and social features, including motion energy, presence of a face, presence of a social interaction, theory of mind (ToM) events, valence and arousal. Using a voxel-wise encoding model trained on these features, we find that models based on visual (motion energy) and social (faces, social interaction, ToM, valence, and arousal) features can both predict brain activity in children as young as three years old across the cortex, with particularly high predictivity in motion selective middle temporal region (MT) and the superior temporal sulcus (STS). Furthermore, models based on individual social features showed that while there may be some development throughout childhood, social interaction information in the STS is present in children as young as three years old and appears adult-like by age seven. The current study, for the first time, links neural activity in children to pre-defined social features in a narrative movie and suggests social interaction perception is supported by early developing neural responses in the STS.Significance Statement This study investigates the neural basis for social scene perception ability in children using fMRI data collected while participants watch a short, animated movie. Unlike most prior studies with movies, we labeled a range of visual and social features in the movie and used machine learning analyses to link each feature to fMRI responses in adults and children ages 3-12. Notably, our results demonstrate strong evidence that children as young as three years old show significant responses to most visual and social features in the movie, including social interaction responses in the superior temporal sulcus (STS), a region in the brain that is well known to be important in social interaction processing in adults.

Fully exploring object relation interaction and hidden state attention for video captioning

Video Captioning (VC) is a challenging task of automatically generating natural language sentences for describing video contents. As a video often contains multiple objects, it is comprehensively crucial to identify multiple objects and model relationships between them. Previous models usually adopt Graph Convolutional Networks (GCN) to infer relational information via object nodes, but there exist uncertainty and over-smoothing issues of relational reasoning. To tackle these issues, we propose a Knowledge Graph based Video Captioning Network (KG-VCN) by fully exploring object relation interaction, hidden state and attention enhancement. In encoding stages, we present a Graph and Convolution Hybrid Encoder (GCHE), which uses an object detector to find visual objects with bounding boxes for Knowledge Graph (KG) and Convolutional Neural Network (CNN). To model intrinsic relations between detected objects, we propose a knowledge graph based Object Relation Graph Interaction (ORGI) module. In ORGI, we design triplets (head, relation, tail) to efficiently mine object relations, and create a global node to enable adequate information flow among all graph nodes for avoiding possibly missed relations. To produce accurate and rich captions, we propose a hidden State and Attention Enhanced Decoder (SAED) by integrating hidden states and dynamically updated attention features. Our SAED accepts both relational and visual features, adopts Long Short-Term Memory (LSTM) to produce hidden states, and dynamically update attention features. Unlike existing methods, we concatenate state and attention features to predict next word sequentially. To demonstrate the effectiveness of our model, we conduct experiments on three well-known datasets (MSVD, MSR-VTT, VaTeX), and our model achieves impressive results significantly outperforming existing state-of-the-art models.

Tightly coupled stereo vision-inertial odometry based on point and line feature

Abstract To improve the accuracy and robustness of visual simultaneous localization and mapping (SLAM) in low-texture environments, this paper proposes a robust and fast stereo vision inertial SLAM pose estimation method that combines point and line features with an inertial measurement unit (IMU). The method tightly couples visual point and line features with IMU constraints, forming a least-squares problem through the error of each constraint term for nonlinear optimization. To address the issues of over-segmentation and time consumption in traditional line segment detection (LSD) algorithms, an improved LSD algorithm is adopted to accelerate line feature extraction. This approach merges nearby line segments based on spatial geometric relationships and filters out invalid segments, improving the time efficiency of the algorithm. Finally, experiments conducted in low-texture environments demonstrate that our algorithm achieves high localization accuracy and robustness.

Enhancing EFL Learning Through Multimodal Integration: The Role of Visual and Auditory Features in Moroccan Textbooks

This study examines the impact of multimodal integration, particularly the combination of visual and auditory elements, on EFL learning through Moroccan textbooks, specifically English Spotlight 1 and English Horizons. It investigates how these educational materials rely on visuals, diagrams, and audio tools to foster comprehension, retention, and engagement among secondary school students in Morocco. Employing a mixed-methods approach that includes both qualitative and quantitative analyses, the study evaluates the effectiveness of these features from the viewpoints of students and teachers. The results indicate that visual components significantly and positively affect vocabulary acquisition and grammatical comprehension, while auditory elements enhance listening skills and pronunciation. The research emphasizes the necessity for more advanced multimodal content in Moroccan EFL education and identifies technological barriers that impede full integration. It recommends an increased emphasis on multimodal resources in textbook development and the improvement of technological infrastructure in educational institutions.

A Patch-Level Region-Aware Module with a Multi-Label Framework for Remote Sensing Image Captioning

Recent Transformer-based works can generate high-quality captions for remote sensing images (RSIs). However, these methods generally feed global or grid visual features to a Transformer-based captioning model for associating cross-modal information, which limits performance. In this work, we investigate unexplored ideas for a remote sensing image captioning task, using a novel patch-level region-aware module with a multi-label framework. Due to an overhead perspective and a significantly larger scale in RSIs, a patch-level region-aware module is designed to filter the redundant information in the RSI scene, which benefits the Transformer-based decoder by attaining improved image perception. Technically, the trainable multi-label classifier capitalizes on semantic features as supplementary to the region-aware features. Moreover, modeling the inner relations of inputs is essential for understanding the RSI. Thus, we introduce region-oriented attention, which associates region features and semantic labels, omits the irrelevant regions to highlight relevant regions, and learns related semantic information. Extensive qualitative and quantitative experimental results show the superiority of our approach on the RSICD, UCM-Captions, and Sydney-Captions. The code for our method will be publicly available.

AnimeSketchNet: Integrating traditional sketching and anime art via an innovative dataset

Abstract. Recent advancements in text-to-image synthesis have shown promise, yet many models struggle with generalizability and adaptability, particularly in specialized genres like anime sketches. To overcome these challenges, we introduce AnimeSketchNet, a novel framework designed to enhance model performance by effectively capturing the complexity and stylistic variability unique to anime. AnimeSketchNet leverages a diverse collection of anime sketch styles with varying levels of detail, enabling the model to learn and adapt to the distinct artistic style of anime. Our dual-branch architecture, featuring a decoupled attention mechanism, seamlessly assimilates textual descriptions along with visual features, producing high-quality anime sketch that are both detailed and faithful the reference sketch. Experimental results demonstrate that AnimeSketchNet surpasses existing state-of-the-art models in both qualitative and quantitative evaluations. This work significantly advances text-to-image synthesis by offering a powerful and flexible tool tailored specifically to the unique demands of anime art.

Evaluating the synergistic effects of sesame cake powder and soy protein isolate on rheological, textural, nutritional, and phenolic profiles of high-moisture extrusion processed meat analogs.

In this study, the high-moisture meat analogs (HmMAs) were developed by incorporating defatted sesame cake powder (DSP) in soy protein isolate (SyPI). The quality attributes of HmMA like visual appearance, specific mechanical energy (SME), mass flow rate (MFR), phenolic profile, textural and rheological properties were assessed after varying DSP concentrations (0%, 20%, 40%, and 60% w/w) and feed moisture (FM) levels (55% and 60%). The HmMA1 (derived solely from SyPI) exhibited higher hardness, chewiness, gumminess, cohesiveness, and springiness. The HmMA prepared from SyPI-DSP blends (HmMA2-8) demonstrated significant improvements in nutritional composition, and their visual characteristics indicated noticeable anisotropy. The interaction between SyPI and DSP influenced the quality of HmMA. Thehigher DSP concentration led to higher MFR and deeper curvatures of U-shaped structures, whereas lower SMS, textural and rheological properties. The DSP incorporation and 55% FM adjustments allowed mimicking meat cuts with thick fiber, influenced color, and proved advantageous in developing white meat analogs with higher free phenolics. The findings of the study suggest avenues for exploring DSP at a suitable level in SyPI for the development of better quality meat analogs.