Visual Task Research Articles

Tasks Reflected in the Eyes: Egocentric Gaze-Aware Visual Task Type Recognition in Virtual Reality.

With eye tracking finding widespread utility in augmented reality and virtual reality headsets, eye gaze has the potential to recognize users' visual tasks and adaptively adjust virtual content displays, thereby enhancing the intelligence of these headsets. However, current studies on visual task recognition often focus on scene-specific tasks, like copying tasks for office environments, which lack applicability to new scenarios, e.g., museums. In this paper, we propose four scene-agnostic task types for facilitating task type recognition across a broader range of scenarios. We present a new dataset that includes eye and head movement data recorded from 20 participants while they engaged in four task types across 15 360-degree VR videos. Using this dataset, we propose an egocentric gaze-aware task type recognition method, TRCLP, which achieves promising results. Additionally, we illustrate the practical applications of task type recognition with three examples. Our work offers valuable insights for content developers in designing task-aware intelligent applications. Our dataset and source code are available at zhimin-wang.github.io/TaskTypeRecognition.html.

The dorsomedial prefrontal cortex promotes self-control by inhibiting the egocentric perspective

The dorsomedial prefrontal cortex (dmPFC) plays a crucial role in social cognitive functions, including perspective-taking. Although perspective-taking has been linked to self-control, the mechanism by which the dmPFC might facilitate self-control remains unclear. Using the multimodal neuroimaging dataset from the Human Connectome Project (Study 1, N =978 adults), we established a reliable association between the dmPFC and self-control, as measured by discounting rate—the tendency to prefer smaller, immediate rewards over larger, delayed ones. Experiments (Study 2, N = 36 adults) involving high-definition transcranial direct current stimulation showed that anodal stimulation of the dmPFC reduces the discounting of delayed rewards and decreases the congruency effect in egocentric but not allocentric perspective in the visual perspective-taking tasks. These findings suggest that the dmPFC promotes self-control by inhibiting the egocentric perspective, offering new insights into the neural underpinnings of self-control and perspective-taking, and opening new avenues for interventions targeting disorders characterized by impaired self-regulation.

Is Video Gaming a Cure for Cybersickness? Gamers Experience Less Cybersickness Than Non-Gamers in a VR Self-Motion Task.

Cybersickness remains a major drawback of Virtual Reality (VR) headsets, as a breadth of stationary experiences with visual self-motion can result in visually-induced motion sickness. However, not everybody experiences the same intensity or type of adverse symptoms. Here we propose that prior experience with virtual environments can predict ones degree of cybersickness. Video gaming can enhance visuospatial abilities, which in-turn relate negatively to cybersickness - meaning that consistently engaging in virtual environments can result in protective habituation effects. In a controlled stationary VR experiment, we found that 'VR-naive' video gamers experienced significantly less cybersickness in a virtual tunnel-travel task and outperformed 'VR-naive' non-video gamers on a visual attention task. These findings strongly motivate the use of non-VR games for training VR cybersickness resilience, with future research needed to further understand the mechanism(s) by which gamers become cybersickness resilient - potentially expanding access to VR for even the most susceptible participants.

Measuring Socioeconomic and Stress Disparities in Infant Declarative Memory Using the Visual Paired Comparison Task.

Research suggests that socioeconomic circumstances and stress predict memory skills in adults and older children, yet few studies have addressed this question in infancy. The current study used the visual-paired comparison paradigm to examine whether socioeconomic circumstances, maternal perceived stress, and/or maternal physiological stress, all measured prenatally, predict memory performance among 6-month-old infants. We found no significant associations between infant memory and any measure of socioeconomic circumstance or stress. Potential explanations for these null findings are discussed.

Searching Across Realities: Investigating ERPs and Eye-Tracking Correlates of Visual Search in Mixed Reality.

Mixed Reality allows us to integrate virtual and physical content into users' environments seamlessly. Yet, how this fusion affects perceptual and cognitive resources and our ability to find virtual or physical objects remains uncertain. Displaying virtual and physical information simultaneously might lead to divided attention and increased visual complexity, impacting users' visual processing, performance, and workload. In a visual search task, we asked participants to locate virtual and physical objects in Augmented Reality and Augmented Virtuality to understand the effects on performance. We evaluated search efficiency and attention allocation for virtual and physical objects using event-related potentials, fixation and saccade metrics, and behavioral measures. We found that users were more efficient in identifying objects in Augmented Virtuality, while virtual objects gained saliency in Augmented Virtuality. This suggests that visual fidelity might increase the perceptual load of the scene. Reduced amplitude in distractor positivity ERP, and fixation patterns supported improved distractor suppression and search efficiency in Augmented Virtuality. We discuss design implications for mixed reality adaptive systems based on physiological inputs for interaction.

A learnable motion preserving pooling for action recognition

Using deep neural networks (DNN) for video understanding tasks is expensive in terms of computation cost. Pooling layers in DNN which are widely used in most vision tasks to resize the spatial dimensions play crucial roles in reducing the computation and memory cost. In video-related tasks, pooling layers are also applied, mostly in the spatial dimension only as the standard average pooling in the temporal domain can significantly reduce its performance. This is because conventional temporal pooling degrades the underlying important motion features in consecutive frames. Such a phenomenon is rarely investigated and most state-of-art methods simply do not adopt temporal pooling, leading to enormous computation costs. In this work, we propose a learnable motion-preserving pooling (MPPool) layer that is able to preserve the general motion progression after the pooling. This pooling layer first locates the frames with the strongest motion features and then keeps these crucial features during pooling. Our experiments demonstrate that MPPool not only reduces the computation cost for video data modeling, but also increases the final prediction accuracy on various motion-centric and appearance-centric datasets.

Seismic random noise suppression via mining multi-scale local and global information

Suppressing random noise is critical for revealing real subsurface structures. Convolutional neural networks (CNNs), the leading seismic data denoising methods, excel at extracting local features but struggle to capture global representations. Unet can extract and reuse multi-scale features, aiding in the precise detection of details and semantic information; however, being based on convolutional operations, it struggles to capture global information. To capture global representations, researchers normally employ Transformers in high-level visual tasks, owing to their self-attention mechanisms. This paper introduces a method for mining multi-scale local and global information based on hybrid-gated Unet (HGUnet), which integrates Transformer, CNN, and Unet architectures to enhance the feature representation capability for seismic random noise suppression tasks. HGUnet comprises hybrid-gated blocks (HGB) embedded within a U-shaped architecture, employing a concurrent structure of Octave convolution and lightweight multi-head self-attention mechanism to efficiently extract multi-scale local and global features simultaneously. Moreover, at the conclusion of the HGB, to precisely leverage information and reduce computing costs, a gated feedforward network is designed to retain valuable information and prune redundancies for feature fusion. Synthetic and field experimental results demonstrate that HGUnet improves denoising quality over traditional and CNN methods without adding significant computing costs.

Jack of all trades, master of one: domain-specific and domain-general contributions to perceptual expertise in visual comparison

Perceptual expertise is typically domain-specific and rarely generalises beyond an expert’s domain of experience. Forensic feature-comparison examiners outperform the norm in domain-specific visual comparison, but emerging research suggests that they show advantages on other similar tasks outside their domain of expertise. For example, fingerprint examiners not only outperform novices in fingerprint comparison, but also in face comparison. Yet, the extent to which their skills generalise is poorly understood. In this study, we investigated the generalisability of perceptual expertise amongst forensic examiners by comparing their performance to novices and other examiners within and outside their area of expertise. We recruited 85 experts from three forensic disciplines (face, fingerprint, and firearms) and asked them to complete four different visual comparison tasks: faces, fingerprints, firearms, and novel-objects. Examiners displayed domain-specific expertise: they outperformed novices and other examiners within their domain of visual comparison expertise. Yet, some of their skill also generalised: examiners also outperformed novices outside their area of expertise. However, while individual differences in examiners’ performance within their domain of experience were associated with their performance in a novel comparison task, they were not related to their performance on tasks outside their expert domain. These results provide key insight into the domain-specific and domain-general contributions of forensic examiners’ perceptual expertise. Forensic expertise lends some generalisable skill to other visual comparison tasks, but best performance is still seen within examiners’ domain of expertise.

A Review of Deep Learning Based Object Detection Algorithms

With the rapid development of artificial intelligence technology, the traditional manual feature-based object detection method has been gradually replaced by deep learning-based object detection technology. Object detection is the basis and prerequisite for many computer vision tasks aimed at recognizing targets in an image and determining their class and location. The purpose of object detection algorithm research based on deep learning is to improve the detection accuracy and detection speed of the detection algorithm, so that the object detection algorithm can be more safe and convenient applied to People's Daily production and life. This article reviews the evolution of object detection algorithms, focusing on two-stage, one-stage and object detection algorithms based on the transformer architecture. In addition, the performance, advantages and disadvantages of different algorithms are compared, and the development trend of object detection algorithms based on deep learning is summarized in the end.

An inherently interpretable deep learning model for local explanations using visual concepts.

Over the past decade, deep learning has become the leading approach for various computer vision tasks and decision support systems. However, the opaque nature of deep learning models raises significant concerns about their fairness, reliability, and the underlying inferences they make. Many existing methods attempt to approximate the relationship between low-level input features and outcomes. However, humans tend to understand and reason based on high-level concepts rather than low-level input features. To bridge this gap, several concept-based interpretable methods have been developed. Most of these methods compute the importance of each discovered concept for a specific class. However, they often fail to provide local explanations. Additionally, these approaches typically rely on labeled concepts or learn directly from datasets, leading to the extraction of irrelevant concepts. They also tend to overlook the potential of these concepts to interpret model predictions effectively. This research proposes a two-stream model called the Cross-Attentional Fast/Slow Thinking Network (CA-SoftNet) to address these issues. The model is inspired by dual-process theory and integrates two key components: a shallow convolutional neural network (sCNN) as System-I for rapid, implicit pattern recognition and a cross-attentional concept memory network as System-II for transparent, controllable, and logical reasoning. Our evaluation across diverse datasets demonstrates the model's competitive accuracy, achieving 85.6%, 83.7%, 93.6%, and 90.3% on CUB 200-2011, Stanford Cars, ISIC 2016, and ISIC 2017, respectively. This performance outperforms existing interpretable models and is comparable to non-interpretable counterparts. Furthermore, our novel concept extraction method facilitates identifying and selecting salient concepts. These concepts are then used to generate concept-based local explanations that align with human thinking. Additionally, the model's ability to share similar concepts across distinct classes, such as in fine-grained classification, enhances its scalability for large datasets. This feature also induces human-like cognition and reasoning within the proposed framework.

Plaque phenotyping using deep learning in optical coherence tomography imaging

Abstract Background Artificial intelligence (AI) techniques have emerged as powerful tools in various computer vision tasks, particularly in classification, detection, and segmentation. Purpose This study aimed to develop a deep learning model for categorizing optical coherence tomography (OCT) frames into different plaque phenotypes, including thin-cap fibroatheroma (TCFA) and fibroatheroma. Methods A prospective multicenter observational study enrolled 168 patients who underwent percutaneous coronary intervention (PCI) and received OCT imaging for non-target lesions with angiographically estimated diameter stenosis greater than 30% (177 vessels). Among them, 129 patients (135 vessels) underwent follow-up OCT imaging at 1 year. A total of 45,072 OCT frames were randomly divided into training and validation sets in a 4:1 ratio. An EfficientNet-B3 convolutional neural network model was developed to classify OCT frames into TCFA, thick-cap fibroatheroma, non-fibroatheroma, adaptive intimal thickening, and normal vessel categories. Results A confusion matrix comparing manually assessed plaque phenotypes with AI-predicted phenotypes was presented in Table. In the validation datasets (9,015 frames), the overall accuracy of the model was 85.1%. For predicting the presence of TCFA, sensitivity was 81.1%, specificity was 98.8%, positive predictive value was 63.9%, and negative predictive value was 99.5%. Regarding the presence of fibroatheroma, sensitivity was 90.3%, specificity was 89.4%, positive predictive value was 93.1%, and negative predictive value was 85.4%. Conclusions The deep learning algorithm demonstrated high accuracy in detecting TCFA and fibroatheroma with excellent reproducibility. This automated process enables operators to promptly identify high-risk lesions during PCI, potentially leading to improved clinical outcomes.

Learning differentiable logic programs for abstract visual reasoning

AbstractVisual reasoning is essential for building intelligent agents that understand the world and perform problem-solving beyond perception. Differentiable forward reasoning has been developed to integrate reasoning with gradient-based machine learning paradigms. However, due to the memory intensity, most existing approaches do not bring the best of the expressivity of first-order logic, excluding a crucial ability to solve abstract visual reasoning, where agents need to perform reasoning by using analogies on abstract concepts in different scenarios. To overcome this problem, we propose NEUro-symbolic Message-pAssiNg reasoNer (NEUMANN), which is a graph-based differentiable forward reasoner, passing messages in a memory-efficient manner and handling structured programs with functors. Moreover, we propose a computationally-efficient structure learning algorithm to perform explanatory program induction on complex visual scenes. To evaluate, in addition to conventional visual reasoning tasks, we propose a new task, visual reasoning behind-the-scenes, where agents need to learn abstract programs and then answer queries by imagining scenes that are not observed. We empirically demonstrate that NEUMANN solves visual reasoning tasks efficiently, outperforming neural, symbolic, and neuro-symbolic baselines.

Technology-assisted instruction with teacher prompts on fraction multiplication word problems: A single-case design with visual analysis and Bayesian multilevel modeling

ABSTRACT This study examines the effects of technology-assisted instruction with teacher prompts on the ability to visualize and solve fraction multiplication word problems for four middle school students with learning disabilities. A multiple-probe design across participants, a type of single-case design, showed a functional relationship between the intervention and the targeted mathematical outcomes. Although there were some fluctuations over time, students demonstrated improvements from the baseline to intervention phases (Tau ranged from 0.76 to 1.00 for visualization and was 1.00 for problem-solving). Researchers employed Bayesian cumulative link mixed effects models to examine the moderating effects of word problem question types. Students showed greater maintenance effects on problem-solving than on visualization tasks, as reflected in changes in level (logit coefficient = 2.6) and trend (logit coefficient = 0.22). Students and teachers perceived vocabulary and multiplication fact practices, as well as the cognitive and metacognitive features embedded in the technology-assisted intervention, to be useful in learning targeted mathematics concepts. This study underscores the role of technology integration in enhancing teachers’ instructional approach and in aiding students’ acquisition and retention of mathematical concepts and skills.

ViCo: Human visual perception-based contour extraction of Chinese Tang dynasty textile patterns

The study of ancient textile patterns not only enriches historical material but also inspires modern costume art design. Recent developments in computer vision technology provide novel analytical methods for the study of ancient textile patterns. As contours are one of the most significant features in computer vision tasks, contour extraction serves as a prerequisite to accomplishing intelligent understanding and automatic design of textile patterns. In this paper, a perception-consistent contour extraction method (ViCo) that simulates the human visual system from the retina, the lateral geniculate nucleus (LGN), to the V1 area in the cortex is proposed, along with an ancient textile pattern dataset represented by TuanKe in the Chinese Tang dynasty. Experiments were conducted both qualitatively and quantitatively to verify the effectiveness of the proposed method on the TuanKe dataset. Analysis of the results implies the effectiveness of ViCo because the retinal and LGN filter enhances edge information and restrains tiny pixel changes, and the cortex simulation is direction selective. This study is among the initial efforts to employ computer vision technology for ancient textile pattern studies. The study presents a TuanKe dataset and a novel bio-inspired contour extraction method, which has significant historical research importance, fabric art design inspiration, and computer application value.

Physics-Driven Image Dehazing from the Perspective of Unmanned Aerial Vehicles

Drone vision is widely used in change detection, disaster response, and military reconnaissance due to its wide field of view and flexibility. However, under haze and thin cloud conditions, image quality is usually degraded due to atmospheric scattering. This results in issues like color distortion, reduced contrast, and lower clarity, which negatively impact the performance of subsequent advanced visual tasks. To improve the quality of unmanned aerial vehicle (UAV) images, we propose a dehazing method based on calibration of the atmospheric scattering model. We designed two specialized neural network structures to estimate the two unknown parameters in the atmospheric scattering model: the atmospheric light intensity A and medium transmission t. However, calculation errors always occur in both processes for estimating the two unknown parameters. The error accumulation for atmospheric light and medium transmission will cause the deviation in color fidelity and brightness. Therefore, we designed an encoder-decoder structure for irradiance guidance, which not only eliminates error accumulation but also enhances the detail in the restored image, achieving higher-quality dehazing results. Quantitative and qualitative evaluations indicate that our dehazing method outperforms existing techniques, effectively eliminating haze from drone images and significantly enhancing image clarity and quality in hazy conditions. Specifically, the compared experiment on the R100 dataset demonstrates that the proposed method improved the peak signal-to-noise ratio (PSNR) and structure similarity index measure (SSIM) metrics by 6.9 dB and 0.08 over the second-best method, respectively. On the N100 dataset, the method improved the PSNR and SSIM metrics by 8.7 dB and 0.05 over the second-best method, respectively.

Convergent representation of values from tactile and visual inputs for efficient goal-directed behavior in the primate putamen

Animals can discriminate diverse sensory values with a limited number of neurons, raising questions about how the brain utilizes neural resources to efficiently process multi-dimensional inputs for decision-making. Here, we demonstrate that this efficiency is achieved by reducing sensory dimensions and converging towards the value dimension essential for goal-directed behavior in the putamen. Humans and monkeys performed tactile and visual value discrimination tasks while their neural responses were examined. Value information, whether originating from tactile or visual stimuli, was found to be processed within the human putamen using fMRI. Notably, at the single-neuron level in the macaque putamen, half of the individual neurons encode values independently of sensory inputs, while the other half selectively encode tactile or visual value. The responses of bimodal value neurons correlate with value-guided finger insertion behavior in both tasks, whereas modality-selective value neurons show task-specific correlations. Simulation using these neurons reveals that the presence of bimodal value neurons enables value discrimination with a significantly reduced number of neurons compared to simulations without them. Our data indicate that individual neurons in the primate putamen process different values in a convergent manner, thereby facilitating the efficient use of constrained neural resources for value-guided behavior.

Increasing transparency of computer-aided detection impairs decision-making in visual search.

Recent developments in artificial intelligence (AI) have led to changes in healthcare. Government and regulatory bodies have advocated the need for transparency in AI systems with recommendations to provide users with more details about AI accuracy and how AI systems work. However, increased transparency could lead to negative outcomes if humans become overreliant on the technology. This study investigated how changes in AI transparency affected human decision-making in a medical-screening visual search task. Transparency was manipulated by either giving or withholding knowledge about the accuracy of an 'AI system'. We tested performance in seven simulated lab mammography tasks, in which observers searched for a cancer which could be correctly or incorrectly flagged by computer-aided detection (CAD) 'AI prompts'. Across tasks, the CAD systems varied in accuracy. In the 'transparent' condition, participants were told the accuracy of the CAD system, in the 'not transparent' condition, they were not. The results showed that increasing CAD transparency impaired task performance, producing an increase in false alarms, decreased sensitivity, an increase in recall rate, and a decrease in positive predictive value. Along with increasing investment in AI, this research shows that it is important to investigate how transparency of AI systems affect human decision-making. Increased transparency may lead to overtrust in AI systems, which can impact clinical outcomes.

An age-group ranking model for facial age estimation

Age prediction has become an important Computer Vision task. Although this task requires the age of an individual to be predicted from a given face, research has shown that it is more intuitive and easier for humans to decide which of two individuals is older than to decide how old an individual is. This work follows this intuition to aid the age prediction of a face by exploiting the age information available from other faces. It goes further to explore the statistical relationships between facial features within age groups to compute age-group ranks for a given face. The resulting age-group rank is low-dimensional and age-discriminatory, thus improving age prediction accuracy when fed into an age predictor. Experiments on publicly available facial ageing datasets (FGnet, PAL, and Adience) reveal the effectiveness of the proposed age-group ranking model when used with traditional Machine learning algorithms as well as Deep Learning algorithms. Cross-dataset validation, a method of training and testing on entirely different datasets, was also employed to further investigate the effectiveness of this method.

Point-MPP: Point Cloud Self-Supervised Learning From Masked Position Prediction.

Masked autoencoding has gained momentum for improving fine-tuning performance in many downstream tasks. However, it tends to focus on low-level reconstruction details, lacking high-level semantics and resulting in weak transfer capability. This article presents a novel jigsaw puzzle solver inspired by the idea that predicting the positions of disordered point cloud patches provides more semantic information, similar to how children learn by solving jigsaw puzzles. Our method adopts the mask-then-predict paradigm, erasing the positions of selected point patches rather than their contents. We first partition input point clouds into irregular patches and randomly erase the positions of some patches. Then, a Transformer-based model is used to learn high-level semantic features and regress the positions of the masked patches. This approach forces the model to focus on learning transfer-robust semantics while paying less attention to low-level details. To tie the predictions within the encoding space, we further introduce a consistency constraint on their latent representations to encourage the encoded features to contain more semantic cues. We demonstrate that a standard Transformer backbone with our pretraining scheme can capture discriminative point cloud semantic information. Furthermore, extensive experiments indicate that our method outperforms the previous best competitor across six popular downstream vision tasks, achieving new state-of-the-art performance. Codes will be available at https://git.openi.org.cn/OpenPointCloud/Point-MPP.

Individual Differences in the Impact of Distracting Environmental Sounds on the Performance of a Continuous Visual Task in Older Adults

Background/Objectives: Vulnerability to sound distraction is commonly reported in older adults with dementia and tends to be associated with adverse impacts on daily activity. However, study outcome heterogeneity is increasingly evident, with preserved resistance to distraction also evident. Contributory factors may include individual differences in distractibility in older adulthood per se, and failure to consider the influence of how difficult a person found the test. Methods: We therefore measured distractibility in a group of older adults by comparing the performance of a primary visual task (Swansea Test of Attentional Control), which includes an adaptive algorithm to take into account how difficult a person finds the test under both no-sound and sound conditions. Results: Analysis revealed no significant difference in group mean performance between no-sound versus sound conditions [t (33) = 0.181, p = 0.858; Cohen’s effect size d = −0.028], but individual differences in performance both within and between sound and no-sound conditions were evident, indicating that for older adults, distracting sounds can be neutral, detrimental, or advantageous with respect to visual task performance. It was not possible to determine individual thresholds for whether sound versus no-sound conditions affected a person’s actual behaviour. Conclusions: Nevertheless, our findings indicate how variable such effects may be in older adults, which in turn may help to explain outcome heterogeneity in studies including people living with dementia. Furthermore, such within-group heterogeneity highlights the importance of considering a person’s individual performance in order to better understand their behaviour and initiate interventions as required.