Visual Task Research Articles

SGNet: Efficient Snow Removal Deep Network with a Global Windowing Transformer

Image restoration under adverse weather conditions poses a challenging task. Previous research efforts have predominantly focused on eliminating rain and fog phenomena from images. However, snow, being another common atmospheric occurrence, also significantly impacts advanced computer vision tasks such as object detection and semantic segmentation. Recently, there has been a surge of methods specifically targeting snow removal, with the majority employing visual Transformers as the backbone network to enhance restoration effectiveness. Nevertheless, due to the quadratic computations required by Transformers to model long-range dependencies, this significantly escalates the time and space consumption of deep learning models. To address this issue, this paper proposes an efficient snow removal Transformer with a global windowing network (SGNet). This method forgoes the localized windowing strategy of previous visual Transformers, opting instead to partition the image into multiple low-resolution subimages containing global information using wavelet sampling, thereby ensuring higher performance while reducing computational overhead. Extensive experimentation demonstrates that our approach achieves outstanding performance across a wide range of benchmark datasets and can rival methods employing CNNs in terms of computational cost.

Towards robust image matching in low-luminance environments: Self-supervised keypoint detection and descriptor-free cross-fusion matching

Image keypoint detection and feature matching are fundamental steps in computer vision tasks. However, variations in environment, time, and viewpoint pose a challenge to the stability of image keypoint detection and matching. Most traditional and deep learning-based methods cannot accurately and efficiently extract highly repeatable keypoints and robust match pairs in low-luminance environments. Therefore, we propose a two-step ‘detection + matching’ framework, which consists of deep neural networks in each step. Firstly, we design a self-supervised robust keypoint detection network, which utilizes multi-scale, multi-angle, and multi-luminance transformation techniques to create pseudo-labeled datasets to improve the model’s keypoint detection repeatability and luminance invariance. Secondly, we propose a descriptor-free cross-fusion matching network, which uses the cross-fusion attention mechanism to establish connections between keypoint-centered image patches and converts the feature-matching task into an image patch assignment task to improve the accuracy and efficiency of matching. Thirdly, the proposed framework is used to replace traditional SIFT in SfM. Experimental results on testing datasets show that the self-supervised robust keypoint detection network achieves higher keypoint repeatability in low luminance environments compared to SIFT, ORB, LIFT, and Superpoint. The descriptor-free cross-fusion matching network’s mean matching accuracy and efficiency are higher than the mainstream Superglue algorithm. Also, SfM achieves better performance regarding the number of sparse point clouds and accuracy.

Sensitivity to Subphonemic Differences in First Language Predicts Vocabulary Size in a Foreign Language

AbstractSome listeners exhibit higher sensitivity to subphonemic acoustic differences (i.e., higher speech gradiency). Here, we asked whether higher gradiency in a listener's first language (L1) facilitates foreign language learning and explored the possible sources of individual differences in L1 gradiency. To address these questions, we tested 164 native Spanish speakers with different linguistic profiles. Speech gradiency was assessed via a Visual Analogue Scale task, and foreign language proficiency was assessed via an English vocabulary test. Possible sources of gradiency included domain‐general auditory acuity, overall exposure to spoken language (indexed by age), and exposure to phonological diversity. Control measures were collected to account for variables such as phoneme categorization consistency, working memory, and musical training. The results revealed a positive link between L1 speech gradiency and vocabulary acquisition in a foreign language over and above all other variables. L1 speech gradiency itself was predicted by domain‐general auditory acuity and overall exposure to spoken language.

Camera calibration optimization algorithm that uses a step function

Camera calibration is very important when planning machine vision tasks. Calibration may involve 3D reconstruction, size measurement, or careful target positioning. Calibration accuracy directly affects the accuracy of machine vision. The parameters in many image distortion models are usually applied to all image pixels. However, this may be associated with rather high pixel reprojection errors at image edges, compromising camera calibration accuracy. In this paper, we present a new camera calibration optimization algorithm that features a step function that splits images into center and edge regions. First, based on the increasing pixel reprojection errors according to the pixel distance away from the image center, we gave a flexible method to divide an image into two regions, center and boundary. Then, the algorithm automatically determines the step position, and the calibration model is rebuilt. The new model can calibrate the distortions at the center and boundary regions separately. Optimized by the method, the number of distortion parameters in the old model is doubled, and different parameters represent different distortions within two regions. In this way, our method can optimize traditional calibration models, which define a global model to describe the distortion of the whole image and get a higher calibration accuracy. Experimentally, the method significantly improved pixel reprojection accuracy, particularly at image edges. Simulations revealed that our method was more flexible than traditional methods.

Image Segmentation Technology Based on Ant Colony Algorithm

Image segmentation is a key task in computer vision, with applications ranging from medical diagnosis to autonomous driving. The Ant Colony Algorithm (ACO), modeled after ant foraging behavior, has emerged as a viable segmentation methodology. However, ACO-based segmentation algorithms frequently generate segmented outputs with jagged or uneven boundaries, which reduces their interpretability and usability. To alleviate this problem, they study the use of boundary-smoothing approaches in ACO-based segmentation. In this paper, they investigate image segmentation technology based on the Ant Colony Algorithm, with a focus on border smoothing. They examine the fundamentals of ACO and its application to image segmentation, emphasizing its strengths and limits. They also look at several boundary smoothing strategies, such as morphological operations, edge-preserving filters, and active contours (snakes), and how they affect segmentation performance. Through experimental validation and comparative analysis, they show that boundary smoothing improves the accuracy and visual quality of segmented images produced by ACO-based segmentation algorithms. These results help to design more robust and visually appealing segmentation algorithms, which have potential applications in medical imaging, remote sensing, and industrial automation.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison between NeRF-based and conventional approaches in the scope of this work.

Hybrid Transfer Learning and Support Vector Machine Models for Asphalt Pavement Distress Classification

Pavement condition evaluation plays a crucial role in assisting with the management of the highway infrastructure. However, the current methods used for assessing pavement conditions are costly, time-consuming, and subjective. There is a growing need to automate these assessment tactics and leverage low-cost technologies to enable widespread deployment. This study aims to develop robust and highly accurate models for classifying asphalt pavement distresses using transfer learning (TL) techniques based on pretrained deep learning (DL) networks. This topic has gained considerable attention in the field since 2015 when DL became the mainstream choice for various computer vision tasks. While progress has been made in TL model development, challenges persist in areas of accuracy, repeatability, and training cost. To tackle these challenges, this study proposes hybrid models that combine DL networks with support vector machines (SVMs). Three strategies were evaluated: single DL models using transfer learning (TLDL), hybrid models combining DL and SVM (DL+SVM), and hybrid models combining TLDL and SVM (TLDL+SVM). The performance of each strategy was assessed using statistical metrics based on the confusion matrix. Results consistently showed that the TLDL+SVM strategy outperformed the other approaches in accuracy and F1 scores, regardless of the DL network type. On average, the hybrid models achieved an accuracy of 95%, surpassing the 80% accuracy of the best single model and the 55% accuracy for DL+SVM without TL. The results clearly indicate that employing transfer-learned models as feature extractors, in combination with SVM as the classifier, consistently achieves exceptional performance.

Insights into the relationship between eye movements and personality traits in restricted visual fields

Previous studies have suggested behavioral patterns, such as visual attention and eye movements, relate to individual personality traits. However, these studies mainly focused on free visual tasks, and the impact of visual field restriction remains inadequately understood. The primary objective of this study is to elucidate the patterns of conscious eye movements induced by visual field restriction and to examine how these patterns relate to individual personality traits. Building on previous research, we aim to gain new insights through two behavioral experiments, unraveling the intricate relationship between visual behaviors and individual personality traits. As a result, both Experiment 1 and Experiment 2 revealed differences in eye movements during free observation and visual field restriction. Particularly, simulation results based on the analyzed data showed clear distinctions in eye movements between free observation and visual field restriction conditions. This suggests that eye movements during free observation involve a mixture of conscious and unconscious eye movements. Furthermore, we observed significant correlations between conscious eye movements and personality traits, with more pronounced effects in the visual field restriction condition used in Experiment 2 compared to Experiment 1. These analytical findings provide a novel perspective on human cognitive processes through visual perception.

Modality-specific impacts of distractors on visual and auditory categorical decision-making: an evidence accumulation perspective.

Our brain constantly processes multisensory inputs to make decisions and guide behaviors, but how goal-relevant processes are influenced by irrelevant information is unclear. Here, we investigated the effects of intermodal and intramodal task-irrelevant information on visual and auditory categorical decision-making. In both visual and auditory tasks, we manipulated the modality of irrelevant inputs (visual vs. auditory vs. none) and used linear discrimination analysis of EEG and hierarchical drift-diffusion modeling (HDDM) to identify when and how task-irrelevant information affected decision-relevant processing. The results revealed modality-specific impacts of irrelevant inputs on visual and auditory categorical decision-making. The distinct effects on the visual task were shown on the neural components, with auditory distractors amplifying the sensory processing whereas visual distractors amplifying the post-sensory process. Conversely, the distinct effects on the auditory task were shown in behavioral performance and underlying cognitive processes. Visual distractors facilitate behavioral performance and affect both stages, but auditory distractors interfere with behavioral performance and impact on the sensory processing rather than the post-sensory decision stage. Overall, these findings suggested that auditory distractors affect the sensory processing stage of both tasks while visual distractors affect the post-sensory decision stage of visual categorical decision-making and both stages of auditory categorical decision-making. This study provides insights into how humans process information from multiple sensory modalities during decision-making by leveraging modality-specific impacts.

AD-DUNet: A dual-branch encoder approach by combining axial Transformer with cascaded dilated convolutions for liver and hepatic tumor segmentation

Liver cancer remains a significant health concern, and accurate segmentation in CT scans is crucial for diagnosis and treatment. Deep learning-based auxiliary diagnosis techniques, especially utilizing U-shaped structures, are widely employed in medical image segmentation. However, traditional methods that utilize Convolutional Neural Networks (CNNs) generally have limitations in modeling long-range dependencies. Inspired by the success of Transformers in various vision tasks, approaches that combine Transformers with CNNs have been spurred. However, many existing hybrid CNN-Transformer models are prone to yielding poor performance on relative small-scale medical image datasets when trained from scratch. Moreover, some of these methods involve additional fusion modules customized, which introduce extra workload and parameters to the model. To address these limitations, we propose AD-DUNet, a hybrid CNN-Transformer model for liver and hepatic tumor segmentation, which comprises a dual-branch encoder and a residual decoder. The Transformer-based encoder, utilizing Axial Transformer (AT) blocks, efficiently captures long-range dependencies across the entire image, while the CNN-based encoder, constructed with cascaded dilated convolutions (CDC) blocks, extracts fine-grained local features. The two encoders synergize in the shared residual decoder, eliminating the need for additional fusion modules. The extensive experiments conducted on the LiTS2017 and 3DIRCAD datasets demonstrate the superiority of AD-DUNet over existing models. Remarkably, our approach achieves state-of-the-art results without relying on pre-trained weights, showcasing its efficiency with low complexity and 4.24M parameters.

Speech, Nonspeech Audio, and Visual Interruptions of a Tracking Task: A Replication and Extension of Nees and Sampsell (2021)

Interruptions from technology—such as alerts from mobile communication devices—are a pervasive aspect of modern life. Interruptions can be detrimental to performance of the ongoing, interrupted task. Designers often can choose whether interruptions are delivered as visual or auditory alerts. Contradictory theories have emerged regarding whether auditory or visual alerts are more harmful to performance of ongoing visual tasks. Multiple Resources Theory predicts better overall performance with auditory alerts, but Auditory Preemption Theory predicts better overall performance with visual alerts. Nees and Sampsell previously found that multitasking was superior with nonspeech auditory alerts as compared to visual alerts. In the current experiment, their methods were replicated and extended to include a speech auditory alerts condition. Performance of the ongoing tracking task was worse with interruption from visual alerts, and perceived workload also was highest in this condition. Reaction time to alerts was fastest with visual alerts. There also was converging evidence to suggest that performance with speech alerts was superior to performance with nonspeech tonal alerts. The current experiment replicated the results of Nees and Sampsell and extended their findings to speech alert sounds. Like in their study, the pattern of findings here support Multiple Resources Theory over Auditory Preemption Theory.

Learning of the same task subserved by substantially different mechanisms between patients with body dysmorphic disorder and healthy individuals.

It has remained unclear whether individuals with psychiatric disorders involving altered visual processing employ similar neuronal mechanisms during perceptual learning of a visual task. We investigated this question by training patients with body dysmorphic disorder, a psychiatric disorder characterized by distressing or impairing preoccupation with nonexistent or slight defects in one's physical appearance, and healthy controls on a visual detection task for human faces with low spatial frequency components. Brain activation during task performance was measured with functional magnetic resonance imaging before the beginning and after the end of behavioral training. Both groups of participants improved performance on the trained task to a similar extent. However, neuronal changes in the fusiform face area were substantially different between groups such that activation for low spatial frequency faces in the right fusiform face area increased after training in body dysmorphic disorder patients but decreased in controls. Moreover, functional connectivity between left and right fusiform face area decreased after training in patients but increased in controls. Our results indicate that neuronal mechanisms involved in perceptual learning of a face detection task differ fundamentally between body dysmorphic disorder patients and controls. Such different neuronal mechanisms in body dysmorphic disorder patients might reflect the brain's adaptations to altered functions imposed by the psychiatric disorder.

Stimulating prefrontal cortex facilitates training transfer by increasing representational overlap.

A recent hypothesis characterizes difficulties in multitasking as being the price humans pay for our ability to generalize learning across tasks. The mitigation of these costs through training has been associated with reduced overlap of constituent task representations within frontal, parietal, and subcortical regions. Transcranial direct current stimulation, which can modulate functional brain activity, has shown promise in generalizing performance gains when combined with multitasking training. However, the relationship between combined transcranial direct current stimulation and training protocols with task-associated representational overlap in the brain remains unexplored. Here, we paired prefrontal cortex transcranial direct current stimulation with multitasking training in 178 individuals and collected functional magnetic resonance imaging data pre- and post-training. We found that 1mA transcranial direct current stimulation applied to the prefrontal cortex paired with multitasking training enhanced training transfer to spatial attention, as assessed via a visual search task. Using machine learning to assess the overlap of neural activity related to the training task in task-relevant brain regions, we found that visual search gains were predicted by changes in classification accuracy in frontal, parietal, and cerebellar regions for participants that received left prefrontal cortex stimulation. These findings demonstrate that prefrontal cortex transcranial direct current stimulation may interact with training-related changes to task representations, facilitating the generalization of learning.

Visuospatial processing in early brain-based visual impairment is associated with differential recruitment of dorsal and ventral visual streams.

Visuospatial processing impairments are prevalent in individuals with cerebral visual impairment (CVI) and are typically ascribed to "dorsal stream dysfunction" (DSD). However, the contribution of other cortical regions, including early visual cortex (EVC), frontal cortex, or the ventral visual stream, to such impairments remains unknown. Thus, here, we examined fMRI activity in these regions, while individuals with CVI (and neurotypicals) performed a visual search task within a dynamic naturalistic scene. First, behavioral performance was measured with eye tracking. Participants were instructed to search and follow a walking human target. CVI participants took significantly longer to find the target, and their eye gaze patterns were less accurate and less precise. Second, we used the same task in the MRI scanner. Along the dorsal stream, activation was reduced in CVI participants, consistent with the proposed DSD in CVI. Intriguingly, however, visual areas along the ventral stream showed the complete opposite pattern, with greater activation in CVI participants. In contrast, we found no differences in either EVC or frontal cortex between groups. These results suggest that the impaired visuospatial processing abilities in CVI are associated with differential recruitment of the dorsal and ventral visual streams, likely resulting from impaired selective attention.

Association between physical health and neurocognition in first-episode schizophrenia

IntroductionImpaired cognition is a core feature of schizophrenia that is evident early in the first episode and is frequently accompanied by compromised physical health. Although physical health confers benefits to cognition, it remains unclear whether physical activity, body mass index (BMI) and cardiorespiratory fitness are associated with neurocognition in first episode schizophrenia patients. The purpose of this study was to examine differences in stimulus categorization and motor response selection processes between first-episode schizophrenia patients compared to age-matched controls and explore associations between physical health and these stages of information processing.MethodsFourteen young adult patients receiving care following a first episode of psychosis and a matched sample of nonpsychiatric controls completed a visual oddball task from which the P3 and LRP (lateralized readiness potential) event-related potential (ERP) components were extracted to assess stimulus categorization and response selection processes, respectively. Physical activity, aerobic fitness, and BMI were correlated with ERP measures.ResultsCompared with controls, patients had lower physical activity levels and longer P3 and LRP latencies. Regardless of stimulus probability, patients had reduced accuracy and slower reaction times relative to controls. In patients, marginal associations were found between physical activity and P3 difference waveform amplitude, and BMI was negatively associated with parent P3 waveform amplitude.DiscussionThe present findings suggest that cognitive impairment in first-episode schizophrenia spans both stimulus- and response-related stages of information processing, and may be targeted through physical activity interventions.

Uncertainty-modulated attentional capture: Outcome variance increases attentional priority.

Our prior experiences shape the way that we prioritize information from the environment for further processing, analysis, and action. We show in three experiments that this process of attentional prioritization is critically modulated by the degree of uncertainty in these previous experiences. Participants completed a visual search task in which they made a saccade to a target to earn a monetary reward. The color of a color-singleton distractor in the search array signaled the reward outcome(s) that were available, with different degrees of variance (uncertainty). Participants were never required to look at the colored distractor, and doing so would slow their response to the target. Nevertheless, across all experiments, participants were more likely to look at distractors associated with high outcome variance versus low outcome variance. This pattern was observed when all distractors had equal expected value (Experiment 1), when the difference in variance was opposed by a difference in expected value (i.e., the high-variance distractor had a low expected value, and vice versa: Experiment 2), and when high- and low-variance distractors were paired with the maximum-value outcome on an equal proportion of trials (Experiment 3). Our findings demonstrate that experience of prediction error plays a fundamental role in guiding "attentional exploration," wherein priority is driven by the potential for a stimulus to reduce future uncertainty through a process of learning, as opposed to maximizing current information gain. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Transfer Learning-Based Models for Comparative Evaluation for the Detection of AI-Generated Images

As the pace of artificial intelligence (AI) evolution accelerates, the line separating authentic from AI-produced imagery becomes increasingly indistinct. This shift carries profound consequences for sectors such as content verification and digital investigation, underscoring the need for proficient AI-generated image identification systems. Our study utilizes established architectures like AlexNet, Convolutional Neural Networks (CNNs), and VGG16 to explore and evaluate the effectiveness of models based on transfer learning for spotting AI-crafted images. Transfer learning, which applies models pre-trained on large datasets, has proven beneficial in numerous computer vision tasks. In this research, we modify the intricate patterns recognized by AlexNet, CNNs, and VGG16 from extensive datasets to specifically target the detection of AI-generated content. We introduce models that are trained, validated, and tested on a comprehensive dataset that includes both real and AI-generated images. Our experimental findings demonstrate the utility of transfer learning methods in discerning between real and synthetic visuals. By conducting a comparative analysis, we highlight the comparative advantages and limitations of each model in terms of metrics such as precision, recall, accuracy, and the F1-score. Further, we investigate the distinct features identified by each model to elucidate their contribution to accurate classification.

Auditory change detection and visual selective attention: association between MMN and N2pc.

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

Category-based depth incorporation for salient object ranking

Salient object ranking is a visual task that aims to mimic the human visual system’s ability to prioritize various salient objects in complex scenes. It integrates various computer vision tasks, including object detection and instance segmentation, and draws insights from psychology and biology. Despite recent advancements in salient object ranking research, there is still a need for a more in-depth exploration of certain underlying factors that influence the ranking. Previous studies have primarily focused on factors such as the scale and position of objects, as well as interactions between objects and their context. However, they often overlooked the crucial interaction between objects and observers. Unlike other visual tasks, salient object ranking is highly subjective and influenced by the observer. Therefore, establishing a meaningful connection between the observed object and the observer becomes crucial. Our model addresses this gap by placing emphasis on the distance between objects and observers. It investigates the implicit relationship between object categories and distance through the Category-Aware Attention (CAA) module. This innovative approach incorporates depth into salient object ranking, resulting in an improvement in ranking performance.

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

Vision Transformer and its variants have demonstrated great potential in various computer vision tasks. However conventional vision transformers often focus on global dependency at a coarse level, which results in a learning challenge on global relationships and fine-grained representation at a token level. In this paper, we introduce Multi-scale Attention Fusion into transformer (MAFormer), which explores local aggregation and global feature extraction in a dual-stream framework for visual recognition. We develop a simple but effective module to explore the full potential of transformers for visual representation by learning fine-grained and coarse-grained features at a token level and dynamically fusing them. Our Multi-scale Attention Fusion (MAF) block consists of: i) a local window attention branch that learns short-range interactions within windows, aggregating fine-grained local features; ii) global feature extraction through a novel Global Learning with Down-sampling (GLD) operation to efficiently capture long-range context information within the whole image; iii) a fusion module that self-explores the integration of both features via attention. Our MAFormer achieves state-of-the-art results on several common vision tasks. In particular, MAFormer-L achieves 85.9% Top-1 accuracy on ImageNet, surpassing CSWin-B and LV-ViT-L by 1.7% and 0.6% respectively. On MSCOCO, MAFormer outperforms the prior art CSWin by 1.7% mAPs on object detection and 1.4% on instance segmentation with similar-sized parameters. With the performance, MAFormer demonstrates the ability to generalize across various visual benchmarks and prospects as a general backbone for different self-supervised pre-training tasks in the future.