Visual Task Research Articles

Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration.

The deep unfolding approach has attracted significant attention in computer vision tasks, which well connects conventional image processing modeling manners with more recent deep learning techniques. Specifically, by establishing a direct correspondence between algorithm operators at each implementation step and network modules within each layer, one can rationally construct an almost "white box" network architecture with high interpretability. In this architecture, only the predefined component of the proximal operator, known as a proximal network, needs manual configuration, enabling the network to automatically extract intrinsic image priors in a data-driven manner. In current deep unfolding methods, such a proximal network is generally designed as a CNN architecture, whose necessity has been proven by a recent theory. That is, CNN structure substantially delivers the translational symmetry image prior, which is the most universally possessed structural prior across various types of images. However, standard CNN-based proximal networks have essential limitations in capturing the rotation symmetry prior, another universal structural prior underlying general images. This leaves a large room for further performance improvement in deep unfolding approaches. To address this issue, this study makes efforts to suggest a high-accuracy rotation equivariant proximal network that effectively embeds rotation symmetry priors into the deep unfolding framework. Especially, we deduce, for the first time, the theoretical equivariant error for such a designed proximal network with arbitrary layers under arbitrary rotation degrees. This analysis should be the most refined theoretical conclusion for such error evaluation to date and is also indispensable for supporting the rationale behind such networks with intrinsic interpretability requirements. Through experimental validation on different vision tasks, including blind image super-resolution, medical image reconstruction, and image de-raining, the proposed method is validated to be capable of directly replacing the proximal network in current deep unfolding architecture and readily enhancing their state-of-the-art performance. This indicates its potential usability in general vision tasks.

The effects of experimental pain on episodic memory and its top-down modulation: a preregistered pooled analysis.

Pain can automatically interfere with ongoing cognitive processes such as attention and memory. The extent of pain's negative effects on cognitive functioning seems to depend on a balance between top-down and bottom-up factors. In this large, preregistered, pooled reanalysis of 8 studies, we investigated the robustness of the detrimental effect of acute pain on recognition memory and whether top-down mechanisms such as pain-related expectations or cognitions (pain-related fear, pain catastrophizing) modulate this effect. Two hundred forty-seven healthy participants underwent similar experimental paradigms, including a visual categorization task with images randomly paired with (or without) concomitant painful stimulation and a subsequent unannounced recognition task. Recognition memory (ie, d', recollection, and familiarity) and categorization performance (ie, reaction time, accuracy) served as proxies for the effect of pain on cognitive performance. Acute painful stimulation significantly impaired recognition performance (d', familiarity). However, recognition performance was not significantly modulated by participants' expectations regarding the effect of pain on task performance or pain-related cognitions in this sample of healthy participants. Our results corroborate the negative effects of pain on (visual) memory encoding reported in previous studies and reports of "memory problems" from patients with chronic pain. To characterize the role of bottom-up and top-down factors for the detrimental effects of pain, large-scale studies with more nuanced study designs are necessary. Future studies in patient cohorts must unravel the interaction of maladaptive pain-related cognitions and the often-reported impaired cognitive performance in chronic pain patients.

Trial history contributes to the optimal tuning of attention.

In visual search tasks, targets are difficult to find when they are similar to the surrounding nontargets. In this scenario, it is optimal to tune attention to target features that maximize the difference between target and nontargets. We investigated whether the optimal tuning of attention is driven by biases arising from previously attended stimuli (i.e., trial history). Consistent with the effects of trial history, we found that optimal tuning was stronger when a single target-nontarget relation was repeated than when two target-nontarget relations alternated randomly. Detailed analysis of blocks with random alternation showed that optimal tuning was stronger when the target-nontarget relation probed in the current trial matched the relation in the previous trial. We evaluated several mechanisms that may underlie the effects of trial history, such as priming of attentional set, switch costs, and sensory adaptation. However, none of the accounts was able to fully account for the pattern of results. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Action Planning Renders Objects in Working Memory More Attentionally Salient.

A rapidly growing body of work suggests that visual working memory (VWM) is fundamentally action oriented. Consistent with this, we recently showed that attention is more strongly biased by VWM representations of objects when we plan to act on those objects in the future. Using EEG and eye tracking, here, we investigated neurophysiological correlates of the interactions between VWM and action. Participants (n = 36) memorized a shape for a subsequent VWM test. At test, a probe was presented along with a secondary object. In the action condition, participants gripped the actual probe if it matched the memorized shape, whereas in the control condition, they gripped the secondary object. Crucially, during the VWM delay, participants engaged in a visual selection task, in which they located a target as fast as possible. The memorized shape could either encircle the target (congruent trials) or a distractor (incongruent trials). Replicating previous findings, we found that eye gaze was biased toward the VWM-matching shape and, importantly, more so when the shape was directly associated with an action plan. Moreover, the ERP results revealed that during the selection task, future action-relevant VWM-matching shapes elicited (1) a stronger Ppc (posterior positivity contralateral), signaling greater attentional saliency; (2) an earlier PD (distractor positivity) component, suggesting faster suppression; (3) a larger inverse (i.e., positive) sustained posterior contralateral negativity in incongruent trials, consistent with stronger suppression of action-associated distractors; and (4) an enhanced response-locked positivity over left motor regions, possibly indicating enhanced inhibition of the response associated with the memorized item during the interim task. Overall, these results suggest that action planning renders objects in VWM more attentionally salient, supporting the notion of selection-for-action in working memory.

Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks.

Despite providing high-performance solutions for computer vision tasks, the deep neural network (DNN) model has been proved to be extremely vulnerable to adversarial attacks. Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. Besides, commonly used adaptive learning and fine-tuning technique is unsuitable for adversarial defense since it is essentially a zero-shot problem when deployed. Thus, to tackle this challenge, we propose an attack-agnostic defense method named Meta Invariance Defense (MID). Specifically, various combinations of adversarial attacks are randomly sampled from a manually constructed Attacker Pool to constitute different defense tasks against unknown attacks, in which a student encoder is supervised by multi-consistency distillation to learn the attack-invariant features via a meta principle. The proposed MID has two merits: 1) Full distillation from pixel-, feature- and prediction-level between benign and adversarial samples facilitates the discovery of attack-invariance. 2) The model simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration. Theoretical and empirical studies on numerous benchmarks such as ImageNet verify the generalizable robustness and superiority of MID under various attacks.

Evoked Component Analysis (ECA): Decomposing the Functional Ultrasound Signal With GLM-Regularization.

Analysis of functional neuroimaging data aims to unveil spatial and temporal patterns of interest. Existing analysis methods fall into two categories: fully data-driven approaches and those reliant on prior information, e.g. the stimulus time course. While using the stimulus signal directly can help identify the activated brain areas, it is known that the relationship between stimuli and the brain's response exhibits nonlinear and time-varying characteristics. As such, relying completely on the stimulus signal to describe the brain's temporal response leads to a restricted interpretation of the brain function. In this paper, we present a new technique called Evoked Component Analysis (ECA), which leverages prior information up to a defined extent. This is achieved by including the general linear model (GLM) design matrix as a regulatory term and estimating the factor matrices in both space and time through an alternating minimization approach. We apply ECA to 2D and swept-3D functional ultrasound (fUS) experiments conducted with mice. When decomposing 2D fUS data, we employ GLM regularization at various intensities to emphasize the role of prior information. Furthermore, we show that incorporating multiple hemodynamic response functions within the design matrix can provide valuable insights into region-specific characteristics of evoked activity. Finally, we use ECA to analyze swept-3D fUS data recorded from five mice engaged in two distinct visual tasks. Swept-3D fUS images the 3D brain sequentially using a moving probe, resulting in different slice acquisition time instants. We show that ECA can estimate factor matrices with a fine resolution at each slice acquisition time instant and yield higher t-statistics compared to GLM and correlation analysis for all subjects.

Deep learning-guided video compression for machine vision tasks

In the video compression industry, video compression tailored to machine vision tasks has recently emerged as a critical area of focus. Given the unique characteristics of machine vision, the current practice of directly employing conventional codecs reveals inefficiency, which requires compressing unnecessary regions. In this paper, we propose a framework that more aptly encodes video regions distinguished by machine vision to enhance coding efficiency. For that, the proposed framework consists of deep learning-based adaptive switch networks that guide the efficient coding tool for video encoding. Through the experiments, it is demonstrated that the proposed framework has superiority over the latest standardization project, video coding for machine benchmark, which achieves a Bjontegaard delta (BD)-rate gain of 5.91% on average and reaches up to a 19.51% BD-rate gain.

Learning changes in educational animation: visuospatial working memory is more predictive than subjective task load

Current theories suggest that visual and spatial processes in working memory are crucial for learning from animation. However, despite over three decades of research on learning from animation, little is known about how visuospatial working memory relates to learning. Instead, animation research often relies on subjective task load to explain and predict learning performance. To better understand how visuospatial working memory and learning from animation are related, a within-subjects study was conducted. Eighty six students learned from two animations of different complexity. The students’ performance on visual learning tasks, visual and spatial working memory capacity, and perceived task load were assessed. Hierarchical regression analyses show that visuospatial working memory capacity is more critical for learning from a complex animation than for learning from a less complex animation. Moreover, visuospatial working memory capacity predicts learning from a complex animation significantly better than subjective task load. The effect size is large. The results provide a coherent picture of the relationships between learning task demands, learners’ visuospatial working memory, perceived task load and learning performance. They not only allow for a more accurate prediction of learning from animation but can also help to tailor the design and use of animations to the learners’ cognitive resources.

3D Object Detection via Residual SqueezeDet

Three-dimensional object detection is a critical task in computer vision with applications in autonomous driving, robotics, and augmented reality. This paper introduces Residual SqueezeDet, a novel network architecture that enhances the performance of 3D object detection on the KITTI dataset. Building upon the efficient SqueezeDet framework, we propose the Residual Fire module, which incorporates skip connections inspired by ResNet architectures into the original Fire module. This innovation improves gradient flow, enhances feature propagation, and allows for more effective training of deeper networks. Our method leverages point cloud and image-based features, employing a Residual SqueezeDet to effectively capture local and global context. Extensive experiments on the KITTI dataset demonstrate that Residual SqueezeDet significantly outperforms the original SqueezeDet, with particularly notable improvements in challenging scenarios. The proposed model maintains computational efficiency while achieving state-of-the-art performance, making it well-suited for real-time applications in autonomous driving. Our work contributes to the field by providing a more accurate and robust solution for 3D object detection, paving the way for improved perception systems in dynamic environments.

Multisensory working memory capture of attention.

During visual search, representations in working memory (WM) can guide the deployment of attention toward memory-matching visual input. Although previous studies have demonstrated that multisensory interactions facilitate WM and visual search, it remains unclear whether multisensory interaction influences attentional capture by WM. To address this issue, the present study adopted a dual-task paradigm, pairing a visual search task with a WM task, in which the memory modality was manipulated to be either visual or audiovisual. The results revealed that memory-driven attentional capture was observed under the visual and the audiovisual condition. Additionally, the capture effects and response time (RT) costs under the audiovisual condition were weaker than those under the visual condition, even on the trials with the earliest RTs. Furthermore, RT benefits under the audiovisual condition were comparable with those under the visual condition. These findings suggest that multisensory interactions can enhance cognitive control, leading to robust strategic effects and improved search performance. In this process, cognitive control tends to suppress the attentional capture by WM-matching distractors rather than enhance the attentional capture by WM-matching targets. The present study offers new insights into the influence of multisensory interactions on attentional capture by WM.

Gastrointestinal image stitching based on improved unsupervised algorithm.

Image stitching is a traditional but challenging computer vision task. The goal is to stitch together multiple images with overlapping areas into a single, natural-looking, high-resolution image without ghosts or seams. This article aims to increase the field of view of gastroenteroscopy and reduce the missed detection rate. To this end, an improved depth framework based on unsupervised panoramic image stitching of the gastrointestinal tract is proposed. In addition, preprocessing for aberration correction of monocular endoscope images is introduced, and a C2f module is added to the image reconstruction network to improve the network's ability to extract features. A comprehensive real image data set, GASE-Dataset, is proposed to establish an evaluation benchmark and training learning framework for unsupervised deep gastrointestinal image splicing. Experimental results show that the MSE, RMSE, PSNR, SSIM and RMSE_SW indicators are improved, while the splicing time remains within an acceptable range. Compared with traditional image stitching methods, the performance of this method is enhanced. In addition, improvements are proposed to address the problems of lack of annotated data, insufficient generalization ability and insufficient comprehensive performance in image stitching schemes based on supervised learning. These improvements provide valuable aids in gastrointestinal examination.

LGIT: local–global interaction transformer for low-light image denoising

Transformer-based methods effectively capture global dependencies in images, demonstrating outstanding performance in multiple visual tasks. However, existing Transformers cannot effectively denoise large noisy images captured under low-light conditions owing to (1) the global self-attention mechanism causing high computational complexity in the spatial dimension owing to a quadratic increase in computation with the number of tokens; (2) the channel-wise self-attention computation unable to optimise the spatial correlations in images. We propose a local–global interaction Transformer (LGIT) that employs an adaptive strategy to select relevant patches for global interaction, achieving low computational complexity in global self-attention computation. A top-N patch cross-attention model (TPCA) is designed based on superpixel segmentation guidance. TPCA selects top-N patches most similar to the target image patch and applies cross attention to aggregate information from them into the target patch, effectively enhancing the utilisation of the image’s nonlocal self-similarity. A mixed-scale dual-gated feedforward network (MDGFF) is introduced for the effective extraction of multiscale local correlations. TPCA and MDGFF were combined to construct a hierarchical encoder-decoder network, LGIT, to compute self-attention within and across patches at different scales. Extensive experiments using real-world image-denoising datasets demonstrated that LGIT outperformed state-of-the-art (SOTA) convolutional neural network (CNN) and Transformer-based methods in qualitative and quantitative results.

SMALE: Hyperspectral Image Classification via Superpixels and Manifold Learning

As an extremely efficient preprocessing tool, superpixels have become more and more popular in various computer vision tasks. Nevertheless, there are still several drawbacks in the application of hyperspectral image (HSl) processing. Firstly, it is difficult to directly apply superpixels because of the high dimension of HSl information. Secondly, existing superpixel algorithms cannot accurately classify the HSl objects due to multi-scale feature categorization. For the processing of high-dimensional problems, we use the principle of PCA to extract three principal components from numerous bands to form three-channel images. In this paper, a novel superpixel algorithm called Seed Extend by Entropy Density (SEED) is proposed to alleviate the seed point redundancy caused by the diversified content of HSl. It also focuses on breaking the dilemma of manually setting the number of superpixels to overcome the difficulty of classification imprecision caused by multi-scale targets. Next, a space–spectrum constraint model, termed Hyperspectral Image Classification via superpixels and manifold learning (SMALE), is designed, which integrates the proposed SEED to generate a dimensionality reduction framework. By making full use of spatial context information in the process of unsupervised dimension reduction, it could effectively improve the performance of HSl classification. Experimental results show that the proposed SEED could effectively promote the classification accuracy of HSI. Meanwhile, the integrated SMALE model outperforms existing algorithms on public datasets in terms of several quantitative metrics.

Three-Dimensional Reconstruction of Indoor Scenes Based on Implicit Neural Representation

Reconstructing 3D indoor scenes from 2D images has always been an important task in computer vision and graphics applications. For indoor scenes, traditional 3D reconstruction methods have problems such as missing surface details, poor reconstruction of large plane textures and uneven illumination areas, and many wrongly reconstructed floating debris noises in the reconstructed models. This paper proposes a 3D reconstruction method for indoor scenes that combines neural radiation field (NeRFs) and signed distance function (SDF) implicit expressions. The volume density of the NeRF is used to provide geometric information for the SDF field, and the learning of geometric shapes and surfaces is strengthened by adding an adaptive normal prior optimization learning process. It not only preserves the high-quality geometric information of the NeRF, but also uses the SDF to generate an explicit mesh with a smooth surface, significantly improving the reconstruction quality of large plane textures and uneven illumination areas in indoor scenes. At the same time, a new regularization term is designed to constrain the weight distribution, making it an ideal unimodal compact distribution, thereby alleviating the problem of uneven density distribution and achieving the effect of floating debris removal in the final model. Experiments show that the 3D reconstruction effect of this paper on ScanNet, Hypersim, and Replica datasets outperforms the state-of-the-art methods.

Tuning Vision-Language Models with Multiple Prototypes Clustering.

Benefiting from advances in large-scale pre-training, foundation models, have demonstrated remarkable capability in the fields of natural language processing, computer vision, among others. However, to achieve expert-level performance in specific applications, such models often need to be fine-tuned with domain-specific knowledge. In this paper, we focus on enabling vision-language models to unleash more potential for visual understanding tasks under few-shot tuning. Specifically, we propose a novel adapter, dubbed as lusterAdapter, which is based on trainable multiple prototypes clustering algorithm, for tuning the CLIP model. It can not only alleviate the concern of catastrophic forgetting of foundation models by introducing anchors to inherit common knowledge, but also improve the utilization efficiency of few annotated samples via bringing in clustering and domain priors, thereby improving the performance of few-shot tuning. We have conducted extensive experiments on 11 common classification benchmarks. The results show our method significantly surpasses the original CLIP and achieves state-of-the-art (SOTA) performance under all benchmarks and settings. For example, under the 16-shot setting, our method exhibits a remarkable improvement over the original CLIP by 19.6%, and also surpasses TIP-Adapter and GraphAdapter by 2.7% and 2.2%, respectively, in terms of average accuracy across the 11 benchmarks. Code is available at https://github.com/uyzhang/Cluster-Adapter.

Fast Comparative Analysis of Merge Trees Using Locality Sensitive Hashing.

Scalar field comparison is a fundamental task in scientific visualization. In topological data analysis, we compare topological descriptors of scalar fields-such as persistence diagrams and merge trees-because they provide succinct and robust abstract representations. Several similarity measures for topological descriptors seem to be both asymptotically and practically efficient with polynomial time algorithms, but they do not scale well when handling large-scale, time-varying scientific data and ensembles. In this paper, we propose a new framework to facilitate the comparative analysis of merge trees, inspired by tools from locality sensitive hashing (LSH). LSH hashes similar objects into the same hash buckets with high probability. We propose two new similarity measures for merge trees that can be computed via LSH, using new extensions to Recursive MinHash and subpath signature, respectively. Our similarity measures are extremely efficient to compute and closely resemble the results of existing measures such as merge tree edit distance or geometric interleaving distance. Our experiments demonstrate the utility of our LSH framework in applications such as shape matching, clustering, key event detection, and ensemble summarization.

Enhancing visual reinforcement learning with State–Action Representation

Despite the remarkable progress made in visual reinforcement learning (RL) in recent years, sample inefficiency remains a major challenge. Many existing approaches attempt to address this by extracting better representations from raw images using techniques like data augmentation or introducing some auxiliary tasks. However, these methods overlook the environmental dynamic information embedded in the collected transitions, which can be crucial for efficient control. In this paper, we present STAR: State-Action Representation Learning, a simple yet effective approach for visual continuous control. STAR learns a joint state–action representation by modeling the dynamics of the environment in the latent space. By incorporating the learned joint state–action representation into the critic, STAR enhances the value estimation with latent dynamics information. We theoretically show that the value function can still converge to the optima when involving additional representation inputs. On various challenging visual continuous control tasks from DeepMind Control Suite, STAR achieves significant improvements in sample efficiency compared to strong baseline algorithms.

Improving Memory through Better Sleep in Community-dwelling Older Adults: A Tai Chi Intervention Study.

The relationship between sleep and memory has been well documented. However, it remains unclear whether a mind-body exercise, i.e., Tai Chi exercise, can improve memory performance in older adults by improving their subjective and objective sleep. A randomized controlled trial was conducted with participants (M = 67.36, 56-79 years) randomly assigned to Tai Chi and control groups. The primary outcomes were sleep, both subjectively reported and objectively assessed by actigraphy, and memory performance, as well as the mediating role of sleep in memory improvement with Tai Chi practice. Tai Chi exercise led to improvements in subjective sleep, as indicated by ISI (p < 0.001, Cohen's d = 0.62) and daytime dysfunction of the PSQI (p = 0.02, Cohen's d = 0.80), and in actigraphy-assessed sleep onset latency (p < 0.01, Cohen's d = 0.61), as well as improved memory performance on digit span forward (p < 0.001, Cohen's d = 1.20) and visual spatial memory tasks (p < 0.01, Cohen's d = 0.83) compared to the control group. Importantly, Tai Chi practice improved digit span forward memory performance through parallel mediation of both subjective sleep (i.e., daytime dysfunction of the PSQI) and objective sleep (i.e., sleep onset latency; b = 0.29, p < 0.01). Our findings uncovered the potential benefits of Tai Chi exercise in relation to both subjective and objective sleep in older adults, in turn, how sleep changes played a role in the link between Tai Chi exercise and memory changes in older adults.

Searching Across Realities: Investigating ERPs and Eye-Tracking Correlates of Visual Search in Mixed Reality.

Mixed Reality allows us to integrate virtual and physical content into users' environments seamlessly. Yet, how this fusion affects perceptual and cognitive resources and our ability to find virtual or physical objects remains uncertain. Displaying virtual and physical information simultaneously might lead to divided attention and increased visual complexity, impacting users' visual processing, performance, and workload. In a visual search task, we asked participants to locate virtual and physical objects in Augmented Reality and Augmented Virtuality to understand the effects on performance. We evaluated search efficiency and attention allocation for virtual and physical objects using event-related potentials, fixation and saccade metrics, and behavioral measures. We found that users were more efficient in identifying objects in Augmented Virtuality, while virtual objects gained saliency in Augmented Virtuality. This suggests that visual fidelity might increase the perceptual load of the scene. Reduced amplitude in distractor positivity ERP, and fixation patterns supported improved distractor suppression and search efficiency in Augmented Virtuality. We discuss design implications for mixed reality adaptive systems based on physiological inputs for interaction.

A lightweight convolutional neural network-based feature extractor for visible images

Feature extraction networks (FENs), as the first stage in many computer vision tasks, play critical roles. Previous studies regarding FENs employed deeper and wider networks to attain higher accuracy, but their approaches were memory-inefficient and computationally intensive. Here, we present an accurate and lightweight feature extractor (RoShuNet) for visible images based on ShuffleNetV2. The provided improvements are threefold. To make ShuffleNetV2 compact without degrading its feature extraction ability, we propose an aggregated dual group convolutional module; to better aid the channel interflow process, we propose a γ-weighted shuffling module; to further reduce the complexity and size of the model, we introduce slimming strategies. Classification experiments demonstrate the state-of-the-art (SOTA) performance of RoShuNet, which yields an increase in accuracy and reduces the complexity and size of the model compared to those of ShuffleNetV2. Generalization experiments verify that the proposed method is also applicable to feature extraction tasks in semantic segmentation and multiple-object tracking scenarios, achieving comparable accuracy to that of other approaches with more memory and greater computational efficiency. Our method provides a novel perspective for designing lightweight models.