Visual Task Research Articles

A learnable motion preserving pooling for action recognition

Using deep neural networks (DNN) for video understanding tasks is expensive in terms of computation cost. Pooling layers in DNN which are widely used in most vision tasks to resize the spatial dimensions play crucial roles in reducing the computation and memory cost. In video-related tasks, pooling layers are also applied, mostly in the spatial dimension only as the standard average pooling in the temporal domain can significantly reduce its performance. This is because conventional temporal pooling degrades the underlying important motion features in consecutive frames. Such a phenomenon is rarely investigated and most state-of-art methods simply do not adopt temporal pooling, leading to enormous computation costs. In this work, we propose a learnable motion-preserving pooling (MPPool) layer that is able to preserve the general motion progression after the pooling. This pooling layer first locates the frames with the strongest motion features and then keeps these crucial features during pooling. Our experiments demonstrate that MPPool not only reduces the computation cost for video data modeling, but also increases the final prediction accuracy on various motion-centric and appearance-centric datasets.

Unsupervised Illumination Adaptation for Low-Light Vision.

Insufficient lighting poses challenges to both human and machine visual analytics. While existing low-light enhancement methods prioritize human visual perception, they often neglect machine vision and high-level semantics. In this paper, we make pioneering efforts to build an illumination enhancement model for high-level vision. Drawing inspiration from camera response functions, our model could enhance images from the machine vision perspective despite being lightweight in architecture and simple in formulation. We also introduce two approaches that leverage knowledge from base enhancement curves and self-supervised pretext tasks to train for different downstream normal-to-low-light adaptation scenarios. Our proposed framework overcomes the limitations of existing algorithms without requiring access to labeled data in low-light conditions. It facilitates more effective illumination restoration and feature alignment, significantly improving the performance of downstream tasks in a plug-and-play manner. This research advances the field of low-light machine analytics and broadly applies to various high-level vision tasks, including classification, face detection, optical flow estimation, and video action recognition.

A novel theoretical analysis on optimal pipeline of multi-frame image super-resolution using sparse coding

Super-resolution is the process of obtaining a high-resolution (HR) image from one or more low-resolution (LR) images. Single image super-resolution (SISR) deals with one LR image while multi-frame super-resolution (MFSR) employs several LR ones to reach the HR output. MFSR pipeline consists of alignment, fusion, and reconstruction. We conduct a theoretical analysis using sparse coding (SC) and iterative shrinkage-thresholding algorithm to fill the gap of mathematical justification in the execution order of the optimal MFSR pipeline. Our analysis recommends executing alignment and fusion before the reconstruction stage (whether through deconvolution or SISR techniques). The suggested order ensures enhanced performance in terms of peak signal-to-noise ratio and structural similarity index. The optimal pipeline also reduces computational complexity compared to intuitive approaches that apply SISR to each input LR image. Also, we demonstrate the usefulness of SC in analysis of computer vision tasks such as MFSR, leveraging the sparsity assumption in natural images. Simulation results support the findings of our theoretical analysis, both quantitatively and qualitatively.

Quaternion Nuclear Norm Minus Frobenius Norm Minimization for color image reconstruction

Color image restoration methods typically represent images as vectors in Euclidean space or combinations of three monochrome channels. However, they often overlook the correlation between these channels, leading to color distortion and artifacts in the reconstructed image. To address this, we present Quaternion Nuclear Norm Minus Frobenius Norm Minimization (QNMF), a novel approach for color image reconstruction. QNMF utilizes quaternion algebra to capture the relationships among RGB channels comprehensively. By employing a regularization technique that involves nuclear norm minus Frobenius norm, QNMF approximates the underlying low-rank structure of quaternion-encoded color images. Theoretical proofs are provided to ensure the method’s mathematical integrity. Demonstrating versatility and efficacy, the QNMF regularizer excels in various color low-level vision tasks, including denoising, deblurring, inpainting, and random impulse noise removal, achieving state-of-the-art results.

Scalable video transformer for full-frame video prediction

Vision Transformers (ViTs) have shown success in many low-level computer vision tasks. However, existing ViT models are limited by their high computation and memory cost when generating high-resolution videos for tasks like video prediction. This paper presents a scalable video transformer for full-frame video predication. Specifically, we design a backbone transformer block for our video transformer. This transformer block decouples the temporal and channel features to reduce the computation cost when processing large-scale spatial–temporal video features. We use transposed attention to focus on the channel dimension instead of the spatial window to further reduce the computation cost. We also design a Global Shifted Multi-Dconv Head Transposed Attention module (GSMDTA) for our transformer block. This module is built upon two key ideas. First, we design a depth shift module to better incorporate the cross-channel or temporal information from video features. Second, we introduce a global query mechanism to capture global information to handle large motion for video prediction. This new transformer block enables our video transformer to predict a full frame from multiple past frames at the resolution of 1024 × 512 with 12 GB VRAM. Experiments on various video prediction benchmarks demonstrate that our method with only RGB input outperforms state-of-the-art methods that require additional data, like segmentation maps and optical flows. Our method exceeds the state-of-the-art RGB-only methods by a large margin (1.2 dB) in PSNR. Our method is also faster than state-of-the-art video prediction transformers.

Vision transformers inference acceleration based on adaptive layer normalization

Over the past decade, deep neural networks (DNNs) have been widely used due to their remarkable accuracy in real-world applications. However, this increase in accuracy often results in computationally expensive models and high memory usage, leading to longer prediction latencies and exorbitant release costs. In this paper, we propose a new adaptive layer normalization (ALN) algorithm for transformer models, which tackles the computational, and memory problems encountered by traditional layer normalization (LN) techniques. The proposed method computes and stores statistical moments during the training and uses them directly during the inference phase, allowing the normalization layer to be merged with the nearest linear layer. The result is a significant acceleration in inference time, by up to 29%. In classification tasks, our evaluations on the ImageNet dataset show an improvement in accuracy of 0.1%, while maintaining comparable accuracy in object detection tasks on the COCO reference dataset. The proposed ALN algorithm is a simple and effective solution for improving the inference time of pre-trained transformer models, making it a valuable tool for natural language processing and computer vision tasks.

Application of transformers in stomatological imaging: A review

Stomatology extensively utilizes detailed imaging to assist clinicians, traditionally requiring labor-intensive manual analysis, which significantly adds to their workload. Transformers are revolutionary neural network in deep learning, gaining substantial momentum in computer vision tasks. Their introduction into medical imaging, particularly in processing large image datasets, has been marked by remarkable precision and efficiency, establishing them as a pivotal tool in emerging research. However, the application of Transformers in stomatological imaging is still in its infancy. Current studies primarily focus on segmenting specific anatomical features such as teeth and jawbones, with some clinical implementations. Yet, the comprehensive analytical potential of Transformers in this field remains largely untapped. This paper presents an introductory examination of Transformers, coupled with an initial synthesis and assessment of its dental applications across various areas. It will highlight the observed advantages and limitations in dental contexts and conclude with a discussion on future research directions. This serves as a foundational guide for in-depth future investigations in this area.

HIRI-ViT: Scaling Vision Transformer With High Resolution Inputs.

The hybrid deep models of Vision Transformer (ViT) and Convolution Neural Network (CNN) have emerged as a powerful class of backbones for vision tasks. Scaling up the input resolution of such hybrid backbones naturally strengthes model capacity, but inevitably suffers from heavy computational cost that scales quadratically. Instead, we present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs. HIRI-ViT is built upon the seminal idea of decomposing the typical CNN operations into two parallel CNN branches in a cost-efficient manner. One high-resolution branch directly takes primary high-resolution features as inputs, but uses less convolution operations. The other low-resolution branch first performs down-sampling and then utilizes more convolution operations over such low-resolution features. Experiments on both recognition task (ImageNet-1K dataset) and dense prediction tasks (COCO and ADE20 K datasets) demonstrate the superiority of HIRI-ViT. More remarkably, under comparable computational cost ( ∼5.0 GFLOPs), HIRI-ViT achieves to-date the best published Top-1 accuracy of 84.3% on ImageNet with 448×448 inputs, which absolutely improves 83.4% of iFormer-S by 0.9% with 224×224 inputs.

Lightweight multi-scale feature dense cascade neural network for scene understanding of intelligent autonomous platform

Understanding the surrounding environment is an essential part of intelligent autonomous platform to complete autonomous driving or other autonomous tasks. Thereinto, scene understanding is a key technology to understand environmental information. For intelligent autonomous platform, it not only requires high accuracy, but also has a strict requirement for the inference speed. Moreover, it should be able to perform normally in complex environments such as target occlusion, the influence of illumination, and objects at different sizes. For tackling these problems, in this study, a lightweight multi-scale feature dense cascade neural network (LMFDCNet) is proposed in real-time scene understanding in complex scenes. Furthermore, trilateral fusion module is adopted to enhance low-level detailed features or high-level semantic features of each branch to describe the object better in different environments. A multi-scale upsampling module that fuses contextual information at different scales allows objects to be represented at different size. The results indicate that LMFDCNet produces 99.1 % classification accuracy on the tobacco dataset. On the Cityscapes dataset, we achieve 74.2 % Mean Intersection over Union (MIoU) with 13.6 M parameters at the speed of 38.2 FPS on a single 1070Ti card. To test the performance on the real-time application in the real world, LMFDCNet is deployed on the autonomous platform for real-time vision task such as classification of tobacco leaf state in curing process and semantic segmentation in the driving process of autonomous platform. The results reveal that LMFDCNet perform well on the AI embedded device and could be applied into intelligent autonomous platform.

Object and spatial discrimination makes weakly supervised local feature better

Local feature extraction plays a crucial role in numerous critical visual tasks. However, there remains room for improvement in both descriptors and keypoints, particularly regarding the discriminative power of descriptors and the localization precision of keypoints. To address these challenges, this study introduces a novel local feature extraction pipeline named OSDFeat (Object and Spatial Discrimination Feature). OSDFeat employs a decoupling strategy, training descriptor and detection networks independently. Inspired by semantic correspondence, we propose an Object and Spatial Discrimination ResUNet (OSD-ResUNet).OSD-ResUNet captures features from the feature map that differentiate object appearance and spatial context, thus enhancing descriptor performance. To further improve the discriminative capability of descriptors, we propose a Discrimination Information Retained Normalization module (DIRN). DIRN complementarily integrates spatial-wise normalization and channel-wise normalization, yielding descriptors that are more distinguishable and informative. In the detection network, we propose a Cross Saliency Pooling module (CSP). CSP employs a cross-shaped kernel to aggregate long-range context in both vertical and horizontal dimensions. By enhancing the saliency of keypoints, CSP enables the detection network to effectively utilize descriptor information and achieve more precise localization of keypoints. Compared to the previous best local feature extraction methods, OSDFeat achieves Mean Matching Accuracy of 79.4% in local feature matching task, improving by 1.9% and achieving state-of-the-art results.Additionally, OSDFeat achieves competitive results in Visual Localization and 3D Reconstruction. The results of this study indicate that object and spatial discrimination can improve the accuracy and robustness of local feature, even in challenging environments. The code is available at https://github.com/pandaandyy/OSDFeat.

A method for rapid assessment of visual ergonomics and lighting conditions (RAVEL): An in-depth development and psychometrics study

BACKGROUND: In workplaces heavily reliant on visual tasks, various factors can significantly influence an individual’s performance, necessitating the use of reliable tools to identify and mitigate these factors. OBJECTIVE: This study aimed to develop a swift assessment method for visual ergonomics and lighting conditions, evaluating its validity in real-world scenarios. METHODS: The questionnaire’s content validity was determined by a panel of experts using the content validity ratio (CVR) and content validity index (CVI). Construct validity was assessed through exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and latent class analysis (LCA). Internal consistency was measured using Cronbach’s alpha coefficient. The RAVEL index, derived from the calculated effect coefficients of items, classified total scores through receiver operator curves (ROCs). RESULTS: The rapid assessment method, comprising two parts with 30 items, demonstrated acceptable reliability with CVR, CVI, and Cronbach’s alpha coefficient (α) at 0.75, 0.87, and 0.896, respectively. The EFA on the first part’s 22 items identified three factors, confirmed by CFA. The LCA on the second part’s eight items revealed that a two-class model best fit the data, with Bayesian information criterion (BIC) = 24249, 17, Akaik information criterion (AIC) = 2179.89, and an entropy R-squared of 0.83, indicating appropriate subject classification based on the model. The RAVEL score was categorized into three levels, with optimal cut points of 55 and 63. CONCLUSIONS: In conclusion, the study demonstrated that this method based on visual ergonomics serves as a rapid and reliable tool for assessing visual ergonomic risks of display users in the workplace.

A maritime image dehazing algorithm based on improved VQGAN

ABSTRACT Maritime image dehazing faces challenges as existing methods struggle to handle different concentrations of sea fog, and the treatment of details in dehazed results is often suboptimal. This paper proposes a maritime image dehazing algorithm based on the VQGAN mode. Specifically, since VQGAN is capable of generating high-quality discrete codebooks, we input the haze-free maritime dataset into VQGAN for pre-training to obtain a high-quality codebook prior. During the vector quantisation stage,high-quality vectors from the discrete codebook undergo distance calculations with the upsampled vectors containing haze.This process replaces the vectors containing haze with the high-quality vectors. To address the issue of residual fog, the authors introducing the FAM-UNet encoder.To tackle distortion issues, the authors introduced the Pyramid, Cascading, and Deformable Convolution Feature Alignment Module (PCD) into the original model. The experimental results indicate that our model can effectively restore clear images, providing assistance for subsequent advanced computer vision tasks..

Discrepancy-based diffusion models for lesion detection in brain MRI

Diffusion probabilistic models (DPMs) have exhibited significant effectiveness in computer vision tasks, particularly in image generation. However, their notable performance heavily relies on labelled datasets, which limits their application in medical images due to the associated high-cost annotations. Current DPM-related methods for lesion detection in medical imaging, which can be categorized into two distinct approaches, primarily rely on image-level annotations. The first approach, based on anomaly detection, involves learning reference healthy brain representations and identifying anomalies based on the difference in inference results. In contrast, the second approach, resembling a segmentation task, employs only the original brain multi-modalities as prior information for generating pixel-level annotations. In this paper, our proposed model – discrepancy distribution medical diffusion (DDMD) – for lesion detection in brain MRI introduces a novel framework by incorporating distinctive discrepancy features, deviating from the conventional direct reliance on image-level annotations or the original brain modalities. In our method, the inconsistency in image-level annotations is translated into distribution discrepancies among heterogeneous samples while preserving information within homogeneous samples. This property retains pixel-wise uncertainty and facilitates an implicit ensemble of segmentation, ultimately enhancing the overall detection performance. Thorough experiments conducted on the BRATS2020 benchmark dataset containing multimodal MRI scans for brain tumour detection demonstrate the great performance of our approach in comparison to state-of-the-art methods.

Alpha transcranial alternating current stimulation as add-on to neglect training: a randomized trial

Abstract Visuospatial neglect is a common and debilitating condition following unilateral stroke, significantly impacting cognitive functioning and daily life. There is an urgent need for effective treatments that can provide clinically relevant and sustained benefits. In addition to traditional stroke treatment, non-invasive brain stimulation, such as transcranial alternating current stimulation, shows promise as a complementary approach to enhance stroke recovery. In the current study, we aimed to evaluate the additive effects of multi-session transcranial alternating current stimulation at alpha frequency when combined with visual scanning training in chronic stroke patients with visuospatial neglect. In this double-blind randomized controlled trial, we compared the effects of active transcranial alternating current stimulation at alpha frequency to sham (placebo) transcranial alternating current stimulation, both combined with visual scanning training. Both groups received eighteen 40-minute training sessions over a 6-week period. A total of 22 chronic visuospatial neglect patients participated in the study (active group n = 12, sham group n = 10). The median age was 61.0 years, with a median time since stroke of 36.1 months. We assessed the patients at six time-points: at baseline, after the first, ninth and eighteenth training sessions, as well as 1 week and 3 months following the completion of the combined neuromodulation intervention. The primary outcome measure was the change in performance on a visual search task, specifically the star cancellation task. Secondary outcomes included performance on a visual detection task, two line bisection tasks and three tasks evaluating visuospatial neglect in daily living. We found significantly improved visual search (primary outcome) and visual detection performance in the neglected side in the active transcranial alternating current stimulation group, compared to the sham transcranial alternating current stimulation group. We did not observe stimulation effects on line bisection performance nor in daily living. Time effects were observed on all but one outcome measures. Multi-session transcranial alternating current stimulation combined with visual scanning training may be a more effective treatment for chronic visuospatial neglect than visual scanning training alone. These findings provide valuable insights into novel strategies for stroke recovery, even long after the injury, with the aim of enhancing cognitive rehabilitation outcomes and improving the overall quality of life for individuals affected by this condition. Trial registration: ClinicalTrials.gov; registration number: NCT05466487; https://clinicaltrials.gov/ct2/show/NCT05466487

SeACPFusion: An Adaptive Fusion Network for infrared and visible images based on brightness perception

Generating a single fused image that highlights important targets and preserves textural details is the aim of fusing visible and infrared images. The majority of deep learning-based fusion algorithms now in use can produce decent fusion outcomes; however, the modeling process still lacks consideration of the different amounts of information in different scenes or regions. Thus, we propose in this research SeACPFusion, a luminance-aware adaptive fusion network for infrared and visible images, which adaptively preserves the intensity information of the noticeable targets of the source images with the texture information of the background in an optimal ratio. Specifically, we design pixel-level luminance loss (PBL) to direct the fusion model’s training in real-time, and PBL retains the optimal intensity information according to the pixel luminance ratio of different source images. In addition, we designed the Channel Transformer (CTF) to consider the relationship between different attributes from the point of view of the feature channel and to focus on the key information by using the self-focusing mechanism to achieve the goal of adaptive fusion. Our extensive tests on the MSRS, RoadScene, and TNO datasets demonstrate that SeACPFusion surpasses nine representative deep learning methods on six objective metrics and achieves the best visual results in scenes such as overexposure or underexposure. In addition, the relatively efficient operation and fewer model parameters make our algorithm promising as a preprocessing module for downstream complicated vision tasks.

Pre-stimulus EEG phase coherence predicts visual target detection failures in schizophrenia: A pilot study

Impaired visual target detection is a common finding in schizophrenia that is linked to poor functional outcomes. However, the neural mechanisms that contribute to this deficit remain unclear. Recent research in healthy samples has identified relationships between the phase of pre-stimulus electroencephalographic (EEG) activity in the alpha band (8-12 Hz) or theta band (4-7 Hz) and the likelihood of visual target detection with and without attentional cueing, but these effects have not yet been explored in schizophrenia. We performed a study to investigate such effects in schizophrenia (n = 19) and healthy participants (n = 14), using a visual target detection task with attentional cues. We found significant relationships between pre-stimulus EEG phase properties and visual target detection in both groups, but also clear differences in the effects as a function of frequency, group, and attentional cueing. Alpha-band phase effects were relatively uniform across groups and conditions. By contrast, theta-band phase effects showed differences by group and attentional condition which could be consistent with attentional hyperfocusing in the schizophrenia group. Thus, our results elucidate a novel neural mechanism that may help to explain known impairments affecting both visual target detection and attention in schizophrenia.

Geospatial foundation models for image analysis: evaluating and enhancing NASA-IBM Prithvi’s domain adaptability

Research on geospatial foundation models (GFMs) has become a trending topic in geospatial artificial intelligence (AI) research due to their potential for achieving high generalizability and domain adaptability, reducing model training costs for individual researchers. Unlike large language models, such as ChatGPT, constructing visual foundation models for image analysis, particularly in remote sensing, encountered significant challenges such as formulating diverse vision tasks into a general problem framework. This paper evaluates the recently released NASA-IBM GFM Prithvi for its predictive performance on high-level image analysis tasks across multiple benchmark datasets. Prithvi was selected because it is one of the first open-source GFMs trained on time-series of high-resolution remote sensing imagery. A series of experiments were designed to assess Prithvi’s performance as compared to other pre-trained task-specific AI models in geospatial image analysis. New strategies, including band adaptation, multi-scale feature generation, and fine-tuning techniques, are introduced and integrated into an image analysis pipeline to enhance Prithvi’s domain adaptation capability and improve model performance. In-depth analyses reveal Prithvi’s strengths and weaknesses, offering insights for both improving Prithvi and developing future visual foundation models for geospatial tasks.

Unsupervised low-light image enhancement by data augmentation and contrastive learning

ABSTRACT Today, with the increasing demand for visual perception and high-level computational vision tasks, the field of low-light enhancement is rapidly developing. However, models trained on existing datasets often fail or suffer significant performance degradation in real-world low-light scenarios. This performance degradation is frequently due to the limitations of current databases, which typically contain small quantities of paired images of a single type. This article proposes an unsupervised model with a unique data augmentation technique that transforms a regular image database into a paired image database. By adjusting image parameters during training to change exposure, a regular image database can be converted into a paired one. The model restores low-exposure images by extracting lighting features through comparative learning. Evaluations of the LOL and DIV2K datasets demonstrate the proposed model's effectiveness, achieving notable results in low-light image enhancement. This method removes dataset restrictions, broadening the model's range of applications.

ConIQA: A deep learning method for perceptual image quality assessment with limited data

Effectively assessing the realism and naturalness of images in virtual (VR) and augmented (AR) reality applications requires Full Reference Image Quality Assessment (FR-IQA) metrics that closely align with human perception. Deep learning-based IQAs that are trained on human-labeled data have recently shown promise in generic computer vision tasks. However, their performance decreases in applications where perfect matches between the reference and the distorted images should not be expected, or whenever distortion patterns are restricted to specific domains. Tackling this issue necessitates training a task-specific neural network, yet generating human-labeled FR-IQAs is costly, and deep learning typically demands substantial labeled data. To address these challenges, we developed ConIQA, a deep learning-based IQA that leverages consistency training and a novel data augmentation method to learn from both labeled and unlabeled data. This makes ConIQA well-suited for contexts with scarce labeled data. To validate ConIQA, we considered the example application of Computer-Generated Holography (CGH) where specific artifacts such as ringing, speckle, and quantization errors routinely occur, yet are not explicitly accounted for by existing IQAs. We developed a new dataset, HQA1k, that comprises 1000 natural images each paired with an image rendered using various popular CGH algorithms, and quality-rated by thirteen human participants. Our results show that ConIQA achieves superior Pearson (0.98), Spearman (0.965), and Kendall’s tau (0.86) correlations over fifteen FR-IQA metrics by up to 5%, showcasing significant improvements in aligning with human perception on the HQA1k dataset.

Exposure to gestational diabetes mellitus in utero impacts hippocampal functional connectivity in response to food cues in children.

Intrauterine exposure to gestational diabetes mellitus (GDM) increases the risk of obesity in the offspring, but little is known about the underlying neural mechanisms. The hippocampus is crucial for food intake regulation and is vulnerable to the effects of obesity. The purpose of the study was to investigate whether GDM exposure affects hippocampal functional connectivity during exposure to food cues using functional magnetic resonance imaging (fMRI). Participants were 90 children age 7-11 years (53 females) who underwent an fMRI-based visual food cue task in the fasted state. Hippocampal functional connectivity (FC) was examined using generalized psychophysiological interaction in response to food versus non-food cues. Hippocampal FC was compared between children with and without GDM exposure, while controlling for possible confounding effects of age, sex and waist-to-hip ratio. In addition, the influence of childhood and maternal obesity were investigated using multiple regression models. While viewing high caloric food cues compared to non-food cure, children with GDM exposure exhibited higher hippocampal FC to the insula and striatum (i.e., putamen, pallidum and nucleus accumbens) compared to unexposed children. With increasing BMI, children with GDM exposure had lower hippocampal FC to the somatosensory cortex (i.e., postcentral gyrus). Intrauterine exposure to GDM was associated with higher food-cue induced hippocampal FC especially to reward processing regions. Future studies with longitudinal measurements are needed to clarify whether altered hippocampal FC may raise the risk of the development of metabolic diseases later in life.