High-resolution Input Research Articles

ACT-FRCNN: Progress Towards Transformer-Based Object Detection

Maintaining a high input resolution is crucial for more complex tasks like detection or segmentation to ensure that models can adequately identify and reflect fine details in the output. This study aims to reduce the computation costs associated with high-resolution input by using a variant of transformer, known as the Adaptive Clustering Transformer (ACT). The proposed model is named ACT-FRCNN. Which integrates ACT with a Faster Region-Based Convolution Neural Network (FRCNN) for a detection task head. In this paper, we proposed a method to improve the detection framework, resulting in better performance for out-of-domain images, improved object identification, and reduced dependence on non-maximum suppression. The ACT-FRCNN represents a significant step in the application of transformer models to challenging visual tasks like object detection, laying the foundation for future work using transformer models. The performance of ACT-FRCNN was evaluated on a variety of well-known datasets including BSDS500, NYUDv2, and COCO. The results indicate that ACT-FRCNN reduces over-detection errors and improves the detection of large objects. The findings from this research have practical implications for object detection and other computer vision tasks.

EHNet: Efficient Hybrid Network with Dual Attention for Image Deblurring

The motion of an object or camera platform makes the acquired image blurred. This degradation is a major reason to obtain a poor-quality image from an imaging sensor. Therefore, developing an efficient deep-learning-based image processing method to remove the blur artifact is desirable. Deep learning has recently demonstrated significant efficacy in image deblurring, primarily through convolutional neural networks (CNNs) and Transformers. However, the limited receptive fields of CNNs restrict their ability to capture long-range structural dependencies. In contrast, Transformers excel at modeling these dependencies, but they are computationally expensive for high-resolution inputs and lack the appropriate inductive bias. To overcome these challenges, we propose an Efficient Hybrid Network (EHNet) that employs CNN encoders for local feature extraction and Transformer decoders with a dual-attention module to capture spatial and channel-wise dependencies. This synergy facilitates the acquisition of rich contextual information for high-quality image deblurring. Additionally, we introduce the Simple Feature-Embedding Module (SFEM) to replace the pointwise and depthwise convolutions to generate simplified embedding features in the self-attention mechanism. This innovation substantially reduces computational complexity and memory usage while maintaining overall performance. Finally, through comprehensive experiments, our compact model yields promising quantitative and qualitative results for image deblurring on various benchmark datasets.

HIRI-ViT: Scaling Vision Transformer With High Resolution Inputs.

The hybrid deep models of Vision Transformer (ViT) and Convolution Neural Network (CNN) have emerged as a powerful class of backbones for vision tasks. Scaling up the input resolution of such hybrid backbones naturally strengthes model capacity, but inevitably suffers from heavy computational cost that scales quadratically. Instead, we present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs. HIRI-ViT is built upon the seminal idea of decomposing the typical CNN operations into two parallel CNN branches in a cost-efficient manner. One high-resolution branch directly takes primary high-resolution features as inputs, but uses less convolution operations. The other low-resolution branch first performs down-sampling and then utilizes more convolution operations over such low-resolution features. Experiments on both recognition task (ImageNet-1K dataset) and dense prediction tasks (COCO and ADE20 K datasets) demonstrate the superiority of HIRI-ViT. More remarkably, under comparable computational cost ( ∼5.0 GFLOPs), HIRI-ViT achieves to-date the best published Top-1 accuracy of 84.3% on ImageNet with 448×448 inputs, which absolutely improves 83.4% of iFormer-S by 0.9% with 224×224 inputs.

Assessing machine learning models to generate permafrost distribution map in Solukhumbu, Nepal

Permafrost is one of the key components of the cryosphere. Previous studies show that the extent of permafrost has shifted to higher elevations in Nepal. These researches, however, has been hampered by inconsistency in their study period. Proxies like rock glaciers and climatic variables, such as multi-decadal annual air temperature, are used to link towards the likely occurrence of permafrost. Here, the rock glacier inventory of Solukhumbu was prepared, and classified based on their activity (Intact/Relict) from Google Earth. Talus-based rock glaciers were observed more than glacier-derived ones. These rock glaciers were highly correlated with Mean Annual Air Temperature, followed by potential incoming solar radiation and slope. Three machine learning models (Logistic Regression, Random Forest and Support Vector Machines) were trained to generate permafrost probability distribution maps based on their prediction of the probability of rock glaciers being intact as opposed to relict. Logistic Regression and Support Vector Machines were able to produce a similar spatial distribution of permafrost. However, the Random Forest has low precision of spatial variation. The permafrost distribution map suggests the likely occurrence of permafrost to be above 5000 m, indicating a potential for rock and landslides should it thaw in the future. While higher-resolution input data can improve the results, this approach remains promising for application in permafrost regions where information about the ice content of rock glaciers is very limited.

Regionally optimized high-resolution input datasets enhance the representation of snow cover in CLM5

Regionally optimized high-resolution input datasets enhance the representation of snow cover in CLM5

SRGAN-enhanced unsafe operation detection and classification of heavy construction machinery using cascade learning

In the inherently hazardous construction industry, where injuries are frequent, the unsafe operation of heavy construction machinery significantly contributes to the injury and accident rates. To reduce these risks, this study introduces a novel framework for detecting and classifying these unsafe operations for five types of construction machinery. Utilizing a cascade learning architecture, the approach employs a Super-Resolution Generative Adversarial Network (SRGAN), Real-Time Detection Transformers (RT-DETR), self-DIstillation with NO labels (DINOv2), and Dilated Neighborhood Attention Transformer (DiNAT) models. The study focuses on enhancing the detection and classification of unsafe operations in construction machinery through upscaling low-resolution surveillance footage and creating detailed high-resolution inputs for the RT-DETR model. This enhancement, by leveraging temporal information, significantly improves object detection and classification accuracy. The performance of the cascaded pipeline yielded an average detection and first-level classification precision of 96%, a second-level classification accuracy of 98.83%, and a third-level classification accuracy of 98.25%, among other metrics. The cascaded integration of these models presents a well-rounded solution for near-real-time surveillance in dynamic construction environments, advancing surveillance technologies and significantly contributing to safety management within the industry.

ICDW-YOLO: An Efficient Timber Construction Crack Detection Algorithm.

A robust wood material crack detection algorithm, sensitive to small targets, is indispensable for production and building protection. However, the precise identification and localization of cracks in wooden materials present challenges owing to significant scale variations among cracks and the irregular quality of existing data. In response, we propose a crack detection algorithm tailored to wooden materials, leveraging advancements in the YOLOv8 model, named ICDW-YOLO (improved crack detection for wooden material-YOLO). The ICDW-YOLO model introduces novel designs for the neck network and layer structure, along with an anchor algorithm, which features a dual-layer attention mechanism and dynamic gradient gain characteristics to optimize and enhance the original model. Initially, a new layer structure was crafted using GSConv and GS bottleneck, improving the model's recognition accuracy by maximizing the preservation of hidden channel connections. Subsequently, enhancements to the network are achieved through the gather-distribute mechanism, aimed at augmenting the fusion capability of multi-scale features and introducing a higher-resolution input layer to enhance small target recognition. Empirical results obtained from a customized wooden material crack detection dataset demonstrate the efficacy of the proposed ICDW-YOLO algorithm in effectively detecting targets. Without significant augmentation in model complexity, the mAP50-95 metric attains 79.018%, marking a 1.869% improvement over YOLOv8. Further validation of our algorithm's effectiveness is conducted through experiments on fire and smoke detection datasets, aerial remote sensing image datasets, and the coco128 dataset. The results showcase that ICDW-YOLO achieves a mAP50 of 69.226% and a mAP50-95 of 44.210%, indicating robust generalization and competitiveness vis-à-vis state-of-the-art detectors.

ROSE: Multi-level super-resolution-oriented semantic embedding for 3D microvasculature segmentation from low-resolution images

Current state-of-the-art segmentation methods often require high-resolution input to attain the high performance, which pushes the limit of data acquisition and brings large computation budgets. Instead, we present an end-to-end deep learning-based method, ROSE, for robust and precise segmentation of high-quality 3D super-resolution (SR) microvasculatures from low-resolution (LR) images as input, which can transform data from the LR imaging domain to the SR semantic domain (cross different modalities and scales). More specifically, a multi-tasking two-stream deep learning framework is proposed to learn the high-fidelity microvasculature SR image and semantic hybrid features simultaneously. During the proposed joint learning process, the high-resolution features of microvasculatures are further enhanced by the learned fine-grained structural/textural features from the microvasculature SR stream with a multi-level embedding scheme through the oriented feature aggregation at different fusion stages. In the constructed joint multi-level hybrid embedding spaces, the instance semantic embedding and the SR imaging embedding can be connected and integrated synergistically. We have conducted extensive experiments using public and real patient micro-cerebrovascular image datasets and compare our framework with traditional 3D vessel segmentation methods and the other state-of-the-art in deep learning. This robust and precise microvascular visualization in different brain regions by our method demonstrates its potential impact in magnetic resonance (MR) angiography and venography for the diagnosis of microvascular disease.

Spatial Downscaling of ESA CCI Soil Moisture Data Based on Deep Learning with an Attention Mechanism

Soil moisture (SM) is a critical variable affecting ecosystem carbon and water cycles and their feedback to climate change. In this study, we proposed a convolutional neural network (CNN) model embedded with a residual block and attention module, named SMNet, to spatially downscale the European Space Agency (ESA) Climate Change Initiative (CCI) SM product. In the SMNet model, a lightweight Convolutional Block Attention Module (CBAM) dual-attention mechanism was integrated to comprehensively extract the spatial and channel information from the high-resolution input remote sensing products, the reanalysis meteorological dataset, and the topographic data. The model was employed to downscale the ESA CCI SM from its original spatial resolution of 25 km to 1 km in California, USA, in the annual growing season (1 May to 30 September) from 2003 to 2021. The original ESA CCI SM data and in situ SM measurements (0–5 cm depth) from the International Soil Moisture Network were used to validate the model’s performance. The results show that compared with the original ESA CCI SM data, the downscaled SM data have comparable accuracy with a mean correlation (R) and root mean square error (RMSE) of 0.82 and 0.052 m3/m3, respectively. Moreover, the model generates reasonable spatiotemporal SM patterns with higher accuracy in the western region and relatively lower accuracy in the eastern Nevada mountainous area. In situ site validation results in the SCAN, the SNOTEL network, and the USCRN reveal that the R and RMSE are 0.62, 0.63, and 0.77, and 0.077 m3/m3, 0.093 m3/m3, and 0.078 m3/m3, respectively. The results are slightly lower than the validation results from the original ESA CCI SM data. Overall, the validation results suggest that the SMNet downscaling model proposed in this study has satisfactory performance in handling the task of soil moisture downscaling. The downscaled SM model not only preserves a high level of spatial consistency with the original ESA CCI SM model but also offers more intricate spatial variations in SM depending on the spatial resolution of model input data.

Temporal variations in micropollutant inlet concentrations matter when planning the design and compliance assessment of stormwater control measures

Stormwater Control Measures (SCMs) contribute to reducing micropollutant emissions from separate sewer systems. SCM planning and design are often performed by looking at the hydrological performance. Assessment of pollutant removal and the ability to comply with discharge concentration limits is often simplified due to a lack of data and limited monitoring resources. This study analyses the impact of using different time resolutions of input stormwater concentrations when assessing the compliance of SCMs against water quality standards. The behaviour of three indicator micropollutants (MP - Copper, Diuron, Benzo[a]pyrene) was assessed in four SCM archetypes, which were defined to represent typical SCM removal processes. High resolution MP data were extrapolated by using high resolution (2 min) measurements of TSS over a long period (343 events). The compliance assessment showed that high resolution input concentrations can result in a different level of compliance with water quality standards, especially when discharged concentrations are close to the limit values. This study underlines the importance of considering the high temporal variability of stormwater micropollutants when planning and designing SCMs to identify the most effective solutions for stormwater pollution management and to ensure a thorough consideration of all the environmental implications.

SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation

High-resolution representation is essential for achieving good performance in human pose estimation models. To obtain such features, existing works utilize high-resolution input images or fine-grained image tokens. However, this dense high-resolution representation brings a significant computational burden. In this paper, we address the following question: "Only sparse human keypoint locations are detected for human pose estimation, is it really necessary to describe the whole image in a dense, high-resolution manner?" Based on dynamic transformer models, we propose a framework that only uses Sparse High-resolution Representations for human Pose estimation (SHaRPose). In detail, SHaRPose consists of two stages. At the coarse stage, the relations between image regions and keypoints are dynamically mined while a coarse estimation is generated. Then, a quality predictor is applied to decide whether the coarse estimation results should be refined. At the fine stage, SHaRPose builds sparse high-resolution representations only on the regions related to the keypoints and provides refined high-precision human pose estimations. Extensive experiments demonstrate the outstanding performance of the proposed method. Specifically, compared to the state-of-the-art method ViTPose, our model SHaRPose-Base achieves 77.4 AP (+0.5 AP) on the COCO validation set and 76.7 AP (+0.5 AP) on the COCO test-dev set, and infers at a speed of 1.4x faster than ViTPose-Base. Code is available at https://github.com/AnxQ/sharpose.

Evaluating SLAM Performance with Synthesized Datasets from Unreal-Based Emulators

The rapid advancement in visual-inertial simultaneous localization and mapping (SLAM) has opened numerous applications in computer vision. However, the scarcity of high quality, publicly accessible datasets hampers the evaluation of SLAM performance in varied and tailored environments. In this study, I employed the AirSim simulator and the Unreal Engine 4 to generate a trajectory resembling that of the TUM VI Room 1 ground truth dataset within the ArchViz indoor environment representing a well-lit, furnished room. I further modified the environment and trajectory through various expansions, addition of features, and data smoothing to ensure a more stable sequence of input frames into the SLAM architecture. I then examined the efficiency of visual ORB-SLAM3 by inputting images of resolution 256×144 and 512×288 at 30 frames per second (FPS), while also adjusting the feature threshold - the maximum number of feature points that ORB-SLAM3 tracks per frame. This investigation of the camera parameters within AirSim and ORB-SLAM3 has led to the essential finding that the resolution of the input images must coincide with the dimensions of the film. The subsequent runs under these variables reveal that higher resolution images lead to considerably better tracking, with an optimal feature threshold ranging between 3000~12000 feature points per frame. Moreover, ORB- SLAM3 demonstrated significantly enhanced robustness within dynamic environments containing moving objects when using higher resolution inputs, with a decreased error of close to 0cm compared to 23.19cm for lower resolutions (averaged over three runs). Finally, I conducted qualitative testing using real-life indoor environments recorded with an iPhone Xr camera, which produces results that highlight the challenges faced by ORB-SLAM3 due to factors such as glare and motion blur.

DualRC: A Dual-Resolution Learning Framework With Neighbourhood Consensus for Visual Correspondences.

We address the problem of establishing accurate correspondences between two images. We present a flexible framework that can easily adapt to both geometric and semantic matching. Our contribution consists of three parts. Firstly, we propose an end-to-end trainable framework that uses the coarse-to-fine matching strategy to accurately find the correspondences. We generate feature maps in two levels of resolution, enforce the neighbourhood consensus constraint on the coarse feature maps by 4D convolutions and use the resulting correlation map to regulate the matches from the fine feature maps. Secondly, we present three variants of the model with different focuses. Namely, a universal correspondence model named DualRC that is suitable for both geometric and semantic matching, an efficient model named DualRC-L tailored for geometric matching with a lightweight neighbourhood consensus module that significantly accelerates the pipeline for high-resolution input images, and the DualRC-D model in which we propose a novel dynamically adaptive neighbourhood consensus module (DyANC) that dynamically selects the most suitable non-isotropic 4D convolutional kernels with the proper neighbourhood size to account for the scale variation. Last, we thoroughly experiment on public benchmarks for both geometric and semantic matching, showing superior performance in both cases.

Rethinking Dual-Stream Super-Resolution Semantic Learning in Medical Image Segmentation.

Image segmentation is fundamental task for medical image analysis, whose accuracy is improved by the development of neural networks. However, the existing algorithms that achieve high-resolution performance require high-resolution input, resulting in substantial computational expenses and limiting their applicability in the medical field. Several studies have proposed dual-stream learning frameworks incorporating a super-resolution task as auxiliary. In this paper, we rethink these frameworks and reveal that the feature similarity between tasks is insufficient to constrain vessels or lesion segmentation in the medical field, due to their small proportion in the image. To address this issue, we propose a DS2F (Dual-Stream Shared Feature) framework, including a Shared Feature Extraction Module (SFEM). Specifically, we present Multi-Scale Cross Gate (MSCG) utilizing multi-scale features as a novel example of SFEM. Then we define a proxy task and proxy loss to enable the features focus on the targets based on the assumption that a limited set of shared features between tasks is helpful for their performance. Extensive experiments on six publicly available datasets across three different scenarios are conducted to verify the effectiveness of our framework. Furthermore, various ablation studies are conducted to demonstrate the significance of our DS2F.

RMLANet: Random Multi-Level Attention Network for Shadow Detection and Removal

This paper addresses the problem of shadow detection and shadow removal from a single image. Despite awareness of utilizing both local and global contexts, previous works only aggregate features level by level in a coarse-to-fine manner. To overcome this problem, we present RMLANet, a novel Random Multi-Level Attention Network. To be specific, we first design a shuffled multi-level feature aggregation module to fuse the multi-level features and the guiding features using the self-attention mechanism. Nevertheless, the computational complexity of dense self-attention is unaffordable when processing high-resolution inputs. We argue that dense attention between any pixel pair is unnecessary due to the local consistency in images. Then we further propose a sparse attention mechanism to reduce the number of attention pairs, which greatly reduces the computational complexity. Through extensive experiments on four shadow detection and three shadow removal benchmark datasets, our proposed RMLANet achieves superior performance over current state-of-the-art approaches for both shadow detection and shadow removal. Codes are publicly available at https://github.com/LeipingJie/RMLANet.

Texture-Driven Adaptive Mesh Refinement with Application to 3D Relief

A high-quality 3D relief requires an appropriate refinement that has accurate carving boundaries with a limited number of added polygons. Most existing refinement methods cannot be applied to 3D reliefs directly, as they exhibit a mixture of problems such as not accurately following the texture contours, creating ill-shaped triangles, and excessively increasing the polygon count. We introduce an efficient texture-driven method to adaptively refine a mesh for 3D reliefs. From the user-provided binary texture, we conduct a feature-preserving self-adaptive sampling of texture contours. Our other inputs are a 3D mesh and the mapping of that mesh to texture space. We adapt a constraint-driven Red–Green subdivision to locally subdivide the mesh around the contours. Then, we conduct the adaptive mesh refinement by introducing a feature-adaptive 3-subdivision. Finally, we apply the proposed algorithm to 3D reliefs, which enables generating a distinct relief. The presented method can attain accurate 3D relief while maintaining good mesh quality without the necessity of a high-resolution input. When compared to alternative approaches, ours consistently demonstrates superior polygon quality and maintains relief boundaries that closely follow the texture contours.

CHEX-MATE: A non-parametric deep learning technique to deproject and deconvolve galaxy cluster X-ray temperature profiles

Temperature profiles of the hot galaxy cluster intracluster medium (ICM) have a complex non-linear structure that traditional parametric modelling may fail to fully approximate. For this study, we made use of neural networks, for the first time, to construct a data-driven non-parametric model of ICM temperature profiles. A new deconvolution algorithm was then introduced to uncover the true (3D) temperature profiles from the observed projected (2D) temperature profiles. An auto-encoder-inspired neural network was first trained by learning a non-linear interpolatory scheme to build the underlying model of 3D temperature profiles in the radial range of [0.02–2] R500, using a sparse set of hydrodynamical simulations from the THREE HUNDRED PROJECT. A deconvolution algorithm using a learning-based regularisation scheme was then developed. The model was tested using high and low resolution input temperature profiles, such as those expected from simulations and observations, respectively. We find that the proposed deconvolution and deprojection algorithm is robust with respect to the quality of the data, the morphology of the cluster, and the deprojection scheme used. The algorithm can recover unbiased 3D radial temperature profiles with a precision of around 5% over most of the fitting range. We apply the method to the first sample of temperature profiles obtained with XMM-Newton for the CHEX-MATE project and compared it to parametric deprojection and deconvolution techniques. Our work sets the stage for future studies that focus on the deconvolution of the thermal profiles (temperature, density, pressure) of the ICM and the dark matter profiles in galaxy clusters, using deep learning techniques in conjunction with X-ray, Sunyaev Zel’Dovich (SZ) and optical datasets.

Ionospheric TEC prediction using FFNN during five different X Class solar flares of 2021 and 2022 and comparison with COKSM and IRI PLAS 2017

Abstract The Ionospheric Total Electron Content (TEC) measured in the ray path of the signals directly contributes to the Range Error (RE) of the satellite signals, which affects positioning and navigation. Employing the Co-Kriging-based Surrogate Model (COKSM) to predict TEC and RE correction has proven prolific. This research attempted to test and compare the prediction capability of COKSM with an Artificial Intelligence-based Feed Forward Neural Network model (FFNN) during five X-Class Solar Flares of 2021–22. Also, the results are validated by comparing them with the IRI PLAS 2017 model. TEC, solar, and geomagnetic parameters data for Hyderabad GPS station located at 17.31° N latitude and 78.55° E longitude were collected from IONOLAB & OMNIWEB servers. The COKSM uses six days of input data to predict the 7th day TEC, whereas prediction using the FFNN model is done using 45 days of data before the prediction date. The performance evaluation is done using RMSE, NRMSE, Correlation Coefficient, and sMAPE. The average RMSE for COKSM varied from 1.9 to 9.05, for FFNN it varied from 2.72 to 7.69, and for IRI PLAS 2017 it varied from 7.39 to 11.24. Likewise, evaluation done for three different models over five different X-class solar flare events showed that the COKSM performed well during the high-intensity solar flare conditions. On the other hand, the FFNN model performed well during high-resolution input data conditions. Also, it is notable that both models performed better than the IRI PLAS 2017 model and are suitable for navigational applications.

Dual Vision Transformer.

Recent advances have presented several strategies to mitigate the computations of self-attention mechanism with high-resolution inputs. Many of these works consider decomposing the global self-attention procedure over image patches into regional and local feature extraction procedures that each incurs a smaller computational complexity. Despite good efficiency, these approaches seldom explore the holistic interactions among all patches, and are thus difficult to fully capture the global semantics. In this paper, we propose a novel Transformer architecture that elegantly exploits the global semantics for self-attention learning, namely Dual Vision Transformer (Dual-ViT). The new architecture incorporates a critical semantic pathway that can more efficiently compress token vectors into global semantics with reduced order of complexity. Such compressed global semantics then serve as useful prior information in learning finer local pixel level details, through another constructed pixel pathway. The semantic pathway and pixel pathway are integrated together and are jointly trained, spreading the enhanced self-attention information in parallel through both of the pathways. Dual-ViT is henceforth able to capitalize on global semantics to boost self-attention learning without compromising much computational complexity. We empirically demonstrate that Dual-ViT provides superior accuracy than SOTA Transformer architectures with comparable training complexity. Source codes are available at https://github.com/YehLi/ImageNetModel.

CCT-Unet: A U-Shaped Network Based on Convolution Coupled Transformer for Segmentation of Peripheral and Transition Zones in Prostate MRI.

The accurate segmentation of prostate region in magnetic resonance imaging (MRI) can provide reliable basis for artificially intelligent diagnosis of prostate cancer. Transformer-based models have been increasingly used in image analysis due to their ability to acquire long-term global contextual features. Although Transformer can provide feature representations of the overall appearance and contour representations at long distance, it does not perform well on small-scale datasets of prostate MRI due to its insensitivity to local variation such as the heterogeneity of the grayscale intensities in the peripheral zone and transition zone across patients; meanwhile, the convolutional neural network (CNN) could retain these local features well. Therefore, a robust prostate segmentation model that can aggregate the characteristics of CNN and Transformer is desired. In this work, a U-shaped network based on the convolution coupled Transformer is proposed for segmentation of peripheral and transition zones in prostate MRI, named the convolution coupled Transformer U-Net (CCT-Unet). The convolutional embedding block is first designed for encoding high-resolution input to retain the edge detail of the image. Then the convolution coupled Transformer block is proposed to enhance the ability of local feature extraction and capture long-term correlation that encompass anatomical information. The feature conversion module is also proposed to alleviate the semantic gap in the process of jumping connection. Extensive experiments have been conducted to compare our CCT-Unet with several state-of-the-art methods on both the ProstateX open dataset and the self-bulit Huashan dataset, and the results have consistently shown the accuracy and robustness of our CCT-Unet in MRI prostate segmentation.