Rich Detail Research Articles

Infrared and visible image fusion in a rolling guided filtering framework based on deep feature extraction

To preserve rich detail information and high contrast, a novel image fusion algorithm is proposed based on rolling-guided filtering combined with deep feature extraction. Firstly, input images are filtered to acquire various scales decomposed images using rolling guided filtering. Subsequently, PCANet is introduced to extract weight maps to guide base layer fusion. For the others layer, saliency maps of input images are extracted by a saliency measure. Then, the saliency maps are optimized by guided filtering to guide the detail layer fusion. Finally, the final fusion result are reconstructed by all fusion layers. The experimental fusion results demonstrate that fusion algorithm in this study obtains following advantages of rich detail information, high contrast, and complete edge information preservation in the subjective evaluation and better results in the objective evaluation index. In particular, the proposed method is 16.9% ahead of the best comparison result in the SD objective evaluation index.

A binarization method for logos

In the field of digital image processing, the binarization of logo images presents a significant challenge due to the diversity in their color distribution and graphical structures. Logos, as distinctive visual symbols, are integral to brand recognition and commercial advertising. Traditional binarization methods often struggle with this diversity, leading to difficulties in preserving the original color and details of the images. To address this issue, this paper introduces a novel binarization approach specifically designed for logo images. The method comprises three stages: preprocessing, binarization, and noise removal. In the binarization stage, the Otsu method is adapted from a global to a local thresholding technique, combining both global and local thresholds for each pixel. The efficacy of this method is demonstrated through comparative experiments with logos of varying colors and structures, showing superior performance in maintaining image color and detail restoration over traditional methods. Additionally, the preprocessing and noise removal steps further enhance the method's effectiveness in handling logos with rich background colors and complex details. Overall, this approach proves to be highly adaptable across a wide range of logo types.

Discriminative context-aware network for camouflaged object detection.

Animals use camouflage (background matching, disruptive coloration, etc.) for protection, confusing predators and making detection difficult. Camouflage Object Detection (COD) tackles this challenge by identifying objects seamlessly blended into their surroundings. Existing COD techniques struggle with hidden objects due to noisy inferences inherent in natural environments. To address this, we propose the Discriminative Context-aware Network (DiCANet) for improved COD performance. DiCANet addresses camouflage challenges through a two-stage approach. First, an adaptive restoration block intelligently learns feature weights, prioritizing informative channels and pixels. This enhances convolutional neural networks' ability to represent diverse data and handle complex camouflage. Second, a cascaded detection module with an enlarged receptive field refines the object prediction map, achieving clear boundaries without post-processing. Without post-processing, DiCANet achieves state-of-the-art performance on challenging COD datasets (CAMO, CHAMELEON, COD10K) by generating accurate saliency maps with rich contextual details and precise boundaries. DiCANet tackles the challenge of identifying camouflaged objects in noisy environments with its two-stage restoration and cascaded detection approach. This innovative architecture surpasses existing methods in COD tasks, as proven by benchmark dataset experiments.

Hybrid-Supervised Dual-Search: Leveraging Automatic Learning for Loss-Free Multi-Exposure Image Fusion

Multi-exposure image fusion (MEF) has emerged as a prominent solution to address the limitations of digital imaging in representing varied exposure levels. Despite its advancements, the field grapples with challenges, notably the reliance on manual designs for network structures and loss functions, and the constraints of utilizing simulated reference images as ground truths. Consequently, current methodologies often suffer from color distortions and exposure artifacts, further complicating the quest for authentic image representation. In addressing these challenges, this paper presents a Hybrid-Supervised Dual-Search approach for MEF, dubbed HSDS-MEF, which introduces a bi-level optimization search scheme for automatic design of both network structures and loss functions. More specifically, we harness a unique dual research mechanism rooted in a novel weighted structure refinement architecture search. Besides, a hybrid supervised contrast constraint seamlessly guides and integrates with searching process, facilitating a more adaptive and comprehensive search for optimal loss functions. We realize the state-of-the-art performance in comparison to various competitive schemes, yielding a 10.61% and 4.38% improvement in Visual Information Fidelity (VIF) for general and no-reference scenarios, respectively, while providing results with high contrast, rich details and colors. The code is available at https://github.com/RollingPlain/HSDS_MEF.

Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping

Adversarial examples generated by a surrogate model typically exhibit limited transferability to unknown target systems. To address this problem, many transferability enhancement approaches (e.g., input transformation and model augmentation) have been proposed. However, they show poor performances in attacking systems having different model genera from the surrogate model. In this paper, we propose a novel and generic attacking strategy, called Deformation-Constrained Warping Attack (DeCoWA), that can be effectively applied to cross model genus attack. Specifically, DeCoWA firstly augments input examples via an elastic deformation, namely Deformation-Constrained Warping (DeCoW), to obtain rich local details of the augmented input. To avoid severe distortion of global semantics led by random deformation, DeCoW further constrains the strength and direction of the warping transformation by a novel adaptive control strategy. Extensive experiments demonstrate that the transferable examples crafted by our DeCoWA on CNN surrogates can significantly hinder the performance of Transformers (and vice versa) on various tasks, including image classification, video action recognition, and audio recognition. Code is made available at https://github.com/LinQinLiang/DeCoWA.

The infrared absorption spectrum of phenylacetylene and its deuterated isotopologue in the mid- to far-IR.

Anharmonicity strongly influences the absorption and emission spectra of polycyclic aromatic hydrocarbon (PAH) molecules. Here, IR-UV ion-dip spectroscopy experiments together with detailed anharmonic computations reveal the presence of fundamental, overtone, as well as 2- and 3-quanta combination band transitions in the far- and mid-infrared absorption spectra of phenylacetylene and its singly deuterated isotopologue. Strong absorption features in the 400-900 cm-1 range originate from CH(D) in-plane and out-of-plane wags and bends, as well as bending motions including the C≡C and CH bonds of the acetylene substituent and the aromatic ring. For phenylacetylene, every absorption feature is assigned either directly or indirectly to a single or multiple vibrational mode(s). The measured spectrum is dense, broad, and structureless in many regions but well characterized by computations. Upon deuteration, large isotopic shifts are observed. At frequencies above 1500 cm-1 for d1-phenylacetylene, a one-to-one match is seen when comparing computations and experiments with all features assigned to combination bands and overtones. The C≡C stretch observed in phenylacetylene is not observed in d1-phenylacetylene due to a computed 40-fold drop in intensity. Overall, a careful treatment of anharmonicity that includes 2- and 3-quanta modes is found to be crucial to understand the rich details of the infrared spectrum of phenylacetylene. Based on these results, it can be expected that such an all-inclusive anharmonic treatment will also be key for unraveling the infrared spectra of PAHs in general.

PixRevive: Latent Feature Diffusion Model for Compressed Video Quality Enhancement.

In recent years, the rapid prevalence of high-definition video in Internet of Things (IoT) systems has been directly facilitated by advances in imaging sensor technology. To adapt to limited uplink bandwidth, most media platforms opt to compress videos to bitrate streams for transmission. However, this compression often leads to significant texture loss and artifacts, which severely degrade the Quality of Experience (QoE). We propose a latent feature diffusion model (LFDM) for compressed video quality enhancement, which comprises a compact edge latent feature prior network (ELPN) and a conditional noise prediction network (CNPN). Specifically, we first pre-train ELPNet to construct a latent feature space that captures rich detail information for representing sharpness latent variables. Second, we incorporate these latent variables into the prediction network to iteratively guide the generation direction, thus resolving the problem that the direct application of diffusion models to temporal prediction disrupts inter-frame dependencies, thereby completing the modeling of temporal correlations. Lastly, we innovatively develop a Grouped Domain Fusion module that effectively addresses the challenges of diffusion distortion caused by naive cross-domain information fusion. Comparative experiments on the MFQEv2 benchmark validate our algorithm's superior performance in terms of both objective and subjective metrics. By integrating with codecs and image sensors, our method can provide higher video quality.

A novel framework combining band selection algorithm and improved 3D prototypical network for tree species classification using airborne hyperspectral images

Fine-grained identification of forest types and tree species represents a critical aspect of forest resource inventory and monitoring. The use of airborne hyperspectral remote sensing imagery stands out for its ability to finely differentiate among tree species, leveraging its exceptional spatial resolution and rich spectral details. However, this approach is limited by several challenges (e.g., high spectral correlation and information redundancy). In accordance, the adoption of a lightweight deep learning approach in the form of a few-shot learning model can effectively resolve the challenges of multi-forest tree species classification. Therefore, integrating a data dimensionality reduction algorithm with a few-shot classification model presents a promising avenue for resolving the fine-grained classification of forest tree species. In this study, we propose the innovative classification framework FAST 3D-CNN P-Net. This framework utilizes CNN for band selection, enhances the fine-grained identification process in hyperspectral data, and integrates an optimized FAST 3D-CNN into the P-Net classifier (a few-shot classifier). First, a CNN-based band selection method is employed to learn the nonlinear dependencies between spectral bands, assign weights to rank the bands, and reconstruct the global spectral information using the most informative bands. It then constructs a novel classification model, designated FAST 3D-CNN P-Net, through the integration of an optimal 3D-CNN with a prototypical network. To enhance classification performance, the FAST 3D-CNN P-Net utilizes reconstructed hyperspectral images derived from the band selection results as input. The effectiveness of the proposed framework was assessed with the airborne GFF dataset and the widely accessible medium-resolution hyperspectral datasets, Indian Pines (IP) and Kennedy Space Center (KSC). The overall classification accuracy reached 98.33 % for the GFF dataset and 97.21 % and 99.43 % for the IP and KSC, respectively, exhibiting performance superiority compared to the standalone 3D-CNN classification network. This classification framework demonstrates efficiency in selecting a subset of hyperspectral bands with minimal redundancy, empowering the rapid and accurate classification and mapping of tree species in complicated, multi-species forest stands, even with a limited quantity of labeled samples.

3D-MRI super-resolution reconstruction using multi-modality based on multi-resolution CNN

Background and ObjectiveHigh-resolution (HR) MR images provide rich structural detail to assist physicians in clinical diagnosis and treatment plan. However, it is arduous to acquire HR MRI due to equipment limitations, scanning time or patient comfort. Instead, HR MRI could be obtained through a number of computer assisted post-processing methods that have proven to be effective and reliable. This paper aims to develop a convolutional neural network (CNN) based super-resolution reconstruction framework for low-resolution (LR) T2w images. MethodIn this paper, we propose a novel multi-modal HR MRI generation framework based on deep learning techniques. Specifically, we construct a CNN based on multi-resolution analysis to learn an end-to-end mapping between LR T2w and HR T2w, where HR T1w is fed into the network to offer detailed a priori information to help generate HR T2w. Furthermore, a low-frequency filtering module is introduced to filter out the interference from HR-T1w during high-frequency information extraction. Based on the idea of multi-resolution analysis, detailed features extracted from HR T1w and LR T2w are fused at two scales in the network and then HR T2w is reconstructed by upsampling and dense connectivity module. ResultsExtensive quantitative and qualitative evaluations demonstrate that the proposed method enhances the recovered HR T2w details and outperforms other state-of-the-art methods. In addition, the experimental results also suggest that our network has a lightweight structure and favorable generalization performance. ConclusionThe results show that the proposed method is capable of reconstructing HR T2w with higher accuracy. Meanwhile, the super-resolution reconstruction results on other dataset illustrate the excellent generalization ability of the method.

Light-field flow cytometry for high-resolution, volumetric and multiparametric 3D single-cell analysis

Imaging flow cytometry (IFC) combines flow cytometry and fluorescence microscopy to enable high-throughput, multiparametric single-cell analysis with rich spatial details. However, current IFC techniques remain limited in their ability to reveal subcellular information with a high 3D resolution, throughput, sensitivity, and instrumental simplicity. In this study, we introduce a light-field flow cytometer (LFC), an IFC system capable of high-content, single-shot, and multi-color acquisition of up to 5,750 cells per second with a near-diffraction-limited resolution of 400-600 nm in all three dimensions. The LFC system integrates optical, microfluidic, and computational strategies to facilitate the volumetric visualization of various 3D subcellular characteristics through convenient access to commonly used epi-fluorescence platforms. We demonstrate the effectiveness of LFC in assaying, analyzing, and enumerating intricate subcellular morphology, function, and heterogeneity using various phantoms and biological specimens. The advancement offered by the LFC system presents a promising methodological pathway for broad cell biological and translational discoveries, with the potential for widespread adoption in biomedical research.

3D Reconstruction of Wheat Plants by Integrating Point Cloud Data and Virtual Design Optimization

The morphology and structure of wheat plants are intricate, containing numerous tillers, rich details, and significant cross-obscuration. Methods of effectively reconstructing three-dimensional (3D) models of wheat plants that reflects the varietal architectural differences using measured data is challenging in plant phenomics and functional–structural plant models. This paper proposes a 3D reconstruction technique for wheat plants that integrates point cloud data and virtual design optimization. The approach extracted single stem number, growth position, length, and inclination angle from the point cloud data of a wheat plant. It then built an initial 3D mesh model of the plant by integrating a wheat 3D phytomer template database with variety resolution. Diverse 3D wheat plant models were subsequently virtually designed by iteratively modifying the leaf azimuth, based on the initial model. Using the 3D point cloud of the plant as the overall constraint and setting the minimum Chamfer distance between the point cloud and the mesh model as the optimization objective, we obtained the optimal 3D model as the reconstruction result of the plant through continuous iterative calculation. The method was validated using 27 winter wheat plants, with nine varieties and three replicates each. The R2 values between the measured data and the reconstructed plants were 0.80, 0.73, 0.90, and 0.69 for plant height, crown width, plant leaf area, and coverage, respectively. Additionally, the Normalized Root Mean Squared Errors (NRMSEs) were 0.10, 0.12, 0.08, and 0.17, respectively. The Mean Absolute Percentage Errors (MAPEs) used to investigate the vertical spatial distribution between the reconstructed 3D models and the point clouds of the plants ranged from 4.95% to 17.90%. These results demonstrate that the reconstructed 3D model exhibits satisfactory consistency with the measured data, including plant phenotype and vertical spatial distribution, and accurately reflects the characteristics of plant architecture and spatial distribution for the utilized wheat cultivars. This method provides technical support for research on wheat plant phenotyping and functional–structural analysis.

Natural Image Reconstruction from fMRI Based on Node-Edge Interaction and Multi-Scale Constraint.

Reconstructing natural stimulus images using functional magnetic resonance imaging (fMRI) is one of the most challenging problems in brain decoding and is also the crucial component of a brain-computer interface. Previous methods cannot fully exploit the information about interactions among brain regions. In this paper, we propose a natural image reconstruction method based on node-edge interaction and a multi-scale constraint. Inspired by the extensive information interactions in the brain, a novel graph neural network block with node-edge interaction (NEI-GNN block) is presented, which can adequately model the information exchange between brain areas via alternatively updating the nodes and edges. Additionally, to enhance the quality of reconstructed images in terms of both global structure and local detail, we employ a multi-stage reconstruction network that restricts the reconstructed images in a coarse-to-fine manner across multiple scales. Qualitative experiments on the generic object decoding (GOD) dataset demonstrate that the reconstructed images contain accurate structural information and rich texture details. Furthermore, the proposed method surpasses the existing state-of-the-art methods in terms of accuracy in the commonly used n-way evaluation. Our approach achieves 82.00%, 59.40%, 45.20% in n-way mean squared error (MSE) evaluation and 83.50%, 61.80%, 46.00% in n-way structural similarity index measure (SSIM) evaluation, respectively. Our experiments reveal the importance of information interaction among brain areas and also demonstrate the potential for developing visual-decoding brain-computer interfaces.

Hyperspectral and multispectral image fusion via residual selective kernel attention-based U-net

ABSTRACT The fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) is a crucial technology for producing high-resolution hyperspectral images. Most existing image fusion algorithms based on deep learning do not fully utilize the ability of neural network to extract and process multi-scale features, which leads to the problem of difficulty in fully learning features and ambiguity of features. In order to overcome these issues, a residual selective kernel attention-based U-net named RSKAU-net is designed for LR-HSI and HR-MSI fusion. RSKAU-net is constructed by a residual selective kernel module with an attention mechanism and a channel attention block. The residual selective kernel attention-based (RSKA) module is designed to process images of different resolutions, which adaptively extracts multi-scale features and efficiently emphasizes significant features through the attention mechanism. The channel attention (CA) module retains important spectral information by assigning different weights to each channel of LR-HSI. The proposed network can enhance the spatial information of LR-HSI while preserving its spectral information. Meanwhile, it effectively fuses the features from the source image to obtain the HR-HSI with rich details. The experimental results demonstrate that the proposed network has advantages in terms of both visual effect and objective quantitative indices when compared to existing HSI-MSI fusion approaches.

Dark Light Image-Enhancement Method Based on Multiple Self-Encoding Prior Collaborative Constraints

The purpose of dark image enhancement is to restore dark images to visual images under normal lighting conditions. Due to the ill-posedness of the enhancement process, previous enhancement algorithms often have overexposure, underexposure, noise increases and artifacts when dealing with complex and changeable images, and the robustness is poor. This article proposes a new enhancement approach consisting in constructing a dim light enhancement network with more robustness and rich detail features through the collaborative constraint of multiple self-coding priors (CCMP). Specifically, our model consists of two prior modules and an enhancement module. The former learns the feature distribution of the dark light image under normal exposure as an a priori term of the enhancement process through multiple specific autoencoders, implicitly measures the enhancement quality and drives the network to approach the truth value. The latter fits the curve mapping of the enhancement process as a fidelity term to restore global illumination and local details. Through experiments, we concluded that the new method proposed in this article can achieve more excellent quantitative and qualitative results, improve detail contrast, reduce artifacts and noise, and is suitable for dark light enhancement in multiple scenes.

Paint-CUT: A Generative Model for Chinese Landscape Painting Based on Shuffle Attentional Residual Block and Edge Enhancement

As one of the precious cultural heritages, Chinese landscape painting has developed unique styles and techniques. Researching the intelligent generation of Chinese landscape paintings from photos can benefit the inheritance of traditional Chinese culture. To address detail loss, blurred outlines, and poor style transfer in present generated results, a model for generating Chinese landscape paintings from photos named Paint-CUT is proposed. In order to solve the problem of detail loss, the SA-ResBlock module is proposed by combining shuffle attention with the resblocks in the generator, which is used to enhance the generator’s ability to extract the main scene information and texture features. In order to solve the problem of poor style transfer, perceptual loss is introduced to constrain the model in terms of content and style. The pre-trained VGG is used to extract the content and style features to calculate the perceptual loss and, then, the loss can guide the model to generate landscape paintings with similar content to landscape photos and a similar style to target landscape paintings. In order to solve the problem of blurred outlines in generated landscape paintings, edge loss is proposed to the model. The Canny edge detection is used to generate edge maps and, then, the edge loss between edge maps of landscape photos and generated landscape paintings is calculated. The generated landscape paintings have clear outlines and details by adding edge loss. Comparison experiments and ablation experiments are performed on the proposed model. Experiments show that the proposed model can generate Chinese landscape paintings with clear outlines, rich details, and realistic style. Generated paintings not only retain the details of landscape photos, such as texture and outlines of mountains, but also have similar styles to the target paintings, such as colors and brush strokes. So, the generation quality of Chinese landscape paintings has improved.

Toponymic surnames and the spatiality of heresy prosecutions: Peter Seila’s register of sentences from the Quercy region (Languedoc), 1241–1242

Our knowledge of the geography of medieval religious dissent and its repression in the Latin West is limited by a lack of systematic study of locational information in inquisition trial records. Spatial analysis of these rich details has the potential to help build a bottom-up picture of interaction between dissidents and inquisitors that moves beyond institutional perspectives. This task is rendered challenging due to the inconsistencies and uncertainties of what inquisitors and their notaries typically recorded about the spatial associations of suspects. Probably the most common indicator of such associations found in inquisition records are toponymic surnames. They present challenges of coverage (not everybody had a toponymic surname) and interpretation (multiple possible meanings). This study attempts to tackle the challenge of interpreting such surnames within the context of the nine sentencing events held by the inquisitor Peter Seila in 1241 and 1242 in the Quercy region of Languedoc: covering 650 sentenced individuals, the register documenting these events is the earliest extant record of an inquisition of such scale. Rather than taking the interpretative challenges of toponymic surnames as reasons to limit ourselves to qualitative analysis, our approach shows the value of rendering and analysing them as structured data. Firstly, we quantify the context of toponymic surnames, placing them against the background of broader name construction practices and other social factors. Secondly, we plot and analyse the geocoded data derived from toponymic surnames with the benefit of this contextualisation, looking especially at the distance of toponyms from their associated sentencing centres, in order to derive narratives that best explain the generality of their meaning. The results allow us to appraise the actual spatial coverage of the nine sentencing events. The first two, centred on the important towns of Montauban and Moissac, seem most likely to have been primarily urban affairs, with little evidence of rural coverage. The remainder, which took place in castra (fortified villages), appear to have covered more of the surrounding countryside. These results geographically contextualise the reports of dissidence conveyed within Peter’s register, and suggest narratives for how Peter optimised his strategy for impact in the face of constraints.

DSANet: Dynamic and Structure-Aware GCN for Sparse and Incomplete Point Cloud Learning.

Learning 3-D structures from incomplete point clouds with extreme sparsity and random distributions is a challenge since it is difficult to infer topological connectivity and structural details from fragmentary representations. Missing large portions of informative structures further aggravates this problem. To overcome this, a novel graph convolutional network (GCN) called dynamic and structure-aware NETwork (DSANet) is presented in this article. This framework is formulated based on a pyramidic auto-encoder (AE) architecture to address accurate structure reconstruction on the sparse and incomplete point clouds. A PointNet-like neural network is applied as the encoder to efficiently aggregate the global representations of coarse point clouds. On the decoder side, we design a dynamic graph learning module with a structure-aware attention (SAA) to take advantage of the topology relationships maintained in the dynamic latent graph. Relying on gradually unfolding the extracted representation into a sequence of graphs, DSANet is able to reconstruct complicated point clouds with rich and descriptive details. To associate analogous structure awareness with semantic estimation, we further propose a mechanism, called structure similarity assessment (SSA). This method allows our model to surmise semantic homogeneity in an unsupervised manner. Finally, we optimize the proposed model by minimizing a new distortion-aware objective end-to-end. Extensive qualitative and quantitative experiments demonstrate the impressive performance of our model in reconstructing unbroken 3-D shapes from deficient point clouds and preserving semantic relationships among different regional structures.

Hierarchical Self-Distilled Feature Learning for Fine-Grained Visual Categorization.

Fine-grained visual categorization (FGVC) relies on hierarchical features extracted by deep convolutional neural networks (CNNs) to recognize closely alike objects. Particularly, shallow layer features containing rich spatial details are vital for specifying subtle differences between objects but are usually inadequately optimized due to gradient vanishing during backpropagation. In this article, hierarchical self-distillation (HSD) is introduced to generate well-optimized CNNs features for accurate fine-grained categorization. HSD inherits from the widely applied deep supervision and implements multiple intermediate losses for reinforced gradients. Besides that, we observe that the hard (one-hot) labels adopted for intermediate supervision hurt the performance of FGVC by enforcing overstrict supervision. As a solution, HSD seeks self-distillation where soft predictions generated by deeper layers of the network are hierarchically exploited to supervise shallow parts. Moreover, self-information entropy loss (SIELoss) is designed in HSD to adaptively soften intermediate predictions and facilitate better convergence. In addition, the gradient detached fusion (GDF) module is incorporated to produce an ensemble result with multiscale features via effective feature fusion. Extensive experiments on four challenging fine-grained datasets show that, with neglectable parameter increase, the proposed HSD framework and the GDF module both bring significant performance gains over different backbones, which also achieves state-of-the-art classification performance.

Efficient Swin Transformer for Remote Sensing Image Super-Resolution.

Remote sensing super-resolution (SR) technique, which aims to generate high-resolution image with rich spatial details from its low-resolution counterpart, play a vital role in many applications. Recently, more and more studies attempt to explore the application of Transformer in remote sensing field. However, they suffer from the high computational burden and memory consumption for remote sensing super-resolution. In this paper, we propose an efficient Swin Transformer (ESTNet) via channel attention for SR of remote sensing images, which is composed of three components. First, a three-layer convolutional operation is utilized to extract shallow features of the input low-resolution image. Then, a residual group-wise attention module is proposed to extract the deep features, which contains an efficient channel attention block (ECAB) and a group-wise attention block (GAB). Finally, the extracted deep features are reconstructed to generate high-resolution remote sensing images. Extensive experimental results proclaim that the proposed ESTNet can obtain better super-resolution results with low computational burden. Compared to the recently proposed Transformer-based remote sensing super-resolution method, the number of parameters is reduced by 82.68% while the computational cost is reduced by 87.84%. The code of the proposed ESTNet will be available at https://github.com/PuhongDuan/ESTNet for reproducibility.

Neuromorphic Imaging With Joint Image Deblurring and Event Denoising.

Neuromorphic imaging reacts to per-pixel brightness changes of a dynamic scene with high temporal precision and responds with asynchronous streaming events as a result. It also often supports a simultaneous output of an intensity image. Nevertheless, the raw events typically involve a large amount of noise due to the high sensitivity of the sensor, while capturing fast-moving objects at low frame rates results in blurry images. These deficiencies significantly degrade human observation and machine processing. Fortunately, the two information sources are inherently complementary - events with microsecond-level temporal resolution, which are triggered by the edges of objects recorded in a latent sharp image, can supply rich motion details missing from the blurry one. In this work, we bring the two types of data together and introduce a simple yet effective unifying algorithm to jointly reconstruct blur-free images and noise-robust events in an iterative coarse-to-fine fashion. Specifically, an event-regularized prior offers precise high-frequency structures and dynamic features for blind deblurring, while image gradients serve as a kind of faithful supervision in regulating neuromorphic noise removal. Comprehensively evaluated on real and synthetic samples, such a synergy delivers superior reconstruction quality for both images with severe motion blur and raw event streams with a storm of noise, and also exhibits greater robustness to challenging realistic scenarios such as varying levels of illumination, contrast and motion magnitude. Meanwhile, it can be driven by much fewer events and holds a competitive edge at computational time overhead, rendering itself preferable as available computing resources are limited. Our solution gives impetus to the improvement of both sensing data and paves the way for highly accurate neuromorphic reasoning and analysis.