Modulation Transfer Research Articles

RGB-D Data Compression via Bi-Directional Cross-Modal Prior Transfer and Enhanced Entropy Modeling

RGB-D data, being homogeneous cross-modal data, demonstrates significant correlations among data elements. However, current research focuses only on a unidirectional pattern of cross-modal contextual information, neglecting the exploration of bidirectional relationships in the compression field. Thus, we propose a joint RGB-D compression scheme, which is combined with Bi-directional Cross-modal Prior Transfer (Bi-CPT) modules and a Bi-directional Cross-modal Enhanced Entropy (Bi-CEE) model. The Bi-CPT module is designed for compact representations of cross-modal features, effectively eliminating spatial and modality redundancies at different granularity levels. In contrast to the traditional entropy models, our proposed Bi-CEE model not only achieves spatial-channel contextual adaptation through partitioning RGB and depth features but also incorporates information from other modalities as prior to enhance the accuracy of probability estimation for latent variables. Furthermore, this model enables parallel multi-stage processing to accelerate coding. Experimental results demonstrate the superiority of our proposed framework over the current compression scheme, outperforming both rate-distortion performance and downstream tasks, including surface reconstruction and semantic segmentation. The source code will be available at https://github.com/xyy7/Learning-based-RGB-D-Image-Compression .

Enhancing 3D planetary atmosphere simulations with a surrogate radiative transfer model

Abstract This work introduces an approach to enhancing the computational efficiency of 3D atmospheric simulations by integrating a machine-learned surrogate model into the OASIS global circulation model (GCM). Traditional GCMs, which are based on repeatedly numerically integrating physical equations governing atmospheric processes across a series of time-steps, are time-intensive, leading to compromises in spatial and temporal resolution of simulations. This research improves upon this limitation, enabling higher resolution simulations within practical timeframes. Speeding up 3D simulations holds significant implications in multiple domains. Firstly, it facilitates the integration of 3D models into exoplanet inference pipelines, allowing for robust characterisation of exoplanets from a previously unseen wealth of data anticipated from JWST and post-JWST instruments. Secondly, acceleration of 3D models will enable higher resolution atmospheric simulations of Earth and Solar System planets, enabling more detailed insights into their atmospheric physics and chemistry. Our method replaces the radiative transfer module in OASIS with a recurrent neural network-based model trained on simulation inputs and outputs. Radiative transfer is typically one of the slowest components of a GCM, thus providing the largest scope for overall model speed-up. The surrogate model was trained and tested on the specific test case of the Venusian atmosphere, to benchmark the utility of this approach in the case of non-terrestrial atmospheres. This approach yields promising results, with the surrogate-integrated GCM demonstrating above 99.0% accuracy and 147 factor GPU speed-up of the entire simulation compared to using the matched original GCM under Venus-like conditions.

Stealthy Vehicle Adversarial Camouflage Texture Generation Based on Neural Style Transfer

Adversarial attacks that mislead deep neural networks (DNNs) into making incorrect predictions can also be implemented in the physical world. However, most of the existing adversarial camouflage textures that attack object detection models only consider the effectiveness of the attack, ignoring the stealthiness of adversarial attacks, resulting in the generated adversarial camouflage textures appearing abrupt to human observers. To address this issue, we propose a style transfer module added to an adversarial texture generation framework. By calculating the style loss between the texture and the specified style image, the adversarial texture generated by the model is guided to have good stealthiness and is not easily detected by DNNs and human observers in specific scenes. Experiments have shown that in both the digital and physical worlds, the vehicle full coverage adversarial camouflage texture we create has good stealthiness and can effectively fool advanced DNN object detectors while evading human observers in specific scenes.

Color restoration of mural images based on a reversible neural network: leveraging reversible residual networks for structure and texture preservation

AbstractThe Mogao Grottoes in Dunhuang, a treasure of China's and the world's cultural heritage, contains rich historical and cultural deposits and has left precious relics of the history of human art. Over centuries, the Mogao Caves have been affected by natural and human factors, resulting in irreversible fading and discoloration of many murals. In recent years, deep learning technology has shown great potential in the field of virtual mural color restoration. Therefore, this paper proposes a mural image color restoration method based on a reversible neural network. The method first employs an automatic reference selection module based on structural and texture similarity to choose suitable reference mural images for the faded murals. Then, it utilizes a reversible residual network to extract deep features of the mural images without information loss. Next, a channel refinement module is used to eliminate redundant information in the network channels. Finally, an unbiased color transfer module restores the color of the faded mural images. Compared to other image color restoration methods, the proposed method achieves superior color restoration effects while effectively preserving the original structure and texture details of the mural images. Compared to baseline methods, the Structural Similarity Index (SSIM), Feature Similarity Index (FSIM), and Perception-based Image Quality Evaluator (PIQE) values are improved by 7.97%, 3.46%, and 13.98%, respectively. The color restoration of the Dunhuang Mural holds significant historical, artistic, cultural, and economic values, and plays a positive role in the preservation and inheritance of Chinese culture, as well as in the promotion of cultural exchange and mutual understanding.

PST-Diff: Achieving High-Consistency Stain Transfer by Diffusion Models With Pathological and Structural Constraints.

Histopathological examinations heavily rely on hematoxylin and eosin (HE) and immunohistochemistry (IHC) staining. IHC staining can offer more accurate diagnostic details but it brings significant financial and time costs. Furthermore, either re-staining HE-stained slides or using adjacent slides for IHC may compromise the accuracy of pathological diagnosis due to information loss. To address these challenges, we develop PST-Diff, a method for generating virtual IHC images from HE images based on diffusion models, which allows pathologists to simultaneously view multiple staining results from the same tissue slide. To maintain the pathological consistency of the stain transfer, we propose the asymmetric attention mechanism (AAM) and latent transfer (LT) module in PST-Diff. Specifically, the AAM can retain more local pathological information of the source domain images, while ensuring the model's flexibility in generating virtual stained images that highly confirm to the target domain. Subsequently, the LT module transfers the implicit representations across different domains, effectively alleviating the bias introduced by direct connection and further enhancing the pathological consistency of PST-Diff. Furthermore, to maintain the structural consistency of the stain transfer, the conditional frequency guidance (CFG) module is proposed to precisely control image generation and preserve structural details according to the frequency recovery process. To conclude, the pathological and structural consistency constraints provide PST-Diff with effectiveness and superior generalization in generating stable and functionally pathological IHC images with the best evaluation score. In general, PST-Diff offers prospective application in clinical virtual staining and pathological image analysis.

Using query semantic and feature transfer fusion to enhance cardinality estimating of property graph queries

With the increasing complexity and diversity of query tasks, cardinality estimation has become one of the most challenging problems in query optimization. In this study, we propose an efficient and accurate cardinality estimation method to address the cardinality estimation problem in property graph queries, particularly in response to the current research gap regarding the neglect of contextual semantic features. We first propose formal representations of the property graph query and define its cardinality estimation problem. Then, through the query featurization, we transform the query into a vector representation that can be learned by the estimation model, and enrich the feature vector representation by the context semantic information of the query. We finally propose an estimation model for property graph queries, specifically introducing a feature information transfer module to dynamically control the information flow meanwhile achieving the model’s feature fusion and inference. Experimental results on three datasets show that the estimation model can accurately and efficiently estimate the cardinality of property graph queries, the mean Q_error and RMSE are reduced by about 30% and 25% than the state-of-art estimation models. The context semantics features of queries can improve the model’s estimation accuracy, the mean Q_error result is reduced by about 20% and the RMSE result is about 5%.

Spectral Calculations of 3D Radiation Magnetohydrodynamic Simulations of Super-Eddington Accretion onto a Stellar-mass Black Hole

We use the Athena++ Monte Carlo (MC) radiation transfer module to postprocess simulation snapshots from nonrelativistic Athena++ radiation magnetohydrodynamic (RMHD) simulations. These simulations were run using a gray (frequency-integrated) approach but were also restarted and ran with a multigroup approach that accounts for Compton scattering with a Kompaneets operator. These simulations produced moderately super-Eddington accretion rates onto a 6.62 M ⊙ black hole. Since we only achieve inflow equilibrium out to 20–25 gravitational radii, we focus on the hard X-ray emission. We provide a comparison between the MC and RMHD simulations, showing that the treatment of Compton scattering in the gray RMHD simulations underestimates the gas temperature in the funnel regions above and below the accretion disk. In contrast, the restarted multigroup snapshots provide a treatment for the radiation field that is more consistent with the MC calculations, and result in postprocessed spectra with harder X-ray emission compared to their gray snapshot counterparts. We characterize these MC postprocessed spectra using commonly employed phenomenological spectral fitting models. We also attempt to fit our MC spectra directly to observations of the ultraluminous X-ray source (ULX) NGC 1313 X-1, finding best-fit values that are competitive to phenomenological model fits, indicating that first principle models of super-Eddington accretion may adequately explain the observed hard X-ray spectra in some ULX sources.

Flow style-aware network for arbitrary style transfer

Researchers have recently proposed arbitrary style transfer methods based on various model frameworks. Although all of them have achieved good results, they still face the problems of insufficient stylization, artifacts and inadequate retention of content structure. In order to solve these problems, we propose a flow style-aware network (FSANet) for arbitrary style transfer, which combines a VGG network and a flow network. FSANet consists of a flow style transfer module (FSTM), a dynamic regulation attention module (DRAM), and a style feature interaction module (SFIM). The flow style transfer module uses the reversible residue block features of the flow network to create a sample feature containing the target content and style. To adapt the FSTM to VGG networks, we design the dynamic regulation attention module and exploit the sample features both at the channel and pixel levels. The style feature interaction module computes a style tensor that optimizes the fused features. Extensive qualitative and quantitative experiments demonstrate that our proposed FSANet can effectively avoid artifacts and enhance the preservation of content details while migrating style features.

Ancient Painting Inpainting with Regional Attention-Style Transfer and Global Context Perception

Ancient paintings, as a vital component of cultural heritage, encapsulate a profound depth of cultural significance. Over time, they often suffer from different degradation conditions, leading to damage. Existing ancient painting inpainting methods struggle with semantic discontinuities, blurred textures, and details in missing areas. To address these issues, this paper proposes a generative adversarial network (GAN)-based ancient painting inpainting method named RG-GAN. Firstly, to address the inconsistency between the styles of missing and non-missing areas, this paper proposes a Regional Attention-Style Transfer Module (RASTM) to achieve complex style transfer while maintaining the authenticity of the content. Meanwhile, a multi-scale fusion generator (MFG) is proposed to use the multi-scale residual downsampling module to reduce the size of the feature map and effectively extract and integrate the features of different scales. Secondly, a multi-scale fusion mechanism leverages the Multi-scale Cross-layer Perception Module (MCPM) to enhance feature representation of filled areas to solve the semantic incoherence of the missing region of the image. Finally, the Global Context Perception Discriminator (GCPD) is proposed for the deficiencies in capturing detailed information, which enhances the information interaction across dimensions and improves the discriminator’s ability to identify specific spatial areas and extract critical detail information. Experiments on the ancient painting and ancient Huaniao++ datasets demonstrate that our method achieves the highest PSNR values of 34.62 and 23.46 and the lowest LPIPS values of 0.0507 and 0.0938, respectively.

Unified View Empirical Study for Large Pretrained Model on Cross-Domain Few-Shot Learning

The challenge of cross-domain few-shot learning (CD-FSL) stems from the substantial distribution disparities between target and source domain images, necessitating a model with robust generalization capabilities. In this work, we posit that large-scale pretrained models are pivotal in addressing the CD-FSL task owing to their exceptional representational and generalization prowess. To our knowledge, no existing research comprehensively investigates the utility of large-scale pretrained models in the CD-FSL context. Addressing this gap, our study presents an exhaustive empirical assessment of the Contrastive Language–Image Pre-Training model within the CD-FSL task. We undertake a comparison spanning six dimensions: base model, transfer module, classifier, loss, data augmentation, and training schedule. Furthermore, we establish a straightforward baseline model, E-base, based on our empirical analysis, underscoring the importance of our investigation. Experimental results substantiate the efficacy of our model, yielding a mean gain of 1.2% in 5-way 5-shot evaluations on the BSCD dataset.

An Integrated Electric Vehicle Charging System of Wireless Power Transfer and Auxiliary Power Module With Shared Converter and Magnetic Coupler

An Integrated Electric Vehicle Charging System of Wireless Power Transfer and Auxiliary Power Module With Shared Converter and Magnetic Coupler

A LiDAR-driven three-dimensional simulation model for far-red solar-induced chlorophyll fluorescence in forests

Solar-induced chlorophyll fluorescence (SIF) is a subtle but informative probe of plant photosynthesis. Quantifying the three-dimensional (3D) distribution of SIF benefits a better understanding of photosynthesis variations over heterogeneous canopies. Although radiative transfer models (RTMs) provide a solid theoretical basis for simulating the 3D SIF distribution, most RTMs use virtual scenes with complex reconstruction processes. This study aims to develop a 3D SIF simulation model (FluorLiDAR) directly driven by terrestrial light detection and ranging (LiDAR) data using leaf and canopy RTMs, including a 3D PAR (photosynthetically active radiation) simulation model, the Fluspect model, the atmosphere radiative transfer module in SCOPE, and the multiple scattering coefficients of sunlit and shaded leaves from the 4-scale model. The results show that (1) the simulated and measured SIF patterns were consistent, with R2 (RMSE) values of 0.73 (0.17 mW/nm/m2/sr) and 0.76 (0.12 mW/nm/m2/sr) for 1-min sampling and 10-min averages, respectively. Moreover, the R2 between FluorLiDAR and the DART simulation reached 0.94. The R2 of FluorLiDAR and DART were both higher than 1D mSCOPE using every 10-min sampling data. (2) Point density, denoted by average NPD (nearest point distance), influenced the performance of our model mainly when it was smaller than 0.1 m. Chlorophyll content had less influence on the model accuracy (R2 and rRMSE), and the bias between simulation and measurement decreased as chlorophyll content increased. (3) The simulated 3D SIF distribution pattern closely resembled PAR within the canopy. Besides, FluorLiDAR can simulate the hot spot effect like DART and mSCOPE though the effect was not as obvious as the other two models. This study highlights the potential of a LiDAR-driven SIF model for 3D SIF simulation over a heterogeneous canopy, which may benefit the understanding of the structural impacts on forest photosynthesis in real forest scenes.

Cross-modality segmentation of ultrasound image with generative adversarial network and dual normalization network

Elastographic ultrasound (EUS) evaluates lesion stiffness, providing valuable diagnostic information for various diseases. However, accessibility, cost, and visual clarity of tissue structures are limitations of EUS compared to conventional B-mode ultrasound (BUS), which hinders the training of segmentation models based on EUS. To address this challenge, the BUS is utilized as the source domain to generate two distinct new domains, one similar to the style of EUS and the other dissimilar, through the implementation of a style transfer module containing the cycle-consistent generative adversarial network, resulting in effective data augmentation. The dual-normalization module is integrated into the segmentation model to capture both the domain-invariant and domain-specific distribution information from the generated domains, enabling the model to generalize to the target domain, namely the EUS. We conduct the cross-modality segmentation on two EUS datasets of Achilles tendons and lymph nodes. The experiments have shown that our method outperforms other advanced segmentation methods.

Disentanglement-inspired single-source domain-generalization network for cross-scene hyperspectral image classification

Cross-scene classification stands as a pivotal frontier in hyperspectral image (HSI) processing, aiming to enhance the generalization capabilities of classification models. However, the diversity of sensor type, shooting environments, and shooting times leads to the spectral heterogeneity problem in HSI. As a result, the same land cover may exhibit varying spectral traits in different domains, posing challenges for cross-scene HSI classification. Drawing inspiration from image disentanglement, we have identified that extracting the latent domain-invariant representation (DIR) of HSI could potentially mitigate the spectral heterogeneity issue. Therefore, we propose a Disentanglement-Inspired Single-Source Domain Generalization Network (DSDGnet) for cross-scene HSI classification in this paper. Firstly, a style transfer module based on a Transformer encoder-transfer-decoder is designed to expand the single source domain to an extended domain. Then, a progressive disentanglement module is proposed to decompose the domain-invariant features and domain-specific features of HSI. Furthermore, a domain combination module is designed to guarantee the accuracy of the progressive disentanglement module and ensure the effectiveness of the domain-invariant feature of HSI. Finally, the domain-invariant features are applied to the classification task, and the domain-specific features are separated to reduce their impact on the generalization ability of classification models. Extensive experiments on three HSI datasets have demonstrated the advanced classification performance of DSDGnet compared to existing domain-generalization methods.

Thermal propagation analysis of 2G HTS stacked wires based on a dimensional coupling method

Stacked structures of high-temperature superconducting (HTS) tapes are usually employed to enhance current-carrying capacity in cable or magnet applications, but their multilayer characteristics may result in local hot spots due to uneven cooling. Besides, the inconsistencies along the length of tapes introduced during manufacturing process and the significant transverse Lorentz forces experienced in magnet operations can lead to weak parts, bringing uncertainties to thermal stability. Finite-element methods are widely used to predict thermal propagations, but 3D models encounter challenges such as distorted mesh elements and convergence issues, particularly when combining electromagnetic and heat transfer modules for long-distance wires. In this work, we have developed a Dimensional Coupling Method (DCM) to assess the thermal impact of weak parts in stacked wires applying coupled 1D and 2D models. The 2D model analyzes the electromagnetic and heat characteristics of stacked surfaces, and provides an initial heat source for the 1D model, which evaluates thermal propagation longitudinally. Simulation results of the 1D module are then transferred back to update the 2D outcomes. Models of distinct dimensions are coupled sequentially in physical steps but simultaneously in the time domain. Our approach is verified by 3D model benchmarks and offers a computational cost reduction of approximately 60 % compared to the benchmarks, making it more suitable for applications with large Iop/Ic ratios. Specially, multi-layer stacked wires under low ratios cases are also been analyzed. What’s more, two influencing factors of heat propagation, the weak-part length and position, are also investigated.

Embracing Large Natural Data: Enhancing Medical Image Analysis via Cross-Domain Fine-Tuning.

With the rapid advancements of Big Data and computer vision, many large-scale natural visual datasets are proposed, such as ImageNet-21K, LAION-400M, and LAION-2B. These large-scale datasets significantly improve the robustness and accuracy of models in the natural vision domain. However, the field of medical images continues to face limitations due to relatively small-scale datasets. In this article, we propose a novel method to enhance medical image analysis across domains by leveraging pre-trained models on large natural datasets. Specifically, a Cross-Domain Transfer Module (CDTM) is proposed to transfer natural vision domain features to the medical image domain, facilitating efficient fine-tuning of models pre-trained on large datasets. In addition, we design a Staged Fine-Tuning (SFT) strategy in conjunction with CDTM to further improve the model performance. Experimental results demonstrate that our method achieves state-of-the-art performance on multiple medical image datasets through efficient fine-tuning of models pre-trained on large natural datasets.

Spatio-temporal action detector with cross-stream knowledge transfer

Abstract Relying on human labor for uninterrupted monitoring through surveillance systems is highly inefficient and prone to oversight, making intelligent surveillance have great application prospects and value. To implement this technology, spatio-temporal action detectors are typically used, which are neural networks capable of locating action instances simultaneously in both time and space dimensions. To address the issue of high computational demand commonly associated with 3D backbone architectures, we propose a Dynamic Cross-Stream MovingCenter Detector (DCMOC) based on a 2D backbone network, an improved version of the MOC detector. In the DCMOC detector, the movement branch and the bounding box branch correspond to the motion and appearance features of action instances, respectively. Through a designed cross-stream knowledge transfer module, feature knowledge is transferred between the two branches to enhance feature representation. To compare with existing spatio-temporal action detectors, experiments were conducted on two datasets, JHMDB and UCF101-24. The experiments demonstrate that the DCMOC detector achieves competitive performance with significantly lower computational requirements than existing 3D architecture models and achieves optimal performance on the video-map metric at high thresholds.

Arbitrary style transfer via multi-feature correlation

Recent research in arbitrary style transfer has highlighted challenges in maintaining the balance between content structure and style patterns. Moreover, the improper application of style patterns onto the content image often results in suboptimal quality. In this paper, a novel style transfer network, called MCNet, is proposed. It is based on multi-feature correlations. To better explore the intrinsic relationship between the style image and the content image and to transfer the most suitable style onto the content image, a novel Global Style-Attentional Transfer Module, named GSATM, is introduced in this work. GSATM comprises two parts: Forward Adaptive Style Transformation (FAST) and Delayed Style Transformation (DST). The former analyzes the relationship between style and content features and fine-tunes the style features, whereas the latter transfers the content features based on the fine-tuned style features. Moreover, a new encoding and decoding structure is designed to effectively handle the output of GSATM. Extensive quantitative and qualitative experiments fully demonstrate the superiority of our algorithm. Project page: https://github.com/XiangJinCherry/MCNet.

3-Dimensional numerical study on carbon based electrodes for vanadium redox flow battery

Electrode materials are critical in determining the performance of Vanadium redox flow batteries (VRFB). This work aims at elucidating the difference between carbon-based electrodes for VRFB. The electrodes considered for the present study are Graphite felt, Carbon felt, Carbon paper, and Carbon cloth. To study the characteristic behavior of different electrode materials, a 3D unit cell numerical model was developed by coupling electrolyte flow and electrochemical reactions and mass transfer module apart from considering the electrode structural and material information such as porosity, permeability, thickness, conductivity, and specific area. The simulation results of this study suggest that the Graphite felt is the optimum choice as an electrode material for VRFB as it offers higher electrocatalytic activity, lower pressure drop, better performance, and lower cost. The study is also extended to a multilayer configuration of Carbon Paper and Carbon Cloth. The performance of multi-layered electrodes by stacking single sheets of Carbon Paper and Carbon Cloth electrodes are matched to meet the performance obtained with the Graphite felt electrode. It has been shown with the help of polarization curve and the pressure drop study that the performance of five layers of carbon cloth and nine layers of carbon paper match the performance of Graphite felt, approximately. The present study provides practical guidance for the design and optimization of VRFB electrodes.

SGRTmreg: A Learning-Based Optimization Framework for Multiple Pairwise Registrations.

Point cloud registration is a fundamental task in computer vision and graphics, which is widely used in 3D reconstruction, object tracking, and atlas reconstruction. Learning-based optimization and deep learning methods have been widely developed in pairwise registration due to their own distinctive advantages. Deep learning methods offer greater flexibility and enable registering unseen point clouds that are not trained. Learning-based optimization methods exhibit enhanced robustness and stability when handling registration under various perturbations, such as noise, outliers, and occlusions. To leverage the strengths of both approaches to achieve a less time-consuming, robust, and stable registration for multiple instances, we propose a novel computational framework called SGRTmreg for multiple pairwise registrations in this paper. The SGRTmreg framework utilizes three components-a Searching scheme, a learning-based optimization method called Graph-based Reweighted discriminative optimization (GRDO), and a Transfer module to achieve multi-instance point cloud registration.Given a collection of instances to be matched, a template as a target point cloud, and an instance as a source point cloud, the searching scheme selects one point cloud from the collection that closely resembles the source. GRDO then learns a sequence of regressors by aligning the source to the target, while the transfer module stores and applies the learned regressors to align the selected point cloud to the target and estimate the transformation of the selected point cloud. In short, SGRTmreg harnesses a shared sequence of regressors to register multiple point clouds to a target point cloud. We conduct extensive registration experiments on various datasets to evaluate the proposed framework. The experimental results demonstrate that SGRTmreg achieves multiple pairwise registrations with higher accuracy, robustness, and stability than the state-of-the-art deep learning and traditional registration methods.