Representational Capacity Research Articles

Neural Methods for Amortized Inference

Simulation-based methods for statistical inference have evolved dramatically over the past 50 years, keeping pace with technological advancements. The field is undergoing a new revolution as it embraces the representational capacity of neural networks, optimization libraries, and graphics processing units for learning complex mappings between data and inferential targets. The resulting tools are amortized, in the sense that, after an initial setup cost, they allow rapid inference through fast feed-forward operations. In this article we review recent progress in the context of point estimation, approximate Bayesian inference, summary-statistic construction, and likelihood approximation. We also cover software and include a simple illustration to showcase the wide array of tools available for amortized inference and the benefits they offer over Markov chain Monte Carlo methods. The article concludes with an overview of relevant topics and an outlook on future research directions.

SIFusion: Lightweight infrared and visible image fusion based on semantic injection.

The objective of image fusion is to integrate complementary features from source images to better cater to the needs of human and machine vision. However, existing image fusion algorithms predominantly focus on enhancing the visual appeal of the fused image for human perception, often neglecting their impact on subsequent high-level visual tasks, particularly the processing of semantic information. Moreover, these fusion methods that incorporate downstream tasks tend to be overly complex and computationally intensive, which is not conducive to practical applications. To address these issues, a lightweight infrared and visible light image fusion method known as SIFusion, which is based on semantic injection, is proposed in this paper. This method employs a semantic-aware branch to extract semantic feature information, and then integrates these features into the fused features through a Semantic Injection Module (SIM) to meet the semantic requirements of high-level visual tasks. Furthermore, to simplify the complexity of the fusion network, this method introduces an Edge Convolution Module (ECB) based on structural reparameterization technology to enhance the representational capacity of the encoder and decoder. Extensive experimental comparisons demonstrate that the proposed method performs excellently in terms of visual appeal and advanced semantics, providing satisfactory fusion results for subsequent high-level visual tasks even in challenging scenarios.

Frequency-domain sound field from the perspective of band-limited functions.

A model that approximates the sound field well is useful in various fields, such as acoustic signal processing and numerical simulation. We have proposed an effective model in which the wideband instantaneous sound field is regarded as an element of a spherically band-limited function space, using the reproducing kernel of that space. In this paper, the frequency-domain sound field is regarded as an element of some band-limited function space, and a representation of the field as a linear combination of the reproducing kernel in that space is proposed. This model has the strongest representational capacity of all function systems when we know only the sound pressure information at arbitrary positions. The proposed model can be considered a generalization of the existing three-dimensional sound field model using the reproducing kernel of the solution space of the Helmholtz equation to the spatial dimension. One of the advantages of capturing the frequency-domain sound field in this way is the simplicity achieved for the estimation formula of the wavenumber spectrum. Two numerical simulations were conducted to validate the proposed methods.

Predicting future evapotranspiration based on remote sensing and deep learning

Study regionThe watersheds of the four flux sites in the United States were selected as the study areas for this research. Study focusThis study validates the efficiency of Convolutional Long Short-Term Memory Network (ConvLSTM) models for site-scale ETa prediction. We enhanced the ConvLSTM model by adding a Spatial Pyramid Pooling module (SPPM) and a Multi-head Self-Attention Module (MSA-Module), creating the Multi-head Self-Attention ConvLSTM (MSA-ConvLSTM) model, which we applied to predicting regional-scale actual evapotranspiration (ETa). This study aims to investigate whether the MSA-ConvLSTM model can enhance the accuracy of predicting regional-scale ETa, considering multiple feature variables. Furthermore, we evaluated different performance indicators, discussed possible reasons for errors in regional ETa prediction, and conducted sensitivity analysis of the model characteristics. New hydrological insights for the regionThe MSA-ConvLSTM model accurately predicts the future state of ETa. The average R2 was 0.81, which is 11.6 % and 5.5 % higher than those of the ConvLSTM and Self-Attention ConvLSTM (SA-ConvLSTM) models, respectively. The average RMSE is 11.94 mm/m, which is 21.5 % and 13.7 % lower than ConvLSTM and SA-ConvLSTM, respectively. The average MAE is 9.46 mm/m, which is 21.3 % and 13 % lower than ConvLSTM and SA-ConvLSTM, respectively. Incorporating of a multi-head self-attention module enhances the model’s capacity for comprehensive understanding of input data features. This improvement allows the model to better adapt to feature relationships at varying scales and angles, enhancing its representational capacity and enabling effective adaptation to complex environmental changes.

Ponte: Represent Totally Binary Neural Network Toward Efficiency

In the quest for computational efficiency, binary neural networks (BNNs) have emerged as a promising paradigm, offering significant reductions in memory footprint and computational latency. In traditional BNN implementation, the first and last layers are typically full-precision, which causes higher logic usage in field-programmable gate array (FPGA) implementation. To solve these issues, we introduce a novel approach named Ponte (Represent Totally Binary Neural Network Toward Efficiency) that extends the binarization process to the first and last layers of BNNs. We challenge the convention by proposing a fully binary layer replacement that mitigates the computational overhead without compromising accuracy. Our method leverages a unique encoding technique, Ponte::encoding, and a channel duplication strategy, Ponte::dispatch, and Ponte::sharing, to address the non-linearity and capacity constraints posed by binary layers. Surprisingly, all of them are back-propagation-supported, which allows our work to be implemented in the last layer through extensive experimentation on benchmark datasets, including CIFAR-10 and ImageNet. We demonstrate that Ponte not only preserves the integrity of input data but also enhances the representational capacity of BNNs. The proposed architecture achieves comparable, if not superior, performance metrics while significantly reducing the computational demands, thereby marking a step forward in the practical deployment of BNNs in resource-constrained environments.

Memorizing Swin-Transformer Denoising Network for Diffusion Model

Diffusion models have garnered significant attention in the field of image generation. However, existing denoising architectures, such as U-Net, face limitations in capturing the global context, while Vision Transformers (ViTs) may struggle with local receptive fields. To address these challenges, we propose a novel Swin-Transformer-based denoising network architecture that leverages the strengths of both U-Net and ViT. Moreover, our approach integrates the k-Nearest Neighbor (kNN) based memorizing attention module into the Swin-Transformer, enabling it to effectively harness crucial contextual information from feature maps and enhance its representational capacity. Finally, we introduce an innovative hierarchical time stream embedding scheme that optimizes the incorporation of temporal cues during the denoising process. This method surpasses basic approaches like simple addition or concatenation of fixed time embeddings, facilitating a more effective fusion of temporal information. Extensive experiments conducted on four benchmark datasets demonstrate the superior performance of our proposed model compared to U-Net and ViT as denoising networks. Our model outperforms baselines on the CRC-VAL-HE-7K and CelebA datasets, achieving improved FID scores of 14.39 and 4.96, respectively, and even surpassing DiT and UViT under our experiment setting. The Memorizing Swin-Transformer architecture, coupled with the hierarchical time stream embedding, sets a new state-of-the-art in denoising diffusion models for image generation.

DenseFusion-DA2: End-to-End Pose-Estimation Network Based on RGB-D Sensors and Multi-Channel Attention Mechanisms.

Notably, 6D pose estimation is a critical technology that enables robotics to perceive and interact with their operational environment. However, occlusion causes a loss of local features, which, in turn, restricts the estimation accuracy. To address these challenges, this paper proposes an end-to-end pose-estimation network based on a multi-channel attention mechanism, DA2Net. Firstly, a multi-channel attention mechanism, designated as "DA2Net", was devised using A2-Nets as its foundation. This mechanism is constructed in two steps. In the first step, the essential characteristics are extracted from the global feature space through the second-order attention pool. In the second step, a feature map is generated by the integration of position and channel attention. Subsequently, the extracted key features are assigned to each position of the feature map, enhancing both the feature representation capacity and the overall performance. Secondly, the designed attention mechanism is introduced into both the feature fusion and pose iterative refinement networks to enhance the network's capacity to acquire local features thus improving its overall performance. The experimental results demonstrated that the estimation accuracy of DenseFusion-DA2 on the LineMOD dataset was approximately 3.4% higher than that of DenseFusion. Furthermore, the estimation accuracy surpassed that of PoseCNN, PVNet, SSD6D, and PointFusion by 8.3%, 11.1%, 20.3%, and 23.8%, respectively. The estimation accuracy also shows a significant advantage on the Occluded LineMOD and HR-Vision datasets. This research not only presents a more efficient solution for robot perception but also introduces novel ideas and methods for technological advancements and applications in related fields.

FCSSL: fusion enhanced contrastive self-supervised learning method for parallel MRI reconstruction

Objective. The implementation of deep learning in magnetic resonance imaging (MRI) has significantly advanced the reduction of data acquisition times. However, these techniques face substantial limitations in scenarios where acquiring fully sampled datasets is unfeasible or costly.Approach. To tackle this problem, we propose a fusion enhanced contrastive self-supervised learning (FCSSL) method for parallel MRI reconstruction, eliminating the need for fully sampledk-space training dataset and coil sensitivity maps. First, we introduce a strategy based on two pairs of re-undersampling masks within a contrastive learning framework, aimed at enhancing the representational capacity to achieve higher quality reconstruction. Subsequently, a novel adaptive fusion network, trained in a self-supervised learning manner, is designed to integrate the reconstruction results of the framework.Results. Experimental results on knee datasets under different sampling masks demonstrate that the proposed FCSSL achieves superior reconstruction performance compared to other self-supervised learning methods. Moreover,the performance of FCSSL approaches that of the supervised methods, especially under the 2DRU and RADU masks, but no need for fully sample data. The proposed FCSSL, trained under the 3× 1DRU and 2DRU masks, can effectively generalize to unseen 1D and 2D undersampling masks, respectively. For target domain data that exhibit significant differences from source domain data, the proposed model, fine-tuned with just a few dozen instances of undersampled data in the target domain, achieves reconstruction performance comparable to that achieved by the model trained with the entire set of undersampled data.Significance. The novel FCSSL model offers a viable solution for reconstructing high-quality MR images without needing fully sampled datasets, thereby overcoming a major hurdle in scenarios where acquiring fully sampled MR data is difficult.

Give me a hint: an explicit prior based image denoising

The performance of an image denoising method highly depends on its ability to produce smoothed homogeneous regions and preserve fine-grained texture details in the restored image. Existing denoising methods rely on implicit learning of Convolutional Neural Networks (CNNs) to restore an image. We show that the implicitly learned representations are limited in capacity, resulting in a sub-optimal pixel estimation of homogeneous regions and loss in texture details. In this work, we argue that explicitly introducing low and high frequency information enhances the representational capacity of denoising CNNs. To this end, we propose XPrior that provides the network with two explicit priors, i.e., a Gaussian smoothed image and an Edge map. This simple yet effective integration of priors with the noisy image enhances the model’s ability to restore images with high perceived quality. The proposed approach can easily be plugged into the existing denoising methods. Comprehensive experimental results show the superiority of the proposed approach that achieves a remarkable performance gain of 1.23 dB and 1.49 dB average PSNR on BSD68 dataset over DRUNet (Zhang in IEEE Trans Pattern Anal Mach Intell 10.1109/TPAMI.2021.3088914, 2021) and DnCNN (Zhang in IEEE Trans Image Process 26:3142–3155, 2017), respectively.

Multi-Source Information-Based Bearing Fault Diagnosis Using Multi-Branch Selective Fusion Deep Residual Network.

Rolling bearing is the core component of industrial machines, but it is difficult for common single signal source-based fault diagnosis methods to ensure reliable results since sensor signals are vulnerable to the pollution of background noises and the attenuation of transmitted information. Recently, multi-source information-based fault diagnosis methods have become popular, but the information redundancy between multiple signals is a tough problem that will negatively impact the representational capacity of deep learning algorithms and the precision of fault diagnosis methods. Besides that, the characteristics of various signals are actually different, but this problem was usually omitted by researchers, and it has potential to further improve the diagnosing performance by adaptively adjusting the feature extraction process for every input signal source. Aimed at solving the above problems, a novel model for bearing fault diagnosis called multi-branch selective fusion deep residual network is proposed in this paper. The model adopts a multi-branch structure design to enable every input signal source to have a unique feature processing channel, avoiding the information of multiple signal sources blindly coupled by convolution kernels. And in each branch, different convolution kernel sizes are assigned according to the characteristics of every input signal, fully digging the precious fault components on respective information sources. Lastly, the dropout technique is used to randomly throw out some activated neurons, alleviating the redundancy and enhancing the quality of the multiscale features extracted from different signals. The proposed method was experimentally compared with other intelligent methods on two authoritative public bearing datasets, and the experimental results prove the feasibility and superiority of the proposed model.

Remote sensing image classification based on non-linear enhanced attention mechanism

Abstract In recent years, Transformer technology has gradually shown promising applications in the field of computer vision, becoming a research hotspot. However, traditional vision Transformers suffer from significant computational burdens. Although previous studies have attempted to alleviate this issue by reducing the computational load of attention mechanisms, their methods still require improvement. Additionally, past research often overlooked the non-linear representation of values within attention mechanisms, posing another challenge to address. Therefore, this paper proposes a novel attention mechanism aimed at reducing the computational burden of traditional attention mechanisms while enhancing their ability to express non-linearities in values. Furthermore, conventional feedforward neural networks typically output results directly after passing through an activation function, which may limit the model’s representational capacity due to the simplistic structure leading to singular feature information. To address this issue, this study introduces a more robust feature fusion convolutional feedforward network. Through these methods, the aim is to enhance the accuracy of Transformers in the field of remote-sensing image classification.

KnoMol: A Knowledge-Enhanced Graph Transformer for Molecular Property Prediction.

Molecular property prediction (MPP) techniques are pivotal in reducing drug development costs by preemptively predicting bioactivity and ADMET properties. Despite the application of numerous deep learning approaches, enhancing the representational capacity of these models remains a significant challenge. This paper presents a novel knowledge-based Transformer framework, KnoMol, designed to improve the understanding of molecular structures. KnoMol integrates expert chemical knowledge into the Transformer, emulating the analytical methods of medicinal chemists. Additionally, the multiperspective attention mechanism provides a more precise way to represent ring systems. In the evaluation experiments, KnoMol achieved state-of-the-art performance on both MoleculeNet and small-scale data sets, surpassing existing models in terms of accuracy and generalization. Further research indicated that the incorporation of knowledge significantly reduces KnoMol's reliance on data volumes, offering a solution to the challenge of data scarcity. Moreover, KnoMol identified several new inhibitors of HER2 in a case study, demonstrating its value in real-world applications. Overall, this research not only provides a powerful tool for MPP but also serves as a successful precedent for embedding knowledge into Transformers, with positive implications for computer-aided drug discovery and the development of MPP algorithms.

Neural Architecture Search as Program Transformation Exploration

Improving the performance of deep neural networks (DNNs) is important to both the compiler and neural architecture search (NAS) communities. Compilers apply program transformations in order to exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs. In this work, we express such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing transformations into a unified optimization framework. This unification allows us to express existing NAS operations as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and were able to significantly reduce inference time and NAS search time.

Exploiting K-Space in Magnetic Resonance Imaging Diagnosis: Dual-Path Attention Fusion for K-Space Global and Image Local Features.

Magnetic resonance imaging (MRI) diagnosis, enhanced by deep learning methods, plays a crucial role in medical image processing, facilitating precise clinical diagnosis and optimal treatment planning. Current methodologies predominantly focus on feature extraction from the image domain, which often results in the loss of global features during down-sampling processes. However, the unique global representational capacity of MRI K-space is often overlooked. In this paper, we present a novel MRI K-space-based global feature extraction and dual-path attention fusion network. Our proposed method extracts global features from MRI K-space data and fuses them with local features from the image domain using a dual-path attention mechanism, thereby achieving accurate MRI segmentation for diagnosis. Specifically, our method consists of four main components: an image-domain feature extraction module, a K-space domain feature extraction module, a dual-path attention feature fusion module, and a decoder. We conducted ablation studies and comprehensive comparisons on the Brain Tumor Segmentation (BraTS) MRI dataset to validate the effectiveness of each module. The results demonstrate that our method exhibits superior performance in segmentation diagnostics, outperforming state-of-the-art methods with improvements of up to 63.82% in the HD95 distance evaluation metric. Furthermore, we performed generalization testing and complexity analysis on the Automated Cardiac Diagnosis Challenge (ACDC) MRI cardiac segmentation dataset. The findings indicate robust performance across different datasets, highlighting strong generalizability and favorable algorithmic complexity. Collectively, these results suggest that our proposed method holds significant potential for practical clinical applications.

SmartFixture: Physics-guided reinforcement learning for automatic fixture layout design in manufacturing systems

Fixture layout design critically impacts the shape deformation of large-scale sheet parts and the quality of the final product in the assembly process. The existing works focus on developing Mathematical-Optimization (MO)-based methods to generate the optimal fixture layout via interacting with Finite Element Analysis (FEA)-based simulations or its surrogate models. Their limitations can be summarized as memorylessness and lack of scalability. Memorylessness indicates that the experience in designing the fixture layout for one part is usually not transferable to others. Scalability becomes an issue for MO-based methods when the design space of fixtures is large. Furthermore, the surrogate models might have limited representation capacity when modeling high-fidelity simulations. To address these limitations, we propose a learning-based framework, SmartFixture, to design the fixture layout by training a Reinforcement learning agent through direct interaction with the FEA-based simulations. The advantages of the proposed framework include: (i) it is generalizable to design fixture layouts for unseen scenarios after offline training; (ii) it is capable of finding the optimal fixture layout over a massive search space. Experiments demonstrate that the proposed framework consistently generates the best fixture layouts that receive the smallest shape deformations on the sheet parts with different initial shape variations.

SiGNN: A spike-induced graph neural network for dynamic graph representation learning

In the domain of dynamic graph representation learning (DGRL), capturing the temporal evolution within real-world networks is of paramount importance. Spiking Neural Networks (SNNs), renowned for their temporal dynamics and low-power characteristics, provide an efficient solution for temporal processing in DGRL tasks. However, due to the spike-based information encoding mechanism of SNNs, existing DGRL methods that employ SNNs face significant limitations in their representational capacity. To address this issue, we propose an innovative framework named Spike-induced Graph Neural Network (SiGNN) for learning enhanced spatiotemporal representations on dynamic graphs. Specifically, we achieve a harmonious integration of SNNs and GNNs through a novel Temporal Activation (TA) mechanism. This mechanism enables SiGNN to effectively harness the temporal dynamics of SNNs while circumventing the representational constraints imposed by the binary nature of spikes. Furthermore, leveraging the inherent adaptability of SNNs, SiGNN incorporates an in-depth analysis of evolutionary patterns within graphs across multiple time granularities, thereby facilitating the acquisition of multiscale temporal node representations. Extensive experiments on various real-world dynamic graph datasets demonstrate the superior performance of SiGNN in the node classification task. Our code is publicly available at https://github.com/CharliedoD/SiGNN.

Memory-Based Learning and Fusion Attention for Few-Shot Food Image Generation Method

Generating food images aims to convert textual food ingredients into corresponding images for the visualization of color and shape adjustments, dietary guidance, and the creation of new dishes. It has a wide range of applications, including food recommendation, recipe development, and health management. However, existing food image generation models, predominantly based on GANs (Generative Adversarial Networks), face challenges in maintaining semantic consistency between image and text, as well as achieving visual realism in the generated images. These limitations are attributed to the constrained representational capacity of sparse ingredient embedding and the lack of diversity in GAN-based food image generation models. To alleviate this problem, this paper proposes a food image generation network, named MLA-Diff, in which ingredient and image features are learned and integrated as ingredient-image pairs to generate initial images, and then image details are refined by using an attention fusion module. The main contributions are as follows: (1) The enhanced CLIP (Contrastive Language-Image Pre-Training) module is constructed by transforming sparse ingredient embedding into compact embedding and capturing multi-scale image features, providing an effective solution to alleviate semantic consistency issues. (2) The Memory module is proposed by embedding a pre-trained diffusion model to generate initial images with diversity and reality. (3) The attention fusion module is proposed by integrating features from diverse modalities to enhance the comprehension between ingredient and image features. Extensive experiments on the Mini-food dataset demonstrate the superiority of the MLA-Diff in terms of semantic consistency and visual realism, generating high-quality food images.

Hyperspectral anomaly detection based on weighted low-rank sparse dictionary learning

The geometric model-based hyperspectral anomaly detection techniques have garnered a lot of interest recently. These methods are predicated on the assumption that the background can be represented by a dictionary of spectral vectors, but the anomaly cannot. Building an objective model with high representational capabilities and obtaining precise background modeling are therefore key aspects of this type of method. The background dictionary is also readily tainted by anomalies due to the limitations of the current approaches, making the detection findings less reliable. To address the above problems, this paper proposes a weighted low-rank sparse dictionary learning method (WLSDL). This model organically combines sparse representation with low-rank representation in order to effectively depict the intricate background distribution. The weights relating to the eigenvalues of the image are designed by mining the physical characteristics of the HSI. This can enhance the nuclear norm’s capacity for representation, resulting in a background dictionary with more representativeness. Furthermore, the capped norm constraint is incorporated in the objective function to decrease the effect of anomalies and noises on background modeling, hence minimizing anomaly pollution on the background dictionary. The residual image effectively increases the accuracy of anomaly identification since it makes it easier to distinguish between anomalies and background. We use three hyperspectral datasets to evaluate the proposed method. The experimental findings demonstrate the effectiveness and superiority of the proposed method over state-of-the-art methods.

From ensemble learning to deep ensemble learning: A case study on multi-indicator prediction of pavement performance

Recently, Big data analytics approaches were combined with emerging machine learning techniques, which can provide more sophisticated insights into information-intensive activity. Compared to traditional shallow-architecture machine learning algorithms, Deep learning could excavate more potential information from the raw features. However, its powerful representational capacity relies on the support of enormous samples. The ensemble trees system performs superior on small sample problems due to better generalization capacity. To merge both benefits of deep learning and ensemble tree system, this paper developed a deep ensemble algorithm applied to multiple indicators prediction of pavement performance, including International Roughness Index and pavement 3-layer modulus. The deep ensemble algorithm is developed by merging a deep neural network (DNN) with the decision manifold property of the decision trees (TabNet) into a cascade ensemble system, combined with a sliding window algorithm to extract dependency information from raw data. During the training stage, the Bayesian Optimization Algorithm (BOA) is used to search for the optimal combination of sub-decision makers in the cascade ensemble. And equipped with GPU, it can speed up by 2.6–4.0 times. In the case study of pavement engineering, with sufficient training samples, it can achieve an average accuracy of 98.74 %, higher than DNN (97.49 %) and XGBoost (96.12 %) in predicting pavement indicators. With insufficient training samples, it can achieve an accuracy improvement of 12 % than XGBoost (75 %) and 24.5 % than DNN (62.5 %).

Squeeze & Excitation joint with Combined Channel and Spatial Attention for Pathology Image Super-Resolution

Super-resolution (SR) techniques are pivotal in enhancing low-resolution images and crucial in medical diagnosis, where detail and clarity are paramount. Traditional pixel-loss-based SR methods, while adept at producing high-resolution (HR) images, often result in artifice content. This loss of information compromises both the visual experience and the accuracy of subsequent diagnoses. Addressing this, we have developed an innovative SR approach integrating a joint Squeeze and Excitation (SE) mechanism with a Combined Channel and Spatial Attention (CCSA) mechanism. The SE mechanism effectively recalibrates channel-wise feature responses, enhancing the representational capacity of the network. Meanwhile, the CCSA mechanism focuses on extracting spatial and channel-wise features, ensuring that critical high-frequency details are preserved. The dual approach significantly refines the quality of the images, maintaining essential details necessary for accurate medical diagnosis. To validate our proposed approach, we used a benchmark dataset, bcSR, tailored to challenge SR models to focus on broader and more critical regions. Comparative analysis proves that our model excels in performance over existing state-of-the-art methods. In conclusion, our proposed SR Network, with its innovative SE and CCSA mechanisms, offers a potent tool for pathology image SR. It elevates the quality of super-resolved images, which will significantly aid in the accuracy and efficiency of medical diagnoses, providing a valuable asset to medical professionals.