Message Passing Research Articles

Adaptive Neural Message Passing for Inductive Learning on Hypergraphs.

Graphs are the most ubiquitous data structures for representing relational datasets and performing inferences in them. They model, however, only pairwise relations between nodes and are not designed for encoding the higher-order relations. This drawback is mitigated by hypergraphs, in which an edge can connect an arbitrary number of nodes. Most hypergraph learning approaches convert the hypergraph structure to that of a graph and then deploy existing geometric deep learning methods. This transformation leads to information loss, and sub-optimal exploitation of the hypergraph's expressive power. We present HyperMSG, a novel hypergraph learning framework that uses a modular two-level neural message passing strategy to accurately and efficiently propagate information within each hyperedge and across the hyperedges. HyperMSG adapts to the data and task by learning an attention weight associated with each node's degree centrality. Such a mechanism quantifies both local and global importance of a node, capturing the structural properties of a hypergraph. HyperMSG is inductive, allowing inference on previously unseen nodes. Further, it is robust and outperforms state-of-the-art hypergraph learning methods on a wide range of tasks and datasets. Finally, we demonstrate the effectiveness of HyperMSG in learning multimodal relations through detailed experimentation on a challenging multimedia dataset.

Adaptive Feature Learning for Unbiased Scene Graph Generation.

Scene Graph Generation (SGG) aims to detect all objects and identify their pairwise relationships in the scene. Recently, tremendous progress has been made in exploring better context relationship representations. Previous work mainly focuses on contextual information aggregation and uses de-biasing strategies on samples to eliminate the preference for head predicates. However, there remain challenges caused by indeterminate feature training. Overlooking the label confusion problem in feature training easily results in a messy feature distribution among the confused categories, thereby affecting the prediction of predicates. To alleviate the aforementioned problem, in this paper, we focus on enhancing predicate representation learning. Firstly, we propose a novel Adaptive Message Passing (AMP) network to dynamically conduct information propagation among neighbors. AMP provides discriminating representations for neighbor nodes under the view of de-noising and adaptive aggregation. Furthermore, we construct a feature-assisted training paradigm alongside the predicate classification branch, guiding predicate feature learning to the corresponding feature space. Moreover, to alleviate biased prediction caused by the long-tailed class distribution and the interference of confused labels, we design a Bi-level Curriculum learning scheme (BiC). The BiC separately considers the training from the feature learning and de-biasing levels, preserving discriminating representations of different predicates while resisting biased predictions. Results on multiple SGG datasets show that our proposed method AMP-BiC has superior comprehensive performance, demonstrating its effectiveness.

High precision deep-learning model combined with high-throughput screening to discover fused [5,5] biheterocyclic energetic materials with excellent comprehensive properties.

Finding novel energetic materials with good comprehensive performance has always been challenging because of the low efficiency in conventional trial and error experimental procedure. In this paper, we established a deep learning model with high prediction accuracy using embedded features in Directed Message Passing Neural Networks. The model combined with high-throughput screening was shown to facilitate rapid discovery of fused [5,5] biheterocyclic energetic materials with high energy and excellent thermal stability. Density Functional Theory (DFT) calculations proved that the performances of the targeting molecules are consistent with the predicted results from the deep learning model. Furthermore, 6,7-trinitro-3H-pyrrolo[1,2-b][1,2,4]triazo-5-amine with both good detonation properties and thermal stability was screened out, whose crystal structure and intermolecular interactions were also analyzed.

A scalable parallel computing method for autonomous platoons

This paper developed a scalable parallel computing method that can be used for platoon simulations and controller validations. A scalable adaptive platooning control law was firstly designed, which accommodates a variety of vehicle-to-vehicle communication topologies. A road vehicle dynamics model that considered the Magic Formula tyre model and suspension dynamics was then derived and validated. The parallel computing method adopted the Message Passing Interface technique to allow fast and scalable simulations. Platoon length changes do not require controller and algorithm changes. An 11-vehicle platoon on a real-world 10 km long road section was simulated. Different localisation sensor errors, communication delays, heterogenous vehicle masses and driving modes were considered. Results show that localisation errors have negligible influences on space errors. Aggressive driving and heterogeneous vehicle masses slightly increase space errors (increases less than 0.23 m). Communication delays are the greatest influencer for space errors. Increases for 15, 45 and 75 ms delays were 0.43, 1.41 and 2.41 m, respectively. It is further shown that parallel computing can improve the computing speed by three times on personal computers and seven to 12 times on workstations.

Approximate Message Passing for sparse matrices with application to the equilibria of large ecological Lotka–Volterra systems

This paper is divided into two parts. The first part is devoted to the study of a class of Approximate Message Passing (AMP) algorithms which are widely used in the fields of statistical physics, machine learning, or communication theory. The AMP algorithms studied in this part are those where the measurement matrix has independent elements, up to the symmetry constraint when this matrix is symmetric, with a variance profile that can be sparse. The AMP problem is solved by adapting the approach of Bayati, Lelarge, and Montanari (2015) to this matrix model.The Lotka–Volterra (LV) model is the standard model for studying the dynamical behavior of large dimensional ecological food chains. The second part of this paper is focused on the study of the statistical distribution of the globally stable equilibrium vector of a LV system in the situation where the random symmetric interaction matrix among the living species is sparse, and in the regime of large dimensions. This equilibrium vector is the solution of a Linear Complementarity Problem, which distribution is shown to be characterized through the AMP approach developed in the first part. In the large dimensional regime, this distribution is close to a mixture of a large number of truncated Gaussians.

Vector Approximate Message Passing Based OFDM-IM Detection for Underwater Acoustic Communications.

Orthogonal frequency division multiplexing with index modulation (OFDM-IM) has great potential for the implementation of high spectral-efficiency underwater acoustic (UWA) communications. However, general receivers consisting of the optimal maximum likelihood detection suffer from high computational load, which prohibits real-time data transmissions in underwater scenarios. In this paper, we propose a detection based on a vector approximate message passing (VAMP) algorithm for UWA OFDM-IM communications. Firstly, a VAMP framework with a non-loopy factor graph for index detection is formulated. Secondly, by utilizing the sparsity inherently existing in OFDM-IM symbols, a novel shrinkage function is derived based on the minimum mean square error criterion, which guarantees better posterior estimation. To reduce the errors from estimated non-existing indices, one trick is utilized to search the elements from the look-up table with the minimal Euclidean distance for the replacement of erroneously estimated indices. Experiments verify the advantages of the proposed detector in terms of low complexity, robustness and effectiveness compared with the state-of-art benchmarks.

Integrated Sensing and Communication via Orthogonal Time Frequency Space Signaling with Hybrid Message Passing Detection and Fractional Parameter Estimation.

For the orthogonal time frequency space (OTFS) modulation, we generally multiplex symbols on a new type of carrier waveform in the delay-Doppler (DD) domain. These two parameters can be used to infer the range (R) and velocity (V) of the communication user and sensing target; thus, it is natural for the OTFS to be implemented in integrated sensing and communication (ISAC). A framework for ISAC based on OTFS modulation is proposed in this paper, in which the matched filter scheme with fractional parameter estimation is implemented for radar sensing. In addition, the hybrid message passing (MP) detection algorithm is developed for OTFS symbol demodulation. According to the simulation results, fractional DD shifts associated with multiple targets can be accurately obtained through the proposed framework. Meanwhile, the bit error rate under the proposed detector is less than 10-4 when the signal-to-noise ratio is high enough.

Initial development of web application to support excipient selection for immediate release tablets using message passing neural network

The formulation of immediate release tablets is a challenging task due to the need to balance multiple factors such as bioavailability and stability. The selection of appropriate excipients is critical in achieving these objectives. The recent successful application of Artificial Intelligence (AI) and Message Passing Neural Network (MPNN) in predicting physicochemical and biological properties of drug molecules suggests for the researchers on the ability to apply these models in selecting excipients. The aim of this study is to develop an innovative approach for selecting excipients using AI and MPNN and to create a user-friendly web application to support excipient selection for immediate release tablets. The study utilized a database of 13,278 immediate-release tablets to train, validate, and test the MPNN model on the basis of Simplified Molecular-Input Line-Entry system (SMILES) of drug substances. The performance of the model was validated based on its ability to predict the probability of selecting an excipient reasonably. A web application named FormAI was developed using the Streamlit web framework and integrated with the trained model. The MPNN model demonstrated good performance, with an average Area Under Curve > 0.98 and R2 > 0.99, indicating its ability to predict the probability of selecting an excipient reasonably. The FormAI application provides a user-friendly platform for excipient selection. The results of the study demonstrate the potential of using AI and MPNN in drug formulation design, specifically in excipient selection for immediate-release tablets. The FormAI application provides a practical solution for pharmaceutical scientists and formulators.

Improving rare relation inferring for scene graph generation using bipartite graph network

Scene graph generation (SGG) task is plagued by insufficient and long-tailed training samples. Thus external text knowledge has been introduced into SGG task, which augments the training dataset. However, rare relations or low-frequency events, such as <dog, standing on, surfboard>, are still hard to be discovered since they also rarely appear in external texts. Addressed these issues, this paper proposes a model-agnostic Bipartite Graph Network with Dual-Group Message Passing (DG-BGN). It extends the relation space of each object pair in training dataset based on the similarity of different object pairs, which cooperates external text knowledge and internal visual information. Specifically, the main framework of DG-BGN consists of three parts, that is, bipartite graph construction (BGC), dual-group message propagation (DG-MP), and pseudo label generation (PLG). For BGC, alignment between external texts and visual dataset is performed to mine information from multimodal resources. Taking object pair as the index of the grouping, dual bipartite graph containing intra/inter-group graph is established to represent predicate label probability distribution of object pair instances. Among them, inter-group bipartite graph is built based on the similarity between object pair instance. For DG-MP, efficient message passing is running on both kinds of bipartite graph through Graph Convolution Neural Networks to refine the graphs, in order to let each object pair instance learn from its similar instances. Finally in PLG, pseudo labels, which are used to train SGG models in fully supervised way, are obtained by aggregating these refined graphs. Systematic experiments on Visual Genome dataset and Conceptual Captions show that our method performs better on discovering rare relations.

Stochastic Gradient Descent for matrix completion: Hybrid parallelization on shared- and distributed-memory systems

The purpose of this study is to investigate the hybrid parallelization of the Stochastic Gradient Descent (SGD) algorithm for solving the matrix completion problem on a high-performance computing platform. We propose a hybrid parallel decentralized SGD framework with asynchronous inter-process communication and a novel flexible partitioning scheme to attain scalability up to hundreds of processors. We utilize Message Passing Interface (MPI) for inter-node communication and POSIX threads for intra-node parallelism. We tested our method by using different real-world benchmark datasets. Experimental results on a hybrid parallel architecture showed that, compared to the state-of-the-art, the proposed algorithm achieves 6× higher throughput on sparse datasets, while it achieves comparable throughput on relatively dense datasets.

Accuracy Assessment of Atomistic Neural Network Potentials: The Impact of Cutoff Radius and Message Passing.

Atomistic neural network potentials have achieved great success in accelerating atomistic simulations in complicated systems in recent years. They are typically based on the atomic decomposition of total properties, truncating the interatomic correlations to a local environment within a given cutoff radius. A more recently developed message passing (MP) neural network framework can, in principle, incorporate nonlocal effects through iteratively correlating some atoms outside the cutoff sphere with atoms inside, a process referred to as MP. However, how the model accuracy depends on the cutoff radius and the MP process has rarely been discussed. In this work, we investigate this dependence using a recursively embedded atom neural network method that possesses both local and MP features, in two representative systems: liquid H2O and solid Al2O3. We focus on how these settings influence predictions for structural and vibrational properties, namely, radial distribution functions (RDFs) and vibrational density of states (VDOSs). We find that while MP lowers test errors of energy and forces in general, it may not improve the prediction for RDFs and/or VDOSs if direct interatomic correlations in the local environment are insufficiently described. A cutoff radius exceeding the first neighbor shell is necessary, beyond which involving MP quickly enhances the model accuracy until convergence. This is a potentially more efficient way to increase the model accuracy than directly increasing the cutoff radius, especially with more memory savings in the GPU implementation. Our findings also suggest that using the mean test error as the measure of the model accuracy alone is inadequate.

Gaussian approximation potentials: Theory, software implementation and application examples.

Gaussian Approximation Potentials (GAPs) are a class of Machine Learned Interatomic Potentials routinely used to model materials and molecular systems on the atomic scale. The software implementation provides the means for both fitting models using abinitio data and using the resulting potentials in atomic simulations. Details of the GAP theory, algorithms and software are presented, together with detailed usage examples to help new and existing users. We review some recent developments to the GAP framework, including Message Passing Interface parallelisation of the fitting code enabling its use on thousands of central processing unit cores and compression of descriptors to eliminate the poor scaling with the number of different chemical elements.

PRINCE: A Pruned AMP Integrated Deep CNN Method for Efficient Channel Estimation of Millimeter-Wave and Terahertz Ultra-Massive MIMO Systems

Millimeter-wave (mmWave) and Terahertz (THz)-band communications exploit the abundant bandwidth to fulfill the increasing data rate demands of 6G wireless communications. To compensate for the high propagation loss with reduced hardware costs, ultra-massive multiple-input multiple-output (UM-MIMO) with a hybrid beamforming structure is a promising technology in the mmWave and THz bands. However, channel estimation (CE) is challenging for hybrid UM-MIMO systems, which requires recovering the high-dimensional channels from severely few channel observations. In this paper, a Pruned Approximate Message Passing (AMP) Integrated Deep Convolutional-neural-network (DCNN) CE (PRINCE) method is firstly proposed, which enhances the estimation accuracy of the AMP method by appending a DCNN network. Moreover, by truncating the insignificant feature maps in the convolutional layers of the DCNN network, a pruning method including training with regularization, pruning and refining procedures is developed to reduce the network scale. Simulation results show that the PRINCE achieves a good trade-off between the CE accuracy and significantly low complexity, with normalized-mean-square-error (NMSE) of -10 dB at signal-to-noise-ratio (SNR) as 10 dB after eliminating 80% feature maps.

Finding Bottlenecks in Message Passing Interface Programs by Scalable Critical Path Analysis

Bottlenecks and imbalance in parallel programs can significantly affect performance of parallel execution. Finding these bottlenecks is a key issue in performance analysis of MPI programs especially on a large scale. One of the ways to discover bottlenecks is to analyze the critical path of the parallel program: the longest execution path in the program activity graph. There are a number of methods of finding the critical path; however, most of them suffer a performance drop when scaled. In this paper, we analyze several methods of critical path finding based on classical Dijkstra and Delta-stepping algorithms along with the proposed algorithm based on topological sorting. Corresponding algorithms for each approach are presented including additional enhancements for increasing performance. The implementation of the algorithms and resulting performance for several benchmark applications (NAS Parallel Benchmarks, CP2K, OpenFOAM, LAMMPS, and MiniFE) are analyzed and discussed.

An efficient watermarking algorithm for digital audio data in security applications

Transform-domain audio watermarking systems are more robust than time-domain systems. However, the main weakness of these systems is their high computational cost, especially for long-duration audio signals. Therefore, they are not desirable for real-time security applications where speed is a critical factor. In this paper, we propose a fast watermarking system for audio signals operating in the hybrid transform domain formed by the fractional Charlier transform (FrCT) and the dual-tree complex wavelet transform (DTCWT). The central idea of the proposed algorithm is to parallelize the intensive and repetitive steps in the audio watermarking system and then implement them simultaneously on the available physical cores on an embedded systems cluster. In order to have a low power consumption and a low-cost cluster with a large number of physical cores, four Raspberry Pis 4B are used where the communication between them is ensured using the Message Passing Interface (MPI). The adopted Raspberry Pi cluster is also characterized by its portability and mobility, which are required in watermarking-based smart city applications. In addition to its resistance to any possible manipulation (intentional or unintentional), high payload capacity, and high imperceptibility, the proposed parallel system presents a temporal improvement of about 70%, 80%, and 90% using 4, 8, and 16 physical cores of the adopted cluster, respectively.

A massive MPI parallel framework of smoothed particle hydrodynamics with optimized memory management for extreme mechanics problems

The dynamic failure process of structures under extreme loadings is very common in many fields of engineering and science. The smoothed particle hydrodynamics (SPH) method offers inherent benefits in dealing with complex interfaces and large material deformations in extreme mechanics problems. However, SPH simulations for 3D engineering applications are time-consuming. To address this issue, we introduce MPI (Message Passing Interface) in our SPH scheme to reduce computational time. Some optimizations are adopted to ensure the massive computation of the SPH method. In particular, an optimized memory management strategy is developed to control the memory footprint. With the present MPI-based massive parallelization of the SPH method, several validation examples are tested and analyzed. By comparing the present numerical results with the reference data, the dynamic failure process of complex structures subjected to extreme loadings like explosive and impact loadings can be well captured. A large number of particles, up to 2.04 billion, are adopted in the present simulations. The scaling tests show that the scalability of the massively parallel SPH program achieves a maximum parallel efficiency of 97% on 10020 CPU cores.

Two different parallel approaches for a hybrid fractional order Coronavirus model

In this paper, two different parallel approaches for a hybrid fractional order Coronavirus (2019-nCov) mathematical model are presented. Both parallel approaches are implemented using Julia high level language. Parallel algorithm implementations are developed for the HPC cluster using Message Passing Interface (MPI) technology and general-purpose computing on GPUs (GPGPU) using Compute Unified Device Architecture (CUDA) based on hardware environments. The algorithm implementation are used to solve the real-world problem of the hybrid fractional order Coronavirus (2019-nCov) mathematical model and to study the parallel efficiency. The introduced hybrid fractional order derivative is defined as a linear combination of the integral of Riemann-Liouville and the fractional order Caputo derivative. A parallel algorithm is designed based on the predictor-corrector method with the discretization of the Caputo proportional constant fractional hybrid operator for solving the model problem numerically. Simulation results show that, both the new parallel approaches achieve significant efficiency.

Scalable simulation of coupled adsorption and transport of methane in confined complex porous media with density preconditioning

The growing significance of shales and tight formations in the transition to less carbon-intensive and clean energy drives the research endeavor to understand the physics of gas flow within these systems. However, shales are composed of massively heterogeneous physical and chemical features. Most nano-sized pores connect to millimeter-scale fractures, leading to multiscale transport. These nano-scale pore throats demonstrate non-classical flow behavior, such as non-negligible slip velocities and adsorbed gas layers at the boundary. As a result, classical computational fluid dynamics models do not capture the physics. In this work, we develop a coupling scheme for the multiple-relaxation-time (MRT) lattice Boltzmann (LB) method that integrates the Peng-Robinson equation of state into a pseudo-potential interaction model to capture the physics of methane flow in irregular networks of channels that represent nano-scale porous media. We use atomistic simulations to calibrate and validate our model in slit nano-channels. We propose a preconditioning scheme to initialize the coupled transport and adsorption simulation of methane in complex porous media. The results of this implementation of LB agree with Direct Simulation Monte Carlo (DSMC) and Molecular Dynamics (MD) simulations. We then scale up the LB implementation through vectorization and indirect addressing. We parallelize it using Message Passing Interface (MPI) and OpenMP frameworks to simulate transport and adsorption in complex media with a million lattices. We analyze the differences between coupled and transport-only simulations in two case studies and show that considering phase behavior, i.e., adsorption, can significantly change the flow behavior. This work constitutes an important step towards bridging the gap between molecular flow and system-scale behavior of complex disordered porous media.

Numerical investigation of coaxial turbulent jet

The paper presents a numerical study of a compressible and turbulent coaxial jet, using a Large Eddy Simulation type approach. This study focuses on the implementation of a high-performance and modern numerical method to meet the requirements in terms of results accuracy and calculation cost. The code was parallelized using the Message Passing Interface 'MPI' library. Inviscid fluxes are evaluated using a new linearization solver for the equations characteristic of approximate Riemann problem and the solution has been advanced over time using an explicit two-step McCormack method. The purpose of this study is to understand the initial stage of the transition process from laminar to turbulent flow in coaxial jets. Therefore, it has been determined that the first step in the transition process from laminar to turbulent flow in coaxial jets is initiated by the development of Helmholtz (primary vortices) in sheer layers, while three-dimensionalization turbulence is initiated in the second step with the emergence of secondary vortices. The transverse evolution of Reynolds tensors showed that the axial Reynolds tensor contribution is associated with primary instability while the transversal Reynolds tensor contribution is associated with secondary instability. Flow self-similarity state is obtained in fully developed turbulence region. Axi-symmetric mode predominates on each shear layer in the initial flow region. The results obtained are in good agreement with experimental results similar to this study's simulated jet.

Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)

Abstract. Some programming languages are easy to develop at the cost of slow execution, while others are fast at runtime but much more difficult to write. Julia is a programming language that aims to be the best of both worlds – a development and production language at the same time. To test Julia's utility in scientific high-performance computing (HPC), we built an unstructured-mesh shallow water model in Julia and compared it against an established Fortran-MPI ocean model, the Model for Prediction Across Scales–Ocean (MPAS-Ocean), as well as a Python shallow water code. Three versions of the Julia shallow water code were created: for single-core CPU, graphics processing unit (GPU), and Message Passing Interface (MPI) CPU clusters. Comparing identical simulations revealed that our first version of the Julia model was 13 times faster than Python using NumPy, where both used an unthreaded single-core CPU. Further Julia optimizations, including static typing and removing implicit memory allocations, provided an additional 10–20× speed-up of the single-core CPU Julia model. The GPU-accelerated Julia code was almost identical in terms of performance to the MPI parallelized code on 64 processes, an unexpected result for such different architectures. Parallelized Julia-MPI performance was identical to Fortran-MPI MPAS-Ocean for low processor counts and ranges from 2× faster to 2× slower for higher processor counts. Our experience is that Julia development is fast and convenient for prototyping but that Julia requires further investment and expertise to be competitive with compiled codes. We provide advice on Julia code optimization for HPC systems.