Network Structure Learning Research Articles

Multilevel switching memristor by compliance current adjustment for off-chip training of neuromorphic system

Multilevel operation is one of the most essential properties for synaptic devices to realize hardware artificial neural networks. Compliance current (Icc) adjustment is a multilevel programming method that can be utilized for a large-scale one-transistor and one-resistor (1T1R) array. It protects the devices from permanent breakdown by regulating abrupt switching. However, according to the reported literature so far, the number of conductance states in the Icc control method is insufficient to implement off-chip-trained neuromorphic systems. Therefore, we experimentally explore the feasibility of a larger number of conductance states using the Icc control method. We fabricated an Al2O3/TiOx-based resistive switching memory array, observed the conductance change while increasing Icc during set operations, and 64-level conductance states were statistically demonstrated. Furthermore, we verified that the 64-level states showed recognition performance close to that of a software-based neural network through off-chip learning of the convolutional neural network (CNN) structure. The fabricated synaptic device array with the Icc-control programming method is expected to contribute to the development of hardware neural network by reducing the information loss in the transfer process.

Application of deep learning for predicting the treatment performance of real municipal wastewater based on one-year operation of two anaerobic membrane bioreactors

In this study, data-driven deep learning methods were applied in order to model and predict the treatment of real municipal wastewater using anaerobic membrane bioreactors (AnMBRs). Based on the one-year operating data of two AnMBRs, six parameters related to the experimental conditions (temperature of reactor, temperature of environment, temperature of influent, influent pH, influent COD, and flux) and eight parameters for wastewater treatment evaluation (effluent pH, effluent COD, COD removal efficiency, biogas composition (CH4, N2, and CO2), biogas production rate, and oxidation-reduction potential) were selected to establish the data sets. Three deep learning network structures were proposed to analyze and reproduce the relationship between the input parameters and output evaluation parameters. The statistical analysis showed that deep learning closely agrees with the AnMBR experimental results. The prediction accuracy rate of the proposed densely connected convolutional network (DenseNet) can reach up to 97.44%, and the single calculation time can be reduced to within 1 s, suggesting the high performance of AnMBR treatment prediction with deep learning methods.

Falcon Optimization Algorithm for Bayesian Networks Structure Learning

In machine-learning, one of the useful scientific models for producing the structure of knowledge is Bayesian network, which can draw probabilistic dependency relationships between variables. The score and search is a method used for learning the structure of a Bayesian network. The authors apply the Falcon Optimization Algorithm (FOA) as a new approach to learning the structure of Bayesian networks. This paper uses the Reversing, Deleting, Moving and Inserting operations to adopt the FOA for approaching the optimal solution of Bayesian network structure. Essentially, the falcon prey search strategy is used in the FOA algorithm. The result of the proposed technique is compared with Pigeon Inspired optimization, Greedy Search, and Simulated Annealing using the BDeu score function. The authors have also examined the performances of the confusion matrix of these techniques utilizing several benchmark data sets. As shown by the evaluations, the proposed method has more reliable performance than the other algorithms including producing better scores and accuracy values.

A Generative Adversarial Network Structure for Learning with Small Numerical Data Sets

In recent years, generative adversarial networks (GANs) have been proposed to generate simulated images, and some works of literature have applied GAN to the analysis of numerical data in many fields, such as the prediction of building energy consumption and the prediction and identification of liver cancer stages. However, these studies are based on sufficient data volume. In the current era of globalization, the demand for rapid decision-making is increasing, but the data available in a short period of time is scarce. As a result, machine learning may not provide precise results. Obtaining more information from a small number of samples has become an important issue. Therefore, this study aimed to modify the generative adversarial network structure for learning with small numerical datasets, starting with the Wasserstein GAN (WGAN) as the GAN architecture, and using mega-trend-diffusion (MTD) to limit the bound of virtual samples that the GAN generates. The model verification of our proposed structure was conducted with two datasets in the UC Irvine Machine Learning Repository, and the performance was evaluated using three criteria: accuracy, standard deviation, and p-value. The experiment result shows that, using this improved GAN architecture (WGAN_MTD), small sample data can also be used to generate virtual samples that are similar to real samples through GAN.

Network structure and social learning

We describe results from Dasaratha and He [DH21a] and Dasaratha and He [DH20] about how network structure influences social learning outcomes. These papers share a tractable sequential model that lets us compare learning dynamics across networks. With Bayesian agents, incomplete networks can generate informational confounding that makes learning arbitrarily inefficient. With naive agents, related forces can lead to mislearning.

Automatic Modulation Classification Based on Cauchy-Score Constellation and Lightweight Network Under Impulsive Noise

Automatic modulation classification in the manner of pattern recognition has caught massive attention due to its simplicity in workflow design. Recently, enormous interest has been aroused because of the boosting development of deep learning and numerous network structures, leading to various substantial contributions. However, to the best of our knowledge, current approaches fail to handle two essential concerns simultaneously: the statistical distribution of the noise and the computational complexity due to the network structure. Since these factors play critical roles when facing practical applications, a modified constellation concept based on the Score function of Cauchy distribution is proposed as a robust feature to impulsive noise. Besides, a lightweight structure based on Shuffle Unit and Gated Recurrent Unit is proposed as the recognizer to lower the potential risk of overfitting and cope with the time-consuming problem. Monte-Carlo experiments involving multiple comparison algorithms are executed under different conditions, and results verify the superior performance of the proposed AMC scheme.

PEWOBS: An efficient Bayesian network learning approach based on permutation and extensible ordering-based search

How to efficiently construct Bayesian network (BN) is a major research hotspot in the field of artificial intelligence. Ordering-based search (OBS) algorithm is a kind of search scoring-based BN structure learning algorithm with good performance. This kind of algorithm constructs the BN structure by optimizing the ordering of variables iteratively. However, it has the problems of poor initial ordering quality, insufficient ordering optimization and low accuracy. In order to solve these problems, this paper proposes a Bayesian network structure learning approach (PEWOBS) based on full permutation and extensible ordering-based search. It firstly uses the local learning and pruning method to process the dataset with small and large number of variables respectively, and obtains the candidate parent sets of each variable efficiently. Then it obtains the high quality of initial ordering in consideration of the global benefits and individual losses. On the basis of the initial optimization of ordering by the extensible window operator, the second optimization of sub-ordering is performed using the full permutation operator. The final BN structure with the largest score will be obtained according to the final ordering. Experimental results show that PEWOBS can get a better network structure in the first iteration. It has better network learning accuracy and efficiency compared with several kinds of existing OBS approaches.Availability and implementation: codes and experiment dataset are available at: http://122.205.95.139/PEWOBS/.

Determining the direction of the local search in topological ordering space for Bayesian network structure learning

Local search in a topological ordering space is an efficient way of learning Bayesian network structures in large-scale problems. However, existing algorithms typically focus on stochastically developing the neighborhood of an ordering without a specific direction and can quickly stop at a local optimum. In this study, a novel approach is proposed to improve the capability of a local search by determining the search direction. The direction of the search step is identified with respect to the priority in a score cache. We also design robust terminal conditions and insertion methods based on the proposed operator. The direction of escaping from local optima is identified by transferring the ordering to a new restart, which implies an equivalent class of the optimal structure. Moreover, we adopt a breadth-first search method for the conversion between structures and orderings. The identification of the direction can accelerate the convergence of the local search and acquire the higher quality structure. Furthermore, our experimental results demonstrated that the proposed methods highly improve the accuracy and efficiency of learning the optimal ordering from the training dataset compared with state-of-the-art algorithms.

Leveraging Domain Knowledge in Multitask Bayesian Network Structure Learning

Network structure learning algorithms have aided network discovery in fields such as bioinformatics, neuroscience, ecology and social science. However, challenges remain in learning informative networks for related sets of tasks because the search space of Bayesian network structures is characterized by large basins of approximately equivalent solutions. Multitask algorithms select a set of networks that are near each other in the search space, rather than a score-equivalent set of networks chosen from independent regions of the space. This selection preference allows a domain expert to see only differences supported by the data. However, the usefulness of these algorithms for scientific datasets is limited because existing algorithms naively assume that all pairs of tasks are equally related. We introduce a framework that relaxes this assumption by incorporating domain knowledge about task-relatedness into the learning objective. Using our framework, we introduce the first multitask Bayesian network algorithm that leverages domain knowledge about the relatedness of tasks. We use our algorithm to explore the effect of task-relatedness on network discovery and show that our algorithm learns networks that are closer to ground truth than naive algorithms and that our algorithm discovers patterns that are interesting.

Markov Network Structure Learning: A Randomized Feature Generation Approach

The structure of a Markov network is typically learned in one of two ways. The first approach is to treat this task as a global search problem. However, these algorithms are slow as they require running the expensive operation of weight (i.e., parameter) learning many times. The second approach involves learning a set of local models and then combining them into a global model. However, it can be computationally expensive to learn the local models for datasets that contain a large number of variables and/or examples. This paper pursues a third approach that views Markov network structure learning as a feature generation problem. The algorithm combines a data-driven, specific-to-general search strategy with randomization to quickly generate a large set of candidate features that all have support in the data. It uses weight learning, with L1 regularization, to select a subset of generated features to include in the model. On a large empirical study, we find that our algorithm is equivalently accurate to other state-of-the-art methods while exhibiting a much faster run time.

A novel MapReduce-based deep convolutional neural network algorithm

Deep convolutional neural networks (DCNNs), with their complex network structure and powerful feature learning and feature expression capabilities, have been remarkable successes in many large-scale recognition tasks. However, with the expectation of memory overhead and response time, along with the increasing scale of data, DCNN faces three non-rival challenges in a big data environment: excessive network parameters, slow convergence, and inefficient parallelism. To tackle these three problems, this paper develops a deep convolutional neural networks optimization algorithm (PDCNNO) in the MapReduce framework. The proposed method first pruned the network to obtain a compressed network in order to effectively reduce redundant parameters. Next, a conjugate gradient method based on modified secant equation (CGMSE) is developed in the Map phase to further accelerate the convergence of the network. Finally, a load balancing strategy based on regulate load rate (LBRLA) is proposed in the Reduce phase to quickly achieve equal grouping of data and thus improving the parallel performance of the system. We compared the PDCNNO algorithm with other algorithms on three datasets, including SVHN, EMNIST Digits, and ISLVRC2012. The experimental results show that our algorithm not only reduces the space and time overhead of network training but also obtains a well-performing speed-up ratio in a big data environment.

Malware Classification Based on Multilayer Perception and Word2Vec for IoT Security

With the construction of smart cities, the number of Internet of Things (IoT) devices is growing rapidly, leading to an explosive growth of malware designed for IoT devices. These malware pose a serious threat to the security of IoT devices. The traditional malware classification methods mainly rely on feature engineering. To improve accuracy, a large number of different types of features will be extracted from malware files in these methods. That brings a high complexity to the classification. To solve these issues, a malware classification method based on Word2Vec and Multilayer Perception (MLP) is proposed in this article. First, for one malware sample, Word2Vec is used to calculate a word vector for all bytes of the binary file and all instructions in the assembly file. Second, we combine these vectors into a 256x256x2-dimensional matrix. Finally, we designed a deep learning network structure based on MLP to train the model. Then the model is used to classify the testing samples. The experimental results prove that the method has a high accuracy of 99.54%.

BIC-based node order learning for improving Bayesian network structure learning

BIC-based node order learning for improving Bayesian network structure learning

Selected Artificial Intelligence Methods in the Risk Analysis of Damage to Masonry Buildings Subject to Long-Term Underground Mining Exploitation

This paper presents an advanced computational approach to assess the risk of damage to masonry buildings subjected to negative kinematic impacts of underground mining exploitation. The research goals were achieved using selected tools from the area of artificial intelligence (AI) methods. Ultimately, two models of damage risk assessment were built using the Naive Bayes classifier (NBC) and Bayesian Networks (BN). The first model was used to compare results obtained using the more computationally advanced Bayesian network methodology. In the case of the Bayesian network, the unknown Directed Acyclic Graph (DAG) structure was extracted using Chow-Liu’s Tree Augmented Naive Bayes (TAN-CL) algorithm. Thus, one of the methods involving Bayesian Network Structure Learning from data (BNSL) was implemented. The application of this approach represents a novel scientific contribution in the interdisciplinary field of mining and civil engineering. The models created were verified with respect to quality of fit to observed data and generalization properties. The connections in the Bayesian network structure obtained were also verified with respect to the observed relations occurring in engineering practice concerning the assessment of the damage intensity to masonry buildings in mining areas. This allowed evaluation of the model and justified the utility of the conducted research in the field of protection of mining areas. The possibility of universal application of the Bayesian network, both in the case of damage prediction and diagnosis of its potential causes, was also pointed out.

Risk spillover network structure learning for correlated financial assets: A directed acyclic graph approach

Using cross-asset return data in global financial markets, we propose a novel empirical framework to identify the causal structure of the asset risk spillover network. The joint return distribution of the global financial system can be characterized using a directed acyclic graph approach. However, since assets tend to be highly correlated during market turbulence, when adopting a nodewise penalized regression approach for neighborhood estimation, parameter estimates will receive large standard errors, and edges cannot be reliably estimated. In this work, we propose a two-stage approach for directed acyclic graph skeleton estimation for highly correlated variables. In the first stage, a variable screening ensemble is incorporated into the sparse partial least squares regression method to both reduce the size of the active variables set and impose an adaptive penalization on the weight vectors. In the second stage, a modified PC algorithm based on Gram-Schmidt orthogonalization is applied to remove the false positive edges. Simulation studies are conducted to demonstrate the effectiveness of the proposed method. Finally, we apply our method to analyze the asset risk spillover channels for international financial assets during the COVID-19 pandemic.

Causal constraint pruning for exact learning of Bayesian network structure

How to improve the efficiency of exact learning of the Bayesian network structure is a challenging issue. In this paper, four different causal constraints algorithms are added into score calculations to prune possible parent sets, improving state-of-the-art learning algorithms' efficiency. Experimental results indicate that exact learning algorithms can significantly improve the efficiency with only a slight loss of accuracy. Under causal constraints, these exact learning algorithms can prune about 70% possible parent sets and reduce about 60% running time while only losing no more than 2% accuracy on average. Additionally, with sufficient samples, exact learning algorithms with causal constraints can also obtain the optimal network. In general, adding max-min parents and children constraints has better results in terms of efficiency and accuracy among these four causal constraints algorithms.

Negative binomial graphical model with excess zeros

AbstractMarkov random field or undirected graphical models (GM) are a popular class of GM useful in various fields because they provide an intuitive and interpretable graph expressing the complex relationship between random variables. The zero‐inflated local Poisson graphical model has been proposed as a graphical model for count data with excess zeros. However, as count data are often characterized by over‐dispersion, the local Poisson graphical model may suffer from a poor fit to data. In this paper, we propose a zero‐inflated local negative binomial (NB) graphical model. Due to the dependencies of parameters in our models, a direct optimization of the objective function is difficult. Instead, we devise expectation‐minimization algorithms based on two different parametrizations for the NB distribution. Through a simulation study, we illustrate the effectiveness of our method for learning network structure from over‐dispersed count data with excess zeros. We further apply our method to real data to estimate its network structure.

Improved Local Search with Momentum for Bayesian Networks Structure Learning.

Bayesian Networks structure learning (BNSL) is a troublesome problem that aims to search for an optimal structure. An exact search tends to sacrifice a significant amount of time and memory to promote accuracy, while the local search can tackle complex networks with thousands of variables but commonly gets stuck in a local optimum. In this paper, two novel and practical operators and a derived operator are proposed to perturb structures and maintain the acyclicity. Then, we design a framework, incorporating an influential perturbation factor integrated by three proposed operators, to escape current local optimal and improve the dilemma that outcomes trap in local optimal. The experimental results illustrate that our algorithm can output competitive results compared with the state-of-the-art constraint-based method in most cases. Meanwhile, our algorithm reaches an equivalent or better solution found by the state-of-the-art exact search and hybrid methods.

A new PC-PSO algorithm for Bayesian network structure learning with structure priors

Bayesian network structure learning is the basis of parameter learning and Bayesian inference. However, it is a NP-hard problem to find the optimal structure of Bayesian networks because the computational complexity increases exponentially with the increasing number of nodes. Hence, numerous algorithms have been proposed to obtain feasible solutions, while almost all of them are of certain limits. In this paper, we adopt a heuristic algorithm to learn the structure of Bayesian networks, and this algorithm can provide a reasonable solution to combine the PC and Particle Swarm Optimization (PSO) algorithms. Moreover, we consider structure priors to improve the performance of our PC-PSO algorithm. Meanwhile, we utilize a new mutation operator called Uniform Mutation by Addition and Deletion (UMAD) and a crossover operator called Uniform Crossover. Experiments on different networks show that the approach proposed in this paper has achieved better Bayesian Information Criterion (BIC) scores than other algorithms.

Heterogeneous Graph Structure Learning for Graph Neural Networks

Heterogeneous Graph Neural Networks (HGNNs) have drawn increasing attention in recent years and achieved outstanding performance in many tasks. The success of the existing HGNNs relies on one fundamental assumption, i.e., the original heterogeneous graph structure is reliable. However, this assumption is usually unrealistic, since the heterogeneous graph in reality is inevitably noisy or incomplete. Therefore, it is vital to learn the heterogeneous graph structure for HGNNs rather than rely only on the raw graph structure. In light of this, we make the first attempt towards learning an optimal heterogeneous graph structure for HGNNs and propose a novel framework HGSL, which jointly performs Heterogeneous Graph Structure Learning and GNN parameters learning for classification task. Different from traditional GSL on homogeneous graph, considering the heterogeneity of different relations in heterogeneous graph, HGSL generates each relation subgraph independently. Specifically, in each generated relation subgraph, HGSL not only considers the feature similarity by generating feature similarity graph, but also considers the complex heterogeneous interactions in features and semantics by generating feature propagation graph and semantic graph. Then, these graphs are fused to a learned heterogeneous graph and optimized together with a GNN towards classification objective. Extensive experiments on real-world graphs demonstrate that the proposed framework significantly outperforms the state-of-the-art methods.