Multiplicative Update Research Articles

Decentralized Federated Learning: Balancing Communication and Computing Costs

Decentralized stochastic gradient descent (SGD) is a driving engine for decentralized federated learning (DFL). The performance of decentralized SGD is jointly influenced by inter-node communications and local updates. In this paper, we propose a general DFL framework, which implements both multiple local updates and multiple inter-node communications periodically, to strike a balance between communication efficiency and model consensus. It can provide a general decentralized SGD analytical framework. We establish strong convergence guarantees for the proposed DFL algorithm without the assumption of convex objectives. The convergence rate of DFL can be optimized to achieve the balance of communication and computing costs under constrained resources. For improving communication efficiency of DFL, compressed communication is further introduced to the proposed DFL as a new scheme, named DFL with compressed communication (C-DFL). The proposed C-DFL exhibits linear convergence for strongly convex objectives. Experiment results based on MNIST and CIFAR-10 datasets illustrate the superiority of DFL over traditional decentralized SGD methods and show that C-DFL further enhances communication efficiency.

Multiple system atrophy therapeutic development update

Multiple system atrophy therapeutic development update

Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning

Federated learning (FL), as an emerging edge artificial intelligence paradigm, enables many edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order to second-order methods. However, these algorithms cannot be applied in scenarios where the gradient information is not available, e.g., federated black-box attack and federated hyperparameter tuning. To address this issue, in this paper we propose a derivative-free federated zeroth-order optimization (FedZO) algorithm featured by performing multiple local updates based on stochastic gradient estimators in each communication round and enabling partial device participation. Under non-convex settings, we derive the convergence performance of the FedZO algorithm on non-independent and identically distributed data and characterize the impact of the numbers of local iterates and participating edge devices on the convergence. To enable communication-efficient FedZO over wireless networks, we further propose an over-the-air computation (AirComp) assisted FedZO algorithm. With an appropriate transceiver design, we show that the convergence of AirComp-assisted FedZO can still be preserved under certain signal-to-noise ratio conditions. Simulation results demonstrate the effectiveness of the FedZO algorithm and validate the theoretical observations.

Convolutive Transfer Function-Based Multichannel Nonnegative Matrix Factorization for Overdetermined Blind Source Separation

Most multichannel blind source separation (BSS) approaches rely on a spatial model to encode the transfer functions from sources to microphones and a source model to encode the source power spectral density. The rank-1 spatial model has been widely exploited in independent component analysis (ICA), independent vector analysis (IVA), and independent low-rank matrix analysis (ILRMA). The full-rank spatial model is also considered in many BSS approaches, such as full-rank spatial covariance matrix analysis (FCA), multichannel nonnegative matrix factorization (MNMF), and FastMNMF, which can improve the separation performance in the case of long reverberation times. This paper proposes a new MNMF framework based on the convolutive transfer function (CTF) for overdetermined BSS. The time-domain convolutive mixture model is approximated by a frequency-wise convolutive mixture model instead of the widely adopted frequency-wise instantaneous mixture model. The iterative projection algorithm is adopted to estimate the demixing matrix, and the multiplicative update rule is employed to estimate nonnegative matrix factorization (NMF) parameters. Finally, the source image is reconstructed using a multichannel Wiener filter. The advantages of the proposed method are twofold. First, the CTF approximation enables us to use a short window to represent long impulse responses. Second, the full-rank spatial model can be derived based on the CTF approximation and slowly time-variant source variances, and close relationships between the proposed method and ILRMA, FCA, MNMF and FastMNMF are revealed. Extensive experiments show that the proposed algorithm achieves a higher separation performance than ILRMA and FastMNMF in reverberant environments.

Multi-resolution beta-divergence NMF for blind spectral unmixing

Many datasets are obtained as a resolution trade-off between two adversarial dimensions; for example between the frequency and the temporal resolutions for the spectrogram of an audio signal, and between the number of wavelengths and the spatial resolution for a hyper/multi-spectral image. To perform blind source separation using observations with different resolutions, a standard approach is to use coupled nonnegative matrix factorizations (NMF). As opposed to most previous works focusing on the least squares error measure, which is the β-divergence for β=2, we formulate this multi-resolution NMF problem for any β-divergence, and propose a novel algorithm based on the multiplicative updates (MU). We show on numerical experiments that the MU are able to obtain high resolutions in both dimensions on two applications: (1) blind unmixing of audio spectrograms: to the best of our knowledge, this is the first time a coupled NMF model is used in this context, and (2) the fusion of hyperspectral and multispectral images: we show that the MU compete favorably with state-of-the-art algorithms in particular in the presence of non-Gaussian noise.

A kind of new time-weighted nonnegative lasso index-tracking model and its application

This study describes improved index-tracking methods to replicate the target index’s market performance in a high-dimensional sparse linear regression with nonnegative constraints on the coefficients. The main objective of this study is to construct a sparse portfolio with a better prediction effect and robustness. Considering the influence of time factors on index tracking, we propose a time-weighted nonnegative lasso index tracking model under different market constraints and define two new time-weighted construction methods. This index tracking model is an extension of Lasso and has variable selection consistency and estimation consistency under time-weighted nonnegative irrepresentable conditions similar to the irrepresentable condition in Lasso. We use the multiplicative updates algorithm to obtain the model’s solution since it is faster and simpler. The constrained index tracking problem in the stock market without short sales is studied in the latter part. The empirical results indicate that the optimized time-weighted nonnegative lasso index tracking model can obtain a smaller out-of-sample tracking error. The constructed portfolio has a better prediction effect and robustness, and we find that the exponential time-weighted method is better than the linear time-weighted method in capturing time information.

Advances in Xmipp for Cryo-Electron Microscopy: From Xmipp to Scipion.

Xmipp is an open-source software package consisting of multiple programs for processing data originating from electron microscopy and electron tomography, designed and managed by the Biocomputing Unit of the Spanish National Center for Biotechnology, although with contributions from many other developers over the world. During its 25 years of existence, Xmipp underwent multiple changes and updates. While there were many publications related to new programs and functionality added to Xmipp, there is no single publication on the Xmipp as a package since 2013. In this article, we give an overview of the changes and new work since 2013, describe technologies and techniques used during the development, and take a peek at the future of the package.

A convergent algorithm for bi-orthogonal nonnegative matrix tri-factorization

A convergent algorithm for nonnegative matrix factorization (NMF) with orthogonality constraints imposed on both basis and coefficient matrices is proposed in this paper. This factorization concept was first introduced by Ding et al. (Proceedings of 12th ACM SIGKDD international conference on knowledge discovery and data mining, pp 126–135, 2006) with intent to further improve clustering capability of NMF. However, as the original algorithm was developed based on multiplicative update rules, the convergence of the algorithm cannot be guaranteed. In this paper, we utilize the technique presented in our previous work Mirzal (J Comput Appl Math 260:149–166, 2014a; Proceedings of the first international conference on advanced data and information engineering (DaEng-2013). Springer, pp 177–184, 2014b; IEEE/ACM Trans Comput Biol Bioinform 11(6):1208–1217, 2014c) to develop a convergent algorithm for this problem and prove that it converges to a stationary point inside the solution space. As it is very hard to numerically show the convergence of an NMF algorithm due to the slow convergence and numerical precision issues, experiments are instead performed to evaluate whether the algorithm has the nonincreasing property (a necessary condition for the convergence) where it is shown that the algorithm has this property. Further, clustering capability of the algorithm is also inspected by using Reuters-21578 data corpus.

Multiple sclerosis incidence update in province of Ferrara, Northern Italy, during COVID19 era

Previous descriptive surveys on multiple sclerosis (MS) in the province of Ferrara carried out by our own epidemiological research group in the previous decades, pointed out that this area was not at low-medium risk for MS. We aim to confirm the above assumption and to update MS frequency estimates in this area

An Updated Review of the Gestational Diabetes

High blood glucose levels during pregnancy can lead to unfavorable outcomes in the mother, fetus, and neonatal state. This review focused on the latest guidelines on diagnosis, monitoring, and treatment of gestational diabetes mellites. This narrative review was conducted by searching through several online databases including PubMed, Science Direct, and Embase for relevant articles using keywords such as "Gestational Diabetes", "Diabetes Mellitus", "Pregnancy", and "guideline" with no date limitations. Based on the literature review, proper treatment of diabetes during pregnancy results in a normal pregnancy, labor, postpartum state. The key to a normal pregnancy is to control and keep your blood sugar levels within the recommended range by various guidelines, which were the discussion subjects of this narrative review in detail. In addition to maintaining normal blood sugar levels before or during pregnancy, there should be a balance between diet, exercise, and insulin intake if indicated for treatment. Gestational diabetes control program needs close monitoring and appraisal with progression in the pregnancy. As a result, recognizing the burden of gestational diabetes is decisive for timely diagnosis and further evaluations by healthcare policymakers. Overall, multiple updates on the guidelines of gestational diabetes management are annually published, and a comparison of these guidelines could inform clinicians to update their approach.

A batch delivery mechanism of network update in software‐defined wide area networks

SummaryIn software‐defined network (SDN) architecture, the network decisions are all located on a logically centralized controller. Based on installing rules into switches of the data plane, the controller directly initiates and schedules a network update and which will cause a delay in the completion time of a network update. The situation is aggravated in wide area networks (WANs) further due to the switches deployed in a wide geographical area bring long propagation delay. On account of adopting the in‐band transmission style to deliver control messages instead of dedicated control channels is economics in WANs, the retransmission of control messages causing by congestion of the data channels is another unbearable delay source. This paper describes a new control information batch delivery mechanism (CIBDM) of network updates to relieve the delay and enhance the reliability of control message transmission in SD‐WANs. The characteristics of the novel scheme include the following: (1) every control information message may contain multiple update information of different forwarding devices, and each update information consists of a set of update instructions with dependencies; (2) by using a similar mechanism of onion routing, in the delivering process of control messages, it attains the security for each control rule of the specified devices; (3) with delegating the update consistency coordination into the data plane, it improves network update on the completion time. A comprehensive evaluation runs on 10 network topologies selected from all over the world. Experimental results show that CIBDM achieves rapid network updates with reliable message transmission.

Distributed Resource Allocation via Accelerated Saddle Point Dynamics

In this paper, accelerated saddle point dynamics is proposed for distributed resource allocation over a multi-agent network, which enables a hyper-exponential convergence rate. Specifically, an inertial fast-slow dynamical system with vanishing damping is introduced, based on which the distributed saddle point algorithm is designed. The dual variables are updated in two time scales, i.e., the fast manifold and the slow manifold. In the fast manifold, the consensus of the Lagrangian multipliers and the tracking of the constraints are pursued by the consensus protocol. In the slow manifold, the updating of the Lagrangian multipliers is accelerated by inertial terms. Hyper-exponential stability is defined to characterize a faster convergence of our proposed algorithm in comparison with conventional primal-dual algorithms for distributed resource allocation. The simulation of the application in the energy dispatch problem verifies the result, which demonstrates the fast convergence of the proposed saddle point dynamics.

High Update Rate Attitude Measurement Method of Star Sensors Based on Star Point Correction of Rolling Shutter Exposure.

Attitude update rate is one of the important indicators of star sensor performance. In order to resolve the problem of the low attitude update rate of star sensors, this paper proposes a star sensor attitude update method based on star point correction of rolling shutter exposure. Based on the characteristics of the asynchronous exposure of the rolling shutter, recursive estimation of the motion attitude and the corrected star point information were combined to realize multiple updates of the attitude in a single frame of the star map. Simulation and experimental results proved that the proposed method could increase the attitude update rate of a star sensor by 15 times, up to 150 Hz.

Graph matching based on fast normalized cut and multiplicative update mapping

Point correspondence is a fundamental problem in pattern recognition and computer vision, which can be tackled by graph matching. Since graph matching is basically an NP-complete problem, some approximate methods are proposed to solve it. Continuous relaxation offers an effective approximate method for graph matching problem. However, the discrete constraint is not taken into consideration in the optimization step. In this paper, a fast normalized cut based graph matching method is proposed, where the discrete constraint is introduced into the optimization step. Specifically, first a semidefinite positive affinity matrix based form objective function is constructed by introducing a regularization term which is related to the discrete constraint. Then the fast normalized cut algorithm is utilized to find the continuous solution. Last, the discrete solution of graph matching is obtained by a multiplicative update algorithm. Experiments on both synthetic points and real-world images validate the effectiveness of the proposed method by comparing it with the state-of-the-art methods.

Distributed associative memory network with memory refreshing loss

Despite recent progress in memory augmented neural network (MANN) research, associative memory networks with a single external memory still show limited performance on complex relational reasoning tasks. Especially the content-based addressable memory networks often fail to encode input data into rich enough representation for relational reasoning and this limits the relation modeling performance of MANN for long temporal sequence data. To address these problems, here we introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) which enhances the relation reasoning performance of MANN. Inspired by how the human brain works, our framework encodes data with distributed representation across multiple memory blocks and repeatedly refreshes the contents for enhanced memorization similar to the rehearsal process of the brain. For this procedure, we replace a single external memory with a set of multiple smaller associative memory blocks and update these sub-memory blocks simultaneously and independently for the distributed representation of input data. Moreover, we propose MRL which assists a task’s target objective while learning relational information existing in data. MRL enables MANN to reinforce an association between input data and task objective by reproducing stochastically sampled input data from stored memory contents. With this procedure, MANN further enriches the stored representations with relational information. In experiments, we apply our approaches to Differential Neural Computer (DNC), which is one of the representative content-based addressing memory models and achieves the state-of-the-art performance on both memorization and relational reasoning tasks.

LOSP: Overlap Synchronization Parallel With Local Compensation for Fast Distributed Training

When running in Parameter Server (PS), the Distributed Stochastic Gradient Descent (D-SGD) incurs significant communication delays and huge communication overhead due to the model synchronization. Moreover, considering the heterogeneity of computational capability among workers, traditional synchronization modes incur under-utilization of computational resources because fast workers have to wait for slow ones finishing the computation. Although our previous work OSP can effectively solve these problems by overlapping the computation and communication procedures and allowing adaptive multiple local updates in distributed training, it causes the staleness problem brought by the overlap, yielding a performance degradation. In this paper, we propose a new method named LOSP by introducing local compensation to our previous synchronization mechanism, which mitigates adverse effects caused by the overlapping synchronization. We theoretically prove that LOSP (1) preserves the same convergence rate as the sequential SGD for non-convex problems, and (2) exhibits good scalability due to the linear speedup property with respect to both the number of workers and the average number of local updates. Evaluations show that LOSP significantly improves performance over the state-of-the-art ones in terms of both convergence accuracy and communication cost.

Non-Negative Latent Factor Model Based on β-Divergence for Recommender Systems

Non-negative latent factor (NLF) models well represent high-dimensional and sparse (HiDS) matrices filled with non-negative data, which are frequently encountered in industrial applications like recommender systems. However, current NLF models mostly adopt Euclidean distance in their objective function, which represents a special case of a β-divergence function. Hence, it is highly desired to design a β-divergence-based NLF ( β-NLF) model that uses a β-divergence function, and investigate its performance in recommender systems as β varies. To do so, we first model β-NLF's learning objective with a β-divergence function. Subsequently, we deduce a general single latent factor-dependent, non-negative and multiplicative update scheme for β-NLF, and then design an efficient β-NLF algorithm. The experimental results on HiDS matrices from industrial applications indicate that by carefully choosing the value of β, β-NLF outperforms an NLF model with Euclidean distance in terms of accuracy for missing data prediction without increasing computational time. The research outcomes show the necessity of using an optimal β-divergence function in order to achieve the best performance of an NLF model on HiDS matrices. Hence, the proposed model has both theoretical and application significance.

A Sparse Dimensionality Reduction Approach Based on False Nearest Neighbors for Nonlinear Fault Detection

As a newly emerging multivariate statistical process monitoring method, non-negative matrix factorization (NMF) and its variants avoid the positive and negative cancellations between the extracted features because of the purely additive combination of non-negative components. Thus, they make a good match with this reality that the negative values of both observations and decomposed components are physically meaningless in many kinds of industrial processes. However, these methods are effective only for linearly separable problems and are not suitable for dealing with nonlinear process monitoring. In this article, the kernel-based method is integrated into the projective NMF (KPNMF) to improve the accuracy of fault detection, and the appropriate multiplicative update method is proven to be convergent. Furthermore, inspired by the false nearest neighbors (FNNs) method, a new dimensionality reduction approach (KPNMF-FNN) is presented to further reduce the original variables for determining the smallest dimension regression vector needed. Compared with the traditional methods, the proposed approach can greatly reduce the time and storage space required on the premise of maintaining a high fault detection rate and low false alarm rate. The experimental results on the Tennessee Eastman benchmark process and the pumping unit system show that the proposed algorithms have excellent performance and can effectively detect faults under the circumstances of retaining only the top 60% of the original variables.

Direction Matters: On Influence-Preserving Graph Summarization and Max-Cut Principle for Directed Graphs

Summarizing large-scale directed graphs into small-scale representations is a useful but less-studied problem setting. Conventional clustering approaches, based on Min-Cut-style criteria, compress both the vertices and edges of the graph into the communities, which lead to a loss of directed edge information. On the other hand, compressing the vertices while preserving the directed-edge information provides a way to learn the small-scale representation of a directed graph. The reconstruction error, which measures the edge information preserved by the summarized graph, can be used to learn such representation. Compared to the original graphs, the summarized graphs are easier to analyze and are capable of extracting group-level features, useful for efficient interventions of population behavior. In this letter, we present a model, based on minimizing reconstruction error with nonnegative constraints, which relates to a Max-Cut criterion that simultaneously identifies the compressed nodes and the directed compressed relations between these nodes. A multiplicative update algorithm with column-wise normalization is proposed. We further provide theoretical results on the identifiability of the model and the convergence of the proposed algorithms. Experiments are conducted to demonstrate the accuracy and robustness of the proposed method.

Bitcoin Transaction Forecasting With Deep Network Representation Learning

Bitcoin and its decentralized computing paradigm for digital currency trading are one of the most disruptive technology in the 21st century. This article presents a novel approach to developing a Bitcoin transaction forecast model, DLForecast, by leveraging deep neural networks for learning Bitcoin transaction network representations. DLForecast makes three original contributions. First, we explore three interesting properties between Bitcoin transaction accounts: topological connectivity pattern of Bitcoin accounts, transaction amount pattern, and transaction dynamics. Second, we construct a time-decaying reachability graph and a time-decaying transaction pattern graph, aiming at capturing different types of spatial-temporal Bitcoin transaction patterns. Third, we employ node embedding on both graphs and develop a Bitcoin transaction forecasting system between user accounts based on historical transactions with built-in time-decaying factor. To maintain an effective transaction forecasting performance, we leverage the multiplicative model update (MMU) ensemble to combine prediction models built on different transaction features extracted from each corresponding Bitcoin transaction graph. Evaluated on real-world Bitcoin transaction data, we show that our spatial-temporal forecasting model is efficient with fast runtime and effective with forecasting accuracy over 60 percent and improves the prediction performance by 50 percent when compared to forecasting model built on the static graph baseline.