Non-separable Functions Research Articles

Cooperative coevolution for non-separable large-scale black-box optimization: Convergence analyses and distributed accelerations

Given the ubiquity of non-separable optimization problems in real worlds, in this paper we analyze and extend the large-scale version of the well-known cooperative coevolution (CC), a divide-and-conquer black-box optimization framework, on non-separable functions. First, we reveal empirical reasons of when decomposition-based methods are preferred or not in practice on some non-separable large-scale problems, which have not been clearly pointed out in many previous CC papers. Then, we formalize CC to a continuous-game model via simplification, but without losing its essential property. Different from previous evolutionary game theory for CC, our new model provides a much simpler but useful viewpoint to analyze its convergence, since only the pure Nash equilibrium concept is needed and more general fitness landscapes can be explicitly considered. Based on convergence analyses, we propose a hierarchical decomposition strategy for better generalization, as for any decomposition, there is a risk of getting trapped into a suboptimal Nash equilibrium. Finally, we use powerful distributed computing to accelerate it under the recent multi-level learning framework, which combines the fine-tuning ability from decomposition with the invariance property of CMA-ES. Experiments on a set of high-dimensional test functions validate both its search performance and scalability (w.r.t. CPU cores) on a clustering computing platform with 400 CPU cores.

Bayesian L 1/2 Regression

It is well known that Bridge regression enjoys superior theoretical properties when compared to traditional LASSO. However, the current latent variable representation of its Bayesian counterpart, based on the exponential power prior, is computationally expensive in higher dimensions. In this article, we show that the exponential power prior has a closed form scale mixture of normal decomposition for α = ( 1 2 ) γ , γ ∈ { 1 , 2 , … } . We call these types of priors L 1 2 prior for short. We develop an efficient partially collapsed Gibbs sampling scheme for computation using the L 1 2 prior and study theoretical properties when p > n . In addition, we introduce a non-separable Bridge penalty function inspired by the fully Bayesian formulation and a novel, efficient coordinate descent algorithm. We prove the algorithm’s convergence and show that the local minimizer from our optimization algorithm has an oracle property. Finally, simulation studies were carried out to illustrate the performance of the new algorithms. Supplementary materials for this article are available online.

DJADE: a reliable and efficient adaptive differential evolution algorithm with selection pressure control

Our main contribution is the proposition of a new and explicit way of adapting the parental selection pressure for Differential Evolution algorithm (DE) and its experimentation on an adaptive version of differential evolution algorithm named JADE in order to improve its reliability without diminishing its efficiency. The new algorithm named DJADE employs a genotypic population diversity measure (a measure of population convergence) to control the selection pressure parameter. A mechanism that increases the parental selection pressure as the search progresses and in the same time serves as a mean to detect when the algorithm begins to converge (the "convergence phase") is presented. In addition, DJADE is a novel adaptive DE, which adapts crossover, mutation and selection pressure parameters. Moreover, the coefficient K of greediness of the mutation scheme DE/current-to- best/ is also adapted. The motivation for the parameter adaptation is to obtain an algorithm that is capable to handle various problems with different characteristics, notably difficult multimodal and/or nonseparable functions. No case of premature convergence is observed during the experiments conducted on 13 classical functions test at 05 and 30 dimensions. The comparison with classical JADE and Classical DE are presented. DJADE is not only very efficient but also reliable on all functions tested.

Mathematical simulation of À LA Einstein loss of the non-separable character of a function for two particles at the quantum-classical boundary

In this article, we present mathematical simulations of non-separable functions (those that would “correspond” to two entangled quantum particles) that lose this character only as a result of approaching the quantum-classical frontier. No mathematical representation of the action of deteriorating agents of quantum entanglement was included in the simulation. Such loss manifests itself both from the point of view of position space and momentum space. For certain limits, compatible with the space considered, the non-separable functions defined here transform into separable functions or cancel each other out at this boundary, thus erasing the (mathematical representation of) the quantum characteristic with no equivalent in the classical world. These simulations do not concern the loss of a physical property or characteristic, but rather the loss of a mathematical characteristic of a function for two quantum particles. The “ghostly action at a distance”, colloquially expressed by Prof. A. Einstein, has a “spatially limited and non-instantaneous action” as it’s opposite, which mathematically takes place in simulations of non-separable quantum functions, as shown here.

Nonlinear and nonseparable structural functions in regression discontinuity designs with a continuous treatment

Many empirical examples of regression discontinuity (RD) designs concern a continuous treatment variable, but the theoretical aspects of such models are less studied. This study examines the identification and estimation of the structural function in fuzzy RD designs with a continuous treatment variable. The structural function fully describes the causal impact of the treatment on the outcome. We show that the nonlinear and nonseparable structural function can be nonparametrically identified at the RD cutoff under shape restrictions, including monotonicity and smoothness conditions. Based on the nonparametric identification equation, we propose a three-step semiparametric estimation procedure and establish the asymptotic normality of the estimator. The semiparametric estimator achieves the same convergence rate as in the case of a binary treatment variable. As an application of the method, we estimate the causal effect of sleep time on health status by using the discontinuity in natural light timing at time zone boundaries.

Efficiency of Stochastic Coordinate Proximal Gradient Methods on Nonseparable Composite Optimization

This paper deals with composite optimization problems having the objective function formed as the sum of two terms; one has a Lipschitz continuous gradient along random subspaces and may be nonconvex, and the second term is simple and differentiable but possibly nonconvex and nonseparable. Under these settings, we design a stochastic coordinate proximal gradient method that takes into account the nonseparable composite form of the objective function. This algorithm achieves scalability by constructing at each iteration a local approximation model of the whole nonseparable objective function along a random subspace with user-determined dimension. We outline efficient techniques for selecting the random subspace, yielding an implementation that has low cost per iteration, also achieving fast convergence rates. We present a probabilistic worst case complexity analysis for our stochastic coordinate proximal gradient method in convex and nonconvex settings; in particular, we prove high-probability bounds on the number of iterations before a given optimality is achieved. Extensive numerical results also confirm the efficiency of our algorithm. Funding: This work was supported by Norway Grants 2014-2021 [Grant ELO-Hyp 24/2020]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii [Grants PN-III-P4-PCE-2021-0720, L2O-MOC, nr 70/2022]; and the ITN-ETN project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement [Grant 861137].

The consumption multiplier of government spending: the role of substitutability between government spending and leisure

This paper presents an alternative explanation for the positive consumption response to government spending that is empirically observed, based on a non-separable utility function. We find that consumption can respond positively when leisure and government spending are substitutes.

Multi-strategy enhanced kernel search optimization and its application in economic emission dispatch problems

Abstract The kernel search optimizer (KSO) is a recent metaheuristic optimization algorithm that is based on kernel theory, eliminating the need for hyper-parameter adjustments, and demonstrating excellent global search capabilities. However, the original KSO exhibits insufficient accuracy in local search, and there is a high probability that it may fail to achieve local optimization in complex tasks. Therefore, this paper proposes a multi-strategy enhanced KSO (MSKSO) to enhance the local search ability of the KSO. The MSKSO combines several control strategies, including chaotic initialization, chaotic local search mechanisms, the high-altitude walk strategy (HWS), and the Levy flight (LF), to effectively balance exploration and exploitation. The MSKSO is compared with ten well-known algorithms on 50 benchmark test functions to validate its performance, including single-peak, multi-peak, separable variable, and non-separable variable functions. Additionally, the MSKSO is applied to two real engineering economic emission dispatch (EED) problems in power systems. Experimental results demonstrate that the performance of the MSKSO nearly optimizes that of other well-known algorithms and achieves favorable results on the EED problem. These case studies verify that the MSKSO outperforms other algorithms and can serve as an effective optimization tool.

Dynamical Sphere Regrouping Particle Swarm Optimization: A Proposed Algorithm for Dealing with PSO Premature Convergence in Large-Scale Global Optimization

Optimizing large-scale numerical problems is a significant challenge with numerous real-world applications. The optimization process is complex due to the multi-dimensional search spaces and possesses several locally optimal regions. In response to this issue, various metaheuristic algorithms and variations have been developed, including evolutionary and swarm intelligence algorithms and hybrids of different artificial intelligence techniques. Previous studies have shown that swarm intelligence algorithms like PSO perform poorly in high-dimensional spaces, even with algorithms focused on reducing the search space. However, we propose a modified version of the PSO algorithm called Dynamical Sphere Regrouping PSO (DSRegPSO) to avoid stagnation in local optimal regions. DSRegPSO is based on the PSO algorithm and modifies inertial behavior with a regrouping dynamical sphere mechanism and a momentum conservation physics effect. These behaviors maintain the swarm’s diversity and regulate the exploration and exploitation of the search space while avoiding stagnation in optimal local regions. The DSRegPSO mechanisms mimic the behavior of birds, moving particles similar to birds when they look for a new food source. Additionally, the momentum conservation effect mimics how birds react to collisions with the boundaries in their search space or when they are looking for food. We evaluated DSRegPSO by testing 15 optimizing functions with up to 1000 dimensions of the CEC’13 benchmark, a standard for evaluating Large-Scale Global Optimization used in Congress on Evolutionary Computation, and several journals. Our proposal improves the behavior of all variants of PSO registered in the toolkit of comparison for CEC’13 and obtains the best result in the non-separable functions against all the algorithms.

The effect of real money balances on international monetary policy transmission

This study examines the effect of real money balances on monetary policy in a two-country new Keynesian model. To accomplish this, the nonseparable utility function between consumption and real money balances is considered in the model. We demonstrate how a change in the money aggregate impacts the international spillover of quantitative easing (QE). This paper shows that whether the degree of the nonseparability parameter impacts international monetary policy spillover depends on the presence of zero lower bounds (ZLB) on nominal interest rates. This paper addresses that when the home and foreign countries face the ZLB, a foreign QE shock might cause a beggar-thy-neighbor effect on the home country.

Self-adaptive Artificial Bee Colony with a Candidate Strategy Pool

As a newly developed metaheuristic algorithm, the artificial bee colony (ABC) has garnered a lot of interest because of its strong exploration ability and easy implementation. However, its exploitation ability is poor and dramatically deteriorates for high-dimension and/or non-separable functions. To fix this defect, a self-adaptive ABC with a candidate strategy pool (SAABC-CS) is proposed. First, several search strategies with different features are assembled in the strategy pool. The top 10% of the bees make up the elite bee group. Then, we choose an appropriate strategy and implement this strategy for the present population according to the success rate learning information. Finally, we simultaneously implement some improved neighborhood search strategies in the scout bee phase. A total of 22 basic benchmark functions and the CEC2013 set of tests were employed to prove the usefulness of SAABC-CS. The impact of combining the five methods and the self-adaptive mechanism inside the SAABC-CS framework was examined in an experiment with 22 fundamental benchmark problems. In the CEC2013 set of tests, the comparison of SAABC-CS with a number of state-of-the-art algorithms showed that SAABC-CS outperformed these widely-used algorithms. Moreover, despite the increasing dimensions of CEC2013, SAABC-CS was robust and offered a higher solution quality.

Graph-based approximate message passing iterations

Abstract Approximate message passing (AMP) algorithms have become an important element of high-dimensional statistical inference, mostly due to their adaptability and concentration properties, the state evolution (SE) equations. This is demonstrated by the growing number of new iterations proposed for increasingly complex problems, ranging from multi-layer inference to low-rank matrix estimation with elaborate priors. In this paper, we address the following questions: is there a structure underlying all AMP iterations that unifies them in a common framework? Can we use such a structure to give a modular proof of state evolution equations, adaptable to new AMP iterations without reproducing each time the full argument? We propose an answer to both questions, showing that AMP instances can be generically indexed by an oriented graph. This enables to give a unified interpretation of these iterations, independent from the problem they solve, and a way of composing them arbitrarily. We then show that all AMP iterations indexed by such a graph verify rigorous SE equations, extending the reach of previous proofs and proving a number of recent heuristic derivations of those equations. Our proof naturally includes non-separable functions and we show how existing refinements, such as spatial coupling or matrix-valued variables, can be combined with our framework.

Stationary nonseparable space-time covariance functions on networks

Abstract The advent of data science has provided an increasing number of challenges with high data complexity. This paper addresses the challenge of space-time data where the spatial domain is not a planar surface, a sphere, or a linear network, but a generalised network (termed a graph with Euclidean edges). Additionally, data are repeatedly measured over different temporal instants. We provide new classes of stationary nonseparable space-time covariance functions where space can be a generalised network, a Euclidean tree, or a linear network, and where time can be linear or circular (seasonal). Because the construction principles are technical, we focus on illustrations that guide the reader through the construction of statistically interpretable examples. A simulation study demonstrates that the correct model can be recovered when compared to misspecified models. In addition, our simulation studies show that we effectively recover simulation parameters. In our data analysis, we consider a traffic accident dataset that shows improved model performance based on covariance specifications and network-based metrics.

On the welfare costs of business cycles: Beyond nondurable goods

This paper constitutes the first effort to measure the welfare costs of economic cycles considering that the consumer derives utility from the consumption of nondurable goods and services and the stock of durable goods. To accomplish such a task, we put forward a novel approach based on a nonseparable utility function. As a result, we take into account the complementarities among the consumption categories and the correlations of the shocks in those series. Our findings for the U.S. economy suggest that the single-good approach in the literature tends to underestimate the welfare costs of business cycles.

Convex and Nonconvex Risk-Based Linear Regression at Scale

The value at risk (VaR) and the conditional value at risk (CVaR) are two popular risk measures to hedge against the uncertainty of data. In this paper, we provide a computational toolbox for solving high-dimensional sparse linear regression problems under either VaR or CVaR measures, the former being nonconvex and the latter convex. Unlike the empirical risk (neutral) minimization models in which the overall losses are decomposable across data, the aforementioned risk-sensitive models have nonseparable objective functions so that typical first order algorithms are not easy to scale. We address this scaling issue by adopting a semismooth Newton-based proximal augmented Lagrangian method of the convex CVaR linear regression problem. The matrix structures of the Newton systems are carefully explored to reduce the computational cost per iteration. The method is further embedded in a majorization–minimization algorithm as a subroutine to tackle the nonconvex VaR-based regression problem. We also discuss an adaptive sieving strategy to iteratively guess and adjust the effective problem dimension, which is particularly useful when a solution path associated with a sequence of tuning parameters is needed. Extensive numerical experiments on both synthetic and real data demonstrate the effectiveness of our proposed methods. In particular, they are about 53 times faster than the commercial package Gurobi for the CVaR-based sparse linear regression with 4,265,669 features and 16,087 observations. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported in part by the NSF, the Division of Computing and Communication Foundations [Grant 2153352], the National Natural Science Foundation of China [Grant 12271187], and the Hong Kong Research Grant Council [Grant 15304019]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1282 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0012 ) at ( http://dx.doi.org/10.5281/zenodo.7483279 ).

An artificial bee colony algorithm with a balance strategy for wireless sensor network

As a popular population based evolutionary computation algorithm, artificial bee colony (ABC) algorithms have attracted many researchers to do further works. In this paper, by utilizing a linkage detection strategy to distinguish nonseparable and separable functions, we present new balance strategies for the updating equations of ABC on the two types of functions respectively. First, for accelerating convergence rate, we propose a historic array to preserve the best individual which the population ever achieved. Second, for the different role of employed and onlooker bees during iterations, compared with the traditional algorithm, we present two updating equations for the two types of bees on the nonseparable and separable functions respectively. Third, a new multi-dimensional updating mechanism for the worst individual and a new updating strategy for scout bees are presented respectively. At last, for verifying its effectiveness, we apply the improved algorithm to optimize the power consumption of sensor nodes in a wireless sensor network. In this paper, verified by the test on the network, the basic benchmark functions and CEC2014, our algorithm shows superior performance with the compared modern algorithms.

Well-posedness of mean field games master equations involving non-separable local Hamiltonians

In this paper we construct short time classical solutions to a class of master equations in the presence of non-degenerate individual noise arising in the theory of mean field games. The considered Hamiltonians are non-separable and local functions of the measure variable, therefore the equation is restricted to absolutely continuous measures whose densities lie in suitable Sobolev spaces. Our results hold for smooth enough Hamiltonians, without any additional structural conditions as convexity or monotonicity.

Reductions of non-separable approximate linear programs for network revenue management

We suggest a novel choice of non-separable basis functions for an approximate linear programming approach to the well-known network revenue management problem. Considering non-separability is particularly important when interdependencies between resources are large. Such a situation can be illustrated for example by a bus line, where different origin-destination pairs have many overlapping segments. Traditional separable approximation approaches tend to ignore the resulting interactions.We suggest to group resources into non-separable subnetworks. For each chosen subnetwork, basis functions either span the whole function space or consist of linear functions. Given this more general choice of basis functions, we extend existing reductions of approximate linear programs. If there is only one subnetwork, for which the basis functions span the whole function space, we prove the equivalence to a compact linear program of polynomial size. For the general case, we suggest an approximate reduction. Numerical examples illustrate our novel upper bounds for the maximum expected revenue and the corresponding competitive policies. In particular, we find that the added benefit of non-separability heavily depends on the network structure and the capacity.Our work helps to better understand the impact of assuming separability in network revenue management. The polynomial sized reductions make it possible to estimate the added average revenue resulting from incorporating interactions between resources. The theory we develop demonstrates how the interpretation of dual variables as state-action probabilities can be applied to reduce exponentially large approximate linear programs via variable aggregation.

An Efficient Hybrid Particle Swarm and Teaching-Learning-Based Optimization for Arch-Dam Shape Design

Particle swarm optimization is a popular meta-heuristic with highly explorative features; however, in its standard form it suffers from a poor convergence rate and weak search refinement on multi-dimensional problems. The present work improves the conventional particle swarm optimizer in three ways: adding a greedy selection for better intensification; embedding an extra movement borrowed from teacher–learner-based optimization; and utilizing a neighborhood strategy by averaging over a random half of the swarm. The performance of the proposed method is subsequently evaluated on three sets of problems. The first set includes uni-modal, multi-model, separable and non-separable test functions. The proposed method is compared with a standard particle swarm optimizer and its variants as well as other meta-heuristic algorithms. Engineering benchmark problems including the optimal design of a tubular column, a coiled spring, a pressure vessel and a cantilever beam constitute the second set. The third set includes constrained sizing design of a 120-bar dome truss and the optimal shape design of the Morrow Point double-arch concrete dam as a practical case study. Numerical results reveal considerable enhancement of the standard particle swarm via the proposed method to exhibit competitive performance with the other studied meta-heuristics. In the optimal design of Morrow Point Dam, the proposed method resulted in a material consumption 21 times smaller than the best of the initial population and 26% better than a recommended practical design.

Jacobi-Style Iteration for Distributed Submodular Maximization

This article presents a novel Jacobi-style iteration algorithm for solving the problem of distributed submodular maximization, in which multiple agents determine their strategies from the private sets so that a global, nonseparable submodular objective function is jointly maximized. Building on the multilinear extension of the submodular function, we expect to achieve the solution from a probabilistic, rather than deterministic, perspective, and thus, transfer the considered problem from the discrete domain into the continuous domain. Since it is observed that an unbiased estimation of the gradient of multilinear extension function can be obtained by sampling the agents’ local strategies, a projected stochastic gradient algorithm is proposed to solve the problem. Our algorithm enables simultaneous updates among all individual agents and guarantees to converge asymptotically to a desirable equilibrium solution. Such an equilibrium solution is shown to be at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1/2$</tex-math></inline-formula> suboptimal, which is comparable to the state-of-art in the literature. The convergence rate, which is characterized by the running average of gradient mapping, is proved to be <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1/T)$</tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$T$</tex-math></inline-formula> is the number of iterations. Moreover, we further enhance the proposed algorithm by handling the scenario in which agents’ communication delays are present. The enhanced algorithm admits a more realistic distributed implementation of our approach. Finally, a movie recommendation task is conducted on a real-world movie rating dataset to validate the numerical performance of our algorithms.