High-dimensional Search Space Research Articles

VolcanoML

End-to-end AutoML has attracted intensive interests from both academia and industry, which automatically searches for ML pipelines in a space induced by feature engineering, algorithm/model selection, and hyper-parameter tuning. Existing AutoML systems, however, suffer from scalability issues when applying to application domains with large, high-dimensional search spaces. We present VOLCANOML, a scalable and extensible framework that facilitates systematic exploration of large AutoML search spaces. VOLCANOML introduces and implements basic building blocks that decompose a large search space into smaller ones, and allows users to utilize these building blocks to compose an execution plan for the AutoML problem at hand. VOLCANOML further supports a Volcano-style execution model - akin to the one supported by modern database systems - to execute the plan constructed. Our evaluation demonstrates that, not only does VOLCANOML raise the level of expressiveness for search space decomposition in AutoML, it also leads to actual findings of decomposition strategies that are significantly more efficient than the ones employed by state-of-the-art AutoML systems such as auto-sklearn.

Charting human development using a multi-endodermal organ atlas and organoid models

SummaryOrgans are composed of diverse cell types that traverse transient states during organogenesis. To interrogate this diversity during human development, we generate a single-cell transcriptome atlas from multiple developing endodermal organs of the respiratory and gastrointestinal tract. We illuminate cell states, transcription factors, and organ-specific epithelial stem cell and mesenchyme interactions across lineages. We implement the atlas as a high-dimensional search space to benchmark human pluripotent stem cell (hPSC)-derived intestinal organoids (HIOs) under multiple culture conditions. We show that HIOs recapitulate reference cell states and use HIOs to reconstruct the molecular dynamics of intestinal epithelium and mesenchyme emergence. We show that the mesenchyme-derived niche cue NRG1 enhances intestinal stem cell maturation in vitro and that the homeobox transcription factor CDX2 is required for regionalization of intestinal epithelium and mesenchyme in humans. This work combines cell atlases and organoid technologies to understand how human organ development is orchestrated.

Quicker ADC : Unlocking the Hidden Potential of Product Quantization With SIMD.

Efficient Nearest Neighbor (NN) search in high-dimensional spaces is a foundation of many multimedia retrieval systems. A common approach is to rely on Product Quantization, which allows the storage of large vector databases in memory and efficient distance computations. Yet, implementations of nearest neighbor search with Product Quantization have their performance limited by the many memory accesses they perform. Following this observation, André et al. proposed Quick ADC with up to 6× faster implementations of PQ m×4 product quantizers (PQ) leveraging specific SIMD instructions. Quicker ADC is a generalization of Quick ADC not limited to PQ m×4 codes and supporting AVX-512, the latest revision of SIMD instruction set. In doing so, Quicker ADC faces the challenge of using efficiently 5,6 and 7-bit shuffles that do not align to computer bytes or words. To this end, we introduce (i) irregular product quantizers combining sub-quantizers of different granularity and (ii) split tables allowing lookup tables larger than registers. We evaluate Quicker ADC with multiple indexes including Inverted Multi-Indexes and IVF HNSW and show that it outperforms the reference optimized implementations (i.e., FAISS and polysemous codes) for numerous configurations. Finally, we release an open-source fork of FAISS enhanced with Quicker ADC.

Creative AI Through Evolutionary Computation: Principles and Examples.

The main power of artificial intelligence is not in modeling what we already know, but in creating solutions that are new. Such solutions exist in extremely large, high-dimensional, and complex search spaces. Population-based search techniques, i.e. variants of evolutionary computation, are well suited to finding them. These techniques make it possible to find creative solutions to practical problems in the real world, making creative AI through evolutionary computation the likely “next deep learning.”

GSA improvement via the von Neumann stability analysis

The performance of the Gravitational Search Algorithm (GSA) depends on the gravitational constant G, which controls the balance of exploration and exploitation abilities. Improving the setting of this parameter has attracted many researchers. In this paper, we analyzed the GSA stability using the von Neumann stability criterion. First, we modeled the iterative process by a second-order differential equation and derived the first and second-order stability conditions. Then, based on these criteria, we suggested a new law to adjust the initial value of the parameter G, depending on the distance between objects and then on the search space. Some supporting simulations were carried out using different update laws of the gravitational constant (e.g., exponential, log-sigmoid, linear, and chaotic) on CEC 2017 benchmark functions in different search space dimensions. The achieved results show that the new setting leads to significantly better outcomes in high-dimensional search spaces (greater than 20). A comparison with other metaheuristics (Particle Swarm Optimization, Artificial Bee Colony, and Grey Wolf Optimizer) reveals that the new setting proffers GSA competitiveness. Tests on 23 real-world problems (CEC 2011 benchmark problems and the iron ores sintering problem) further proved all the merits of the proposed parameter setting.

Surrogate-Assisted Evolutionary Multitasking for Expensive Minimax Optimization in Multiple Scenarios

Minimax optimization is a widely-used formulation for robust design in multiple operating or environmental scenarios, where the worst-case performance among multiple scenarios is the optimization objective requiring a large number of quality assessments. Consequently, minimax optimization using evolutionary algorithms becomes prohibitive when each quality assessment involves computationally expensive numerical simulations or costly physical experiments. This work employs evolutionary multitasking optimization and surrogate techniques to address the challenges of the high-dimensional search space and high computation cost of minimax optimization. To this end, finding the worst-case scenario for different candidate solutions is considered as the optimization of multiple problems that can be solved simultaneously using the evolutionary multitasking optimization approach. In order to further speed up the proposed algorithm, a surrogate model in the joint space of the decision and scenario spaces is built to replace part of the expensive function evaluations. A generation-based model management strategy using a statistical hypothesis test is designed to manage the surrogate model. Experimental results on both benchmark problems and an airfoil design application indicate that the proposed algorithm can find satisfactory solutions with a very limited computational budget.

EI-LSH: An early-termination driven I/O efficient incremental c-approximate nearest neighbor search

Nearest neighbor in high-dimensional space has been widely used in various fields such as databases, data mining and machine learning. The problem has been well solved in low-dimensional space. However, when it comes to high-dimensional space, due to the curse of dimensionality, the problem is challenging. As a trade-off between accuracy and efficiency, c-approximate nearest neighbor (c-ANN) is considered instead of an exact NN search in high-dimensional space. A variety of c-ANN algorithms have been proposed, one of the important schemes for the c-ANN problem is called Locality-sensitive hashing (LSH), which projects a high-dimensional dataset into a low-dimensional dataset and can return a c-ANN with a constant probability. In this paper, we propose a new aggressive early-termination (ET) condition which stops the algorithm with LSH scheme earlier under the same theoretical guarantee, leading to a smaller I/O cost and less running time. Unlike the “conservative” early termination conditions used in previous studies, we propose an “aggressive” early termination condition which can stop much earlier. Though it is not absolutely safe and may result in the probability of failure, we can still devise more efficient algorithms under the same theoretical guarantee by carefully considering the failure probabilities brought by LSH scheme and early termination. Furthermore, we also introduce an incremental searching strategy. Unlike the previous LSH methods, which expand the bucket width in an exponential way, we employ a more natural search strategy to incrementally access the hash values of the objects. We also provide a rigorous theoretical analysis to underpin our incremental search strategy and the new early termination technique. Our comprehensive experiment results show that, compared with the state-of-the-art I/O efficient c-ANN techniques, our proposed algorithm, namely EI-LSH, can achieve much better I/O efficiency under the same theoretical guarantee.

Efficient locality-sensitive hashing over high-dimensional streaming data

Approximate nearest neighbor (ANN) search in high-dimensional spaces is fundamental in many applications. Locality-sensitive hashing (LSH) is a well-known methodology to solve the ANN problem. Existing LSH-based ANN solutions typically employ a large number of individual indexes optimized for searching efficiency. Updating such indexes might be impractical when processing high-dimensional streaming data. In this paper, we present a novel disk-based LSH index that offers efficient support for both searches and updates. The contributions of our work are threefold. First, we use the write-friendly LSM-trees to store the LSH projections to facilitate efficient updates. Second, we develop a novel estimation scheme to estimate the number of required LSH functions, with which the disk storage and access costs are effectively reduced. Third, we exploit both the collision number and the projection distance to improve the efficiency of candidate selection, improving the search performance with theoretical guarantees on the result quality. Experiments on four real-world datasets show that our proposal outperforms the state-of-the-art schemes.

Success History-Based Adaptive Differential Evolution Using Turning-Based Mutation

Single objective optimization algorithms are the foundation of establishing more complex methods, like constrained optimization, niching and multi-objective algorithms. Therefore, improvements to single objective optimization algorithms are important because they can impact other domains as well. This paper proposes a method using turning-based mutation that is aimed to solve the problem of premature convergence of algorithms based on SHADE (Success-History based Adaptive Differential Evolution) in high dimensional search space. The proposed method is tested on the Single Objective Bound Constrained Numerical Optimization (CEC2020) benchmark sets in 5, 10, 15, and 20 dimensions for all SHADE, L-SHADE, and jSO algorithms. The effectiveness of the method is verified by population diversity measure and population clustering analysis. In addition, the new versions (Tb-SHADE, TbL-SHADE and Tb-jSO) using the proposed turning-based mutation get apparently better optimization results than the original algorithms (SHADE, L-SHADE, and jSO) as well as the advanced DISH and the jDE100 algorithms in 10, 15, and 20 dimensional functions, but only have advantages compared with the advanced j2020 algorithm in 5 dimensional functions.

Bladder Boundary Estimation by Gravitational Search Algorithm Using Electrical Impedance Tomography

The noninvasive method for monitoring urine collection in the bladder is highly recommended for paraplegia patients. These patients are unable to discharge urine at the right time due to a weaker sensation for bladder volume. If the urine is not discharged in time, then the bladder size will increase and affect the neighboring organs and tissues. Electrical impedance tomography (EIT) is a noninvasive alternative method that can be used for bladder status monitoring. Size estimation of the bladder with EIT can clarify the bladder status. The bladder is a nonuniform structure with complex shape; therefore, higher order Fourier series is needed to represent the true shape. In estimating the higher order Fourier coefficients, the performance of a conventional modified Newton-Raphson (mNR) algorithm does not give the desired performance. In this article, the gravitational search algorithm (GSA) that is known for solving optimization problems in high-dimensional search space is proposed for the shape estimation of the bladder. GSA is a global optimization algorithm that has fast convergence and does not require the computation of Jacobian. Numerical experiments and phantom studies are performed to estimate the bladder size, and the results showed good performance.

Pruning SMAC search space based on key hyperparameters

SummaryMachine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a crucial role to optimize hyperparameters of algorithms. Generally, the procedure tuning the exposed hyperparameters of ML algorithm to achieve better performance is called Hyperparameters Optimization. Traditional hyperparameters optimization methods are manually exhaustive search, which is unavailable for high dimensional search spaces and large datasets. Recent automated sequential model‐based optimization led to substantial improvements for this problem, whose core idea is fitting a regression model to describe the importance and dependence of algorithm's performance on certain given hyperparameter setting. Sequential model‐based algorithm configuration (SMAC) is a the‐state‐of‐art approach, which is specified by four components, Initialize, FitModel, SelectConfigurations, and Intensify. In this article, we propose to add a pruning procedure into SMAC approach, it quantifies the importance of hyperparameters by analyzing the performance of a list of promising configurations and reduces search space by discarding noncritical and bad key hyperparameters. To investigate the impact of pruning for model's performance, we conducted experiments on the configuration space constructed by Auto‐Sklearn and compared the effect of run time and pruning ratio with our algorithm. The experiments results verified that, our method made the configuration selected by SMAC more stable and achieved better performance.

Research on power-law distribution of long-tail data and its application to tourism recommendation

PurposeOne challenge for tourism recommendation systems (TRSs) is the long-tail phenomenon of ratings or popularity among tourist products. This paper aims to improve the diversity and efficiency of TRSs utilizing the power-law distribution of long-tail data.Design/methodology/approachUsing Sina Weibo check-in data for example, this paper demonstrates that the long-tail phenomenon exists in user travel behaviors and fits the long-tail travel data with power-law distribution. To solve data sparsity in the long-tail part and increase recommendation diversity of TRSs, the paper proposes a collaborative filtering (CF) recommendation algorithm combining with power-law distribution. Furthermore, by combining power-law distribution with locality sensitive hashing (LSH), the paper optimizes user similarity calculation to improve the calculation efficiency of TRSs.FindingsThe comparison experiments show that the proposed algorithm greatly improves the recommendation diversity and calculation efficiency while maintaining high precision and recall of recommendation, providing basis for further dynamic recommendation.Originality/valueTRSs provide a better solution to the problem of information overload in the tourism field. However, based on the historical travel data over the whole population, most current TRSs tend to recommend hot and similar spots to users, lacking in diversity and failing to provide personalized recommendations. Meanwhile, the large high-dimensional sparse data in online social networks (OSNs) brings huge computational cost when calculating user similarity with traditional CF algorithms. In this paper, by integrating the power-law distribution of travel data and tourism recommendation technology, the authors’ work solves the problem existing in traditional TRSs that recommendation results are overly narrow and lack in serendipity, and provides users with a wider range of choices and hence improves user experience in TRSs. Meanwhile, utilizing locality sensitive hash functions, the authors’ work hashes users from high-dimensional vectors to one-dimensional integers and maps similar users into the same buckets, which realizes fast nearest neighbors search in high-dimensional space and solves the extreme sparsity problem of high dimensional travel data. Furthermore, applying the hashing results to user similarity calculation, the paper greatly reduces computational complexity and improves calculation efficiency of TRSs, which reduces the system load and enables TRSs to provide effective and timely recommendations for users.

Trading Convergence Rate with Computational Budget in High Dimensional Bayesian Optimization

Scaling Bayesian optimisation (BO) to high-dimensional search spaces is a active and open research problems particularly when no assumptions are made on function structure. The main reason is that at each iteration, BO requires to find global maximisation of acquisition function, which itself is a non-convex optimization problem in the original search space. With growing dimensions, the computational budget for this maximisation gets increasingly short leading to inaccurate solution of the maximisation. This inaccuracy adversely affects both the convergence and the efficiency of BO. We propose a novel approach where the acquisition function only requires maximisation on a discrete set of low dimensional subspaces embedded in the original high-dimensional search space. Our method is free of any low dimensional structure assumption on the function unlike many recent high-dimensional BO methods. Optimising acquisition function in low dimensional subspaces allows our method to obtain accurate solutions within limited computational budget. We show that in spite of this convenience, our algorithm remains convergent. In particular, cumulative regret of our algorithm only grows sub-linearly with the number of iterations. More importantly, as evident from our regret bounds, our algorithm provides a way to trade the convergence rate with the number of subspaces used in the optimisation. Finally, when the number of subspaces is "sufficiently large", our algorithm's cumulative regret is at most O*(√TγT) as opposed to O*(√DTγT) for the GP-UCB of Srinivas et al. (2012), reducing a crucial factor √D where D being the dimensional number of input space. We perform empirical experiments to evaluate our method extensively, showing that its sample efficiency is better than the existing methods for many optimisation problems involving dimensions up to 5000.

Combined fitness–violation epsilon constraint handling for differential evolution

Over recent decades, several efficient constraint-handling methods have been proposed in the area of evolutionary computation, and the $$\varepsilon $$ constraint method is considered as a state-of-the-art method for both single and multiobjective optimization. Still, very few attempts have been made to improve this method when applied to the differential evolution algorithm. This study proposes several novel constraint-handling methods following similar ideas, where the $$\varepsilon $$ level is defined based on the current violation in the population, individual $$\varepsilon $$ levels are maintained for every constraint, and a combination of fitness and constraint violation is used for determining infeasible solutions. The proposed approaches demonstrate superior performance compared to other approaches in terms of the feasibility rate in high-dimensional search spaces, as well as convergence to global optima. The experiments are performed using the CEC’2017 constrained suite benchmark functions and a set of Economic Load Dispatch problems.

Political Optimizer: A novel socio-inspired meta-heuristic for global optimization

This paper proposes a novel global optimization algorithm called Political Optimizer (PO), inspired by the multi-phased process of politics. PO is the mathematical mapping of all the major phases of politics such as constituency allocation, party switching, election campaign, inter-party election, and parliamentary affairs. The proposed algorithm assigns each solution a dual role by logically dividing the population into political parties and constituencies, which facilitates each candidate to update its position with respect to the party leader and the constituency winner. Moreover, a novel position updating strategy called recent past-based position updating strategy (RPPUS) is introduced, which is the mathematical modeling of the learning behaviors of the politicians from the previous election. The proposed algorithm is benchmarked with 50 unimodal, multimodal, and fixed dimensional functions against 15 state of the art algorithms. We show through experiments that PO has an excellent convergence speed with good exploration capability in early iterations. Root cause of such behavior of PO is incorporation of RPPUS and logical division of the population to assign dual role to each candidate solution. Using Wilcoxon rank-sum test, PO demonstrates statistically significant performance over the other algorithms. The results show that PO outperforms all other algorithms, and consistency in performance on such a comprehensive suite of benchmark functions proves the versatility of the algorithm. Furthermore, experiments demonstrate that PO is invariant to function shifting and performs consistently in very high dimensional search spaces. Finally, the applicability on real-world applications is demonstrated by efficiently solving four engineering optimization problems.

A flexible and adaptive grid algorithm for global optimization utilizing basin hopping Monte Carlo.

Global optimization is an active area of research in atomistic simulations, and many algorithms have been proposed to date. A prominent example is basin hopping Monte Carlo, which performs a modified Metropolis Monte Carlo search to explore the potential energy surface of the system of interest. These simulations can be very demanding due to the high-dimensional configurational search space. The effective search space can be reduced by utilizing grids for the atomic positions, but at the cost of possibly biasing the results if fixed grids are employed. In this paper, we present a flexible grid algorithm for global optimization that allows us to exploit the efficiency of grids without biasing the simulation outcome. The method is general and applicable to very heterogeneous systems, such as interfaces between two materials of different crystal structures or large clusters supported at surfaces. As a benchmark case, we demonstrate its performance for the well-known global optimization problem of Lennard-Jones clusters containing up to 100 particles. Despite the simplicity of this model potential, Lennard-Jones clusters represent a challenging test case since the global minima for some "magic" numbers of particles exhibit geometries that are very different from those of clusters with only a slightly different size.

Privacy-preserving high-dimensional data publishing for classification

With increasing amounts of personal information being collected by various organizations, many privacy models have been proposed for masking the collected data so that the data can be published without releasing individual privacy. However, most existing privacy models are not applicable to high-dimensional data, because of the sparseness of high-dimensional search space. In this paper, we present our solution to release high-dimensional data for privacy preservation and classification analysis. The challenge facing us is how to reduce high dimensions from the perspective of privacy models while preserving as much information as possible for classification. Our proposed approach tackles it by using an idea of vertical partition, which is to vertically divide the raw data into different disjointed subsets of smaller dimensionality. Specifically, our partition metric considers both the correlation between attributes and the proportion of attributes in each subset. Then a generalization method based on local recoding is employed to each subset separately for achieving k-anonymity. Considering the hardness of the optimal implementation of k-anonymity, the local recoding method finds a near-optimal solution with the goal of improving efficiency. The proposed approach was evaluated using two datasets, and the experimental results showed that it outperformed two related approaches in data utility at the same privacy level.

IPDDF: an improved precision dense descriptor based flow estimation

Large displacement optical flow algorithms are generally categorised into descriptor-based matching and pixel-based matching. Descriptor-based approaches are robust to geometric variation, however they have inherent localisation precision limitation due to histogram nature. This work presents a novel method called improved precision dense descriptor flow (IPDDF). The authors introduce an additional pixel-based matching cost within an existing dense Daisy descriptor framework to improve the flow estimation precision. Pixel-based features such as pixel colour and gradient are computed on top of the original descriptor in the authors' matching cost formulation. The pixel-based cost only requires a light-weight pre-computation and can be adapted seamlessly into the matching cost formulation. The framework is built based on the Daisy Filter Flow work. In the framework, Daisy descriptor and a filter-based efficient flow inference technique, as well as a randomised fast patch match search algorithm, are adopted. Given the novel matching cost formulation, the framework enables efficiently solving dense correspondence field estimation in a high-dimensional search space, which includes scale and orientation. Experiments on various challenging image pairs demonstrate the proposed algorithm enhances flow estimation accuracy as well as generate a spatially coherent yet edge-aware flow field result efficiently.

PM-LSH

Nearest neighbor (NN) search in high-dimensional spaces is inherently computationally expensive due to the curse of dimensionality. As a well-known solution to approximate NN search, locality-sensitive hashing (LSH) is able to answer c-approximate NN ( c -ANN) queries in sublinear time with constant probability. Existing LSH methods focus mainly on building hash bucket based indexing such that the candidate points can be retrieved quickly. However, existing coarse-grained structures fail to offer accurate distance estimation for candidate points, which translates into additional computational overhead when having to examine unnecessary points. This in turn reduces the performance of query processing. In contrast, we propose a fast and accurate LSH framework, called PM-LSH, that aims to compute the c -ANN query on large- scale, high-dimensional datasets. First, we adopt a simple yet effective PM-tree to index the data points. Second, we develop a tunable confidence interval to achieve accurate distance estimation and guarantee high result quality. Third, we propose an efficient algorithm on top of the PM-tree to improve the performance of computing c -ANN queries. Extensive experiments with real-world data offer evidence that PM-LSH is capable of outperforming existing proposals with respect to both efficiency and accuracy.

Manifold learning for efficient gravitational search algorithm

Metaheuristic algorithms provide a practical tool for optimization in a high-dimensional search space. Some mimic phenomenons of nature such as swarms and flocks. Prominent one is the Gravitational Search Algorithm (GSA) inspired by Newton’s law of gravity to manipulate agents modeled as point masses in the search space. The law of gravity states that interaction forces are inversely proportional to the squared distance in the Euclidean space between two objects. In this paper we claim that when the set of solutions lies in a lower-dimensional manifold, the Euclidean distance would yield unfitted forces and bias in the results, thus causing suboptimal and slower convergence. We propose to modify the algorithm and utilize geodesic distances gained through manifold learning via diffusion maps. In addition, we incorporate elitism by storing exploration data. We show the high performance of this approach in terms of the final solution value and the rate of convergence compared to other meta-heuristic algorithms including the original GSA. In this paper we also provide a comparative analysis of the state-of-the-art optimization algorithms on a large set of standard benchmark functions.