High-dimensional Search Space Research Articles

EI-LSH: An early-termination driven I/O efficient incremental c-approximate nearest neighbor search

Nearest neighbor in high-dimensional space has been widely used in various fields such as databases, data mining and machine learning. The problem has been well solved in low-dimensional space. However, when it comes to high-dimensional space, due to the curse of dimensionality, the problem is challenging. As a trade-off between accuracy and efficiency, c-approximate nearest neighbor (c-ANN) is considered instead of an exact NN search in high-dimensional space. A variety of c-ANN algorithms have been proposed, one of the important schemes for the c-ANN problem is called Locality-sensitive hashing (LSH), which projects a high-dimensional dataset into a low-dimensional dataset and can return a c-ANN with a constant probability. In this paper, we propose a new aggressive early-termination (ET) condition which stops the algorithm with LSH scheme earlier under the same theoretical guarantee, leading to a smaller I/O cost and less running time. Unlike the “conservative” early termination conditions used in previous studies, we propose an “aggressive” early termination condition which can stop much earlier. Though it is not absolutely safe and may result in the probability of failure, we can still devise more efficient algorithms under the same theoretical guarantee by carefully considering the failure probabilities brought by LSH scheme and early termination. Furthermore, we also introduce an incremental searching strategy. Unlike the previous LSH methods, which expand the bucket width in an exponential way, we employ a more natural search strategy to incrementally access the hash values of the objects. We also provide a rigorous theoretical analysis to underpin our incremental search strategy and the new early termination technique. Our comprehensive experiment results show that, compared with the state-of-the-art I/O efficient c-ANN techniques, our proposed algorithm, namely EI-LSH, can achieve much better I/O efficiency under the same theoretical guarantee.

Efficient locality-sensitive hashing over high-dimensional streaming data

Approximate nearest neighbor (ANN) search in high-dimensional spaces is fundamental in many applications. Locality-sensitive hashing (LSH) is a well-known methodology to solve the ANN problem. Existing LSH-based ANN solutions typically employ a large number of individual indexes optimized for searching efficiency. Updating such indexes might be impractical when processing high-dimensional streaming data. In this paper, we present a novel disk-based LSH index that offers efficient support for both searches and updates. The contributions of our work are threefold. First, we use the write-friendly LSM-trees to store the LSH projections to facilitate efficient updates. Second, we develop a novel estimation scheme to estimate the number of required LSH functions, with which the disk storage and access costs are effectively reduced. Third, we exploit both the collision number and the projection distance to improve the efficiency of candidate selection, improving the search performance with theoretical guarantees on the result quality. Experiments on four real-world datasets show that our proposal outperforms the state-of-the-art schemes.

Success History-Based Adaptive Differential Evolution Using Turning-Based Mutation

Single objective optimization algorithms are the foundation of establishing more complex methods, like constrained optimization, niching and multi-objective algorithms. Therefore, improvements to single objective optimization algorithms are important because they can impact other domains as well. This paper proposes a method using turning-based mutation that is aimed to solve the problem of premature convergence of algorithms based on SHADE (Success-History based Adaptive Differential Evolution) in high dimensional search space. The proposed method is tested on the Single Objective Bound Constrained Numerical Optimization (CEC2020) benchmark sets in 5, 10, 15, and 20 dimensions for all SHADE, L-SHADE, and jSO algorithms. The effectiveness of the method is verified by population diversity measure and population clustering analysis. In addition, the new versions (Tb-SHADE, TbL-SHADE and Tb-jSO) using the proposed turning-based mutation get apparently better optimization results than the original algorithms (SHADE, L-SHADE, and jSO) as well as the advanced DISH and the jDE100 algorithms in 10, 15, and 20 dimensional functions, but only have advantages compared with the advanced j2020 algorithm in 5 dimensional functions.

Bladder Boundary Estimation by Gravitational Search Algorithm Using Electrical Impedance Tomography

The noninvasive method for monitoring urine collection in the bladder is highly recommended for paraplegia patients. These patients are unable to discharge urine at the right time due to a weaker sensation for bladder volume. If the urine is not discharged in time, then the bladder size will increase and affect the neighboring organs and tissues. Electrical impedance tomography (EIT) is a noninvasive alternative method that can be used for bladder status monitoring. Size estimation of the bladder with EIT can clarify the bladder status. The bladder is a nonuniform structure with complex shape; therefore, higher order Fourier series is needed to represent the true shape. In estimating the higher order Fourier coefficients, the performance of a conventional modified Newton-Raphson (mNR) algorithm does not give the desired performance. In this article, the gravitational search algorithm (GSA) that is known for solving optimization problems in high-dimensional search space is proposed for the shape estimation of the bladder. GSA is a global optimization algorithm that has fast convergence and does not require the computation of Jacobian. Numerical experiments and phantom studies are performed to estimate the bladder size, and the results showed good performance.

Pruning SMAC search space based on key hyperparameters

SummaryMachine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a crucial role to optimize hyperparameters of algorithms. Generally, the procedure tuning the exposed hyperparameters of ML algorithm to achieve better performance is called Hyperparameters Optimization. Traditional hyperparameters optimization methods are manually exhaustive search, which is unavailable for high dimensional search spaces and large datasets. Recent automated sequential model‐based optimization led to substantial improvements for this problem, whose core idea is fitting a regression model to describe the importance and dependence of algorithm's performance on certain given hyperparameter setting. Sequential model‐based algorithm configuration (SMAC) is a the‐state‐of‐art approach, which is specified by four components, Initialize, FitModel, SelectConfigurations, and Intensify. In this article, we propose to add a pruning procedure into SMAC approach, it quantifies the importance of hyperparameters by analyzing the performance of a list of promising configurations and reduces search space by discarding noncritical and bad key hyperparameters. To investigate the impact of pruning for model's performance, we conducted experiments on the configuration space constructed by Auto‐Sklearn and compared the effect of run time and pruning ratio with our algorithm. The experiments results verified that, our method made the configuration selected by SMAC more stable and achieved better performance.

Research on power-law distribution of long-tail data and its application to tourism recommendation

PurposeOne challenge for tourism recommendation systems (TRSs) is the long-tail phenomenon of ratings or popularity among tourist products. This paper aims to improve the diversity and efficiency of TRSs utilizing the power-law distribution of long-tail data.Design/methodology/approachUsing Sina Weibo check-in data for example, this paper demonstrates that the long-tail phenomenon exists in user travel behaviors and fits the long-tail travel data with power-law distribution. To solve data sparsity in the long-tail part and increase recommendation diversity of TRSs, the paper proposes a collaborative filtering (CF) recommendation algorithm combining with power-law distribution. Furthermore, by combining power-law distribution with locality sensitive hashing (LSH), the paper optimizes user similarity calculation to improve the calculation efficiency of TRSs.FindingsThe comparison experiments show that the proposed algorithm greatly improves the recommendation diversity and calculation efficiency while maintaining high precision and recall of recommendation, providing basis for further dynamic recommendation.Originality/valueTRSs provide a better solution to the problem of information overload in the tourism field. However, based on the historical travel data over the whole population, most current TRSs tend to recommend hot and similar spots to users, lacking in diversity and failing to provide personalized recommendations. Meanwhile, the large high-dimensional sparse data in online social networks (OSNs) brings huge computational cost when calculating user similarity with traditional CF algorithms. In this paper, by integrating the power-law distribution of travel data and tourism recommendation technology, the authors’ work solves the problem existing in traditional TRSs that recommendation results are overly narrow and lack in serendipity, and provides users with a wider range of choices and hence improves user experience in TRSs. Meanwhile, utilizing locality sensitive hash functions, the authors’ work hashes users from high-dimensional vectors to one-dimensional integers and maps similar users into the same buckets, which realizes fast nearest neighbors search in high-dimensional space and solves the extreme sparsity problem of high dimensional travel data. Furthermore, applying the hashing results to user similarity calculation, the paper greatly reduces computational complexity and improves calculation efficiency of TRSs, which reduces the system load and enables TRSs to provide effective and timely recommendations for users.

Trading Convergence Rate with Computational Budget in High Dimensional Bayesian Optimization

Scaling Bayesian optimisation (BO) to high-dimensional search spaces is a active and open research problems particularly when no assumptions are made on function structure. The main reason is that at each iteration, BO requires to find global maximisation of acquisition function, which itself is a non-convex optimization problem in the original search space. With growing dimensions, the computational budget for this maximisation gets increasingly short leading to inaccurate solution of the maximisation. This inaccuracy adversely affects both the convergence and the efficiency of BO. We propose a novel approach where the acquisition function only requires maximisation on a discrete set of low dimensional subspaces embedded in the original high-dimensional search space. Our method is free of any low dimensional structure assumption on the function unlike many recent high-dimensional BO methods. Optimising acquisition function in low dimensional subspaces allows our method to obtain accurate solutions within limited computational budget. We show that in spite of this convenience, our algorithm remains convergent. In particular, cumulative regret of our algorithm only grows sub-linearly with the number of iterations. More importantly, as evident from our regret bounds, our algorithm provides a way to trade the convergence rate with the number of subspaces used in the optimisation. Finally, when the number of subspaces is "sufficiently large", our algorithm's cumulative regret is at most O*(√TγT) as opposed to O*(√DTγT) for the GP-UCB of Srinivas et al. (2012), reducing a crucial factor √D where D being the dimensional number of input space. We perform empirical experiments to evaluate our method extensively, showing that its sample efficiency is better than the existing methods for many optimisation problems involving dimensions up to 5000.

Combined fitness–violation epsilon constraint handling for differential evolution

Over recent decades, several efficient constraint-handling methods have been proposed in the area of evolutionary computation, and the $$\varepsilon $$ constraint method is considered as a state-of-the-art method for both single and multiobjective optimization. Still, very few attempts have been made to improve this method when applied to the differential evolution algorithm. This study proposes several novel constraint-handling methods following similar ideas, where the $$\varepsilon $$ level is defined based on the current violation in the population, individual $$\varepsilon $$ levels are maintained for every constraint, and a combination of fitness and constraint violation is used for determining infeasible solutions. The proposed approaches demonstrate superior performance compared to other approaches in terms of the feasibility rate in high-dimensional search spaces, as well as convergence to global optima. The experiments are performed using the CEC’2017 constrained suite benchmark functions and a set of Economic Load Dispatch problems.

Political Optimizer: A novel socio-inspired meta-heuristic for global optimization

This paper proposes a novel global optimization algorithm called Political Optimizer (PO), inspired by the multi-phased process of politics. PO is the mathematical mapping of all the major phases of politics such as constituency allocation, party switching, election campaign, inter-party election, and parliamentary affairs. The proposed algorithm assigns each solution a dual role by logically dividing the population into political parties and constituencies, which facilitates each candidate to update its position with respect to the party leader and the constituency winner. Moreover, a novel position updating strategy called recent past-based position updating strategy (RPPUS) is introduced, which is the mathematical modeling of the learning behaviors of the politicians from the previous election. The proposed algorithm is benchmarked with 50 unimodal, multimodal, and fixed dimensional functions against 15 state of the art algorithms. We show through experiments that PO has an excellent convergence speed with good exploration capability in early iterations. Root cause of such behavior of PO is incorporation of RPPUS and logical division of the population to assign dual role to each candidate solution. Using Wilcoxon rank-sum test, PO demonstrates statistically significant performance over the other algorithms. The results show that PO outperforms all other algorithms, and consistency in performance on such a comprehensive suite of benchmark functions proves the versatility of the algorithm. Furthermore, experiments demonstrate that PO is invariant to function shifting and performs consistently in very high dimensional search spaces. Finally, the applicability on real-world applications is demonstrated by efficiently solving four engineering optimization problems.

A flexible and adaptive grid algorithm for global optimization utilizing basin hopping Monte Carlo.

Global optimization is an active area of research in atomistic simulations, and many algorithms have been proposed to date. A prominent example is basin hopping Monte Carlo, which performs a modified Metropolis Monte Carlo search to explore the potential energy surface of the system of interest. These simulations can be very demanding due to the high-dimensional configurational search space. The effective search space can be reduced by utilizing grids for the atomic positions, but at the cost of possibly biasing the results if fixed grids are employed. In this paper, we present a flexible grid algorithm for global optimization that allows us to exploit the efficiency of grids without biasing the simulation outcome. The method is general and applicable to very heterogeneous systems, such as interfaces between two materials of different crystal structures or large clusters supported at surfaces. As a benchmark case, we demonstrate its performance for the well-known global optimization problem of Lennard-Jones clusters containing up to 100 particles. Despite the simplicity of this model potential, Lennard-Jones clusters represent a challenging test case since the global minima for some "magic" numbers of particles exhibit geometries that are very different from those of clusters with only a slightly different size.

Privacy-preserving high-dimensional data publishing for classification

With increasing amounts of personal information being collected by various organizations, many privacy models have been proposed for masking the collected data so that the data can be published without releasing individual privacy. However, most existing privacy models are not applicable to high-dimensional data, because of the sparseness of high-dimensional search space. In this paper, we present our solution to release high-dimensional data for privacy preservation and classification analysis. The challenge facing us is how to reduce high dimensions from the perspective of privacy models while preserving as much information as possible for classification. Our proposed approach tackles it by using an idea of vertical partition, which is to vertically divide the raw data into different disjointed subsets of smaller dimensionality. Specifically, our partition metric considers both the correlation between attributes and the proportion of attributes in each subset. Then a generalization method based on local recoding is employed to each subset separately for achieving k-anonymity. Considering the hardness of the optimal implementation of k-anonymity, the local recoding method finds a near-optimal solution with the goal of improving efficiency. The proposed approach was evaluated using two datasets, and the experimental results showed that it outperformed two related approaches in data utility at the same privacy level.

IPDDF: an improved precision dense descriptor based flow estimation

Large displacement optical flow algorithms are generally categorised into descriptor-based matching and pixel-based matching. Descriptor-based approaches are robust to geometric variation, however they have inherent localisation precision limitation due to histogram nature. This work presents a novel method called improved precision dense descriptor flow (IPDDF). The authors introduce an additional pixel-based matching cost within an existing dense Daisy descriptor framework to improve the flow estimation precision. Pixel-based features such as pixel colour and gradient are computed on top of the original descriptor in the authors' matching cost formulation. The pixel-based cost only requires a light-weight pre-computation and can be adapted seamlessly into the matching cost formulation. The framework is built based on the Daisy Filter Flow work. In the framework, Daisy descriptor and a filter-based efficient flow inference technique, as well as a randomised fast patch match search algorithm, are adopted. Given the novel matching cost formulation, the framework enables efficiently solving dense correspondence field estimation in a high-dimensional search space, which includes scale and orientation. Experiments on various challenging image pairs demonstrate the proposed algorithm enhances flow estimation accuracy as well as generate a spatially coherent yet edge-aware flow field result efficiently.

PM-LSH

Nearest neighbor (NN) search in high-dimensional spaces is inherently computationally expensive due to the curse of dimensionality. As a well-known solution to approximate NN search, locality-sensitive hashing (LSH) is able to answer c-approximate NN ( c -ANN) queries in sublinear time with constant probability. Existing LSH methods focus mainly on building hash bucket based indexing such that the candidate points can be retrieved quickly. However, existing coarse-grained structures fail to offer accurate distance estimation for candidate points, which translates into additional computational overhead when having to examine unnecessary points. This in turn reduces the performance of query processing. In contrast, we propose a fast and accurate LSH framework, called PM-LSH, that aims to compute the c -ANN query on large- scale, high-dimensional datasets. First, we adopt a simple yet effective PM-tree to index the data points. Second, we develop a tunable confidence interval to achieve accurate distance estimation and guarantee high result quality. Third, we propose an efficient algorithm on top of the PM-tree to improve the performance of computing c -ANN queries. Extensive experiments with real-world data offer evidence that PM-LSH is capable of outperforming existing proposals with respect to both efficiency and accuracy.

Manifold learning for efficient gravitational search algorithm

Metaheuristic algorithms provide a practical tool for optimization in a high-dimensional search space. Some mimic phenomenons of nature such as swarms and flocks. Prominent one is the Gravitational Search Algorithm (GSA) inspired by Newton’s law of gravity to manipulate agents modeled as point masses in the search space. The law of gravity states that interaction forces are inversely proportional to the squared distance in the Euclidean space between two objects. In this paper we claim that when the set of solutions lies in a lower-dimensional manifold, the Euclidean distance would yield unfitted forces and bias in the results, thus causing suboptimal and slower convergence. We propose to modify the algorithm and utilize geodesic distances gained through manifold learning via diffusion maps. In addition, we incorporate elitism by storing exploration data. We show the high performance of this approach in terms of the final solution value and the rate of convergence compared to other meta-heuristic algorithms including the original GSA. In this paper we also provide a comparative analysis of the state-of-the-art optimization algorithms on a large set of standard benchmark functions.

3D Object Global Localization Using Particle Filter with Back Projection-based Sampling on Saliency

Estimating the 3D pose of a target object using particle filter has an important problem of high dimensional search space. Because the objects probably appear anywhere in the search space along the camera ray in the 3D world space, a huge number of samples covering the whole search space are required, which necessitates costly expensive computation time and iterations until convergence. For this reason, we propose a particle filter base on back projection sampling on saliency technique. We obtain the object boundaries as foreground regions using saliency segmentation based on color and depth information that is robust to complex environments. Moreover, we apply the particle filter with sampling, which is based on the back projection technique, using the concept of a relationship between 3D world space and the 2D image plane. The sampling dimension of whole samples along the camera ray can be omitted by generating the samples in the 2D image plane on saliencies before they are back projected into 3D world space using depth information. The required number of samples and iterations are drastically decreased. In addition, our method can perceive the salient regions that may be the region of the target object. Most of the samples will be predicted into these promising regions that make the algorithm converges rapidly.

Incorporating neuro-fuzzy with extended Kalman filter for simultaneous localization and mapping

Extended Kalman filter is well-known as a popular solution to the simultaneous localization and mapping problem for mobile robot platforms or vehicles. In this article, the development of a neuro-fuzzy-based adaptive extended Kalman filter technique is presented. The objective is to estimate the proper values of the R matrix at each step. We design an adaptive neuro-fuzzy extended Kalman filter to minimize the difference between the actual and theoretical covariance matrices of the innovation consequence. The parameters of the adaptive neuro-fuzzy extended Kalman filter is then trained offline using a particle swarm optimization technique. With this approach, the advantages of high-dimensional search space can be exploited and more effective training can be achieved. In the experiments, the mobile robot navigation with a number of landmarks under two benchmark situations is evaluated. The results have demonstrated that the proposed adaptive neuro-fuzzy extended Kalman filter technique provides the improvement in both performance efficiency and computational cost.

Integration of Multiple Data Sources for Gene Network Inference Using Genetic Perturbation Data.

The inference of gene networks from large-scale human genomic data is challenging due to the difficulty in identifying correct regulators for each gene in a high-dimensional search space. We present a Bayesian approach integrating external data sources with knockdown data from human cell lines to infer gene regulatory networks. In particular, we assemble multiple data sources, including gene expression data, genome-wide binding data, gene ontology, and known pathways, and use a supervised learning framework to compute prior probabilities of regulatory relationships. We show that our integrated method improves the accuracy of inferred gene networks as well as extends some previous Bayesian frameworks both in theory and applications. We apply our method to two different human cell lines, namely skin melanoma cell line A375 and lung cancer cell line A549, to illustrate the capabilities of our method. Our results show that the improvement in performance could vary from cell line to cell line and that we might need to choose different external data sources serving as prior knowledge if we hope to obtain better accuracy for different cell lines.

Mirrored Orthogonal Sampling for Covariance Matrix Adaptation Evolution Strategies.

Generating more evenly distributed samples in high dimensional search spaces is the major purpose of the recently proposed mirrored sampling technique for evolution strategies. The diversity of the mutation samples is enlarged and the convergence rate is therefore improved by the mirrored sampling. Motivated by the mirrored sampling technique, this article introduces a new derandomized sampling technique called mirrored orthogonal sampling. The performance of this new technique is both theoretically analyzed and empirically studied on the sphere function. In particular, the mirrored orthogonal sampling technique is applied to the well-known Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The resulting algorithm is experimentally tested on the well-known Black-Box Optimization Benchmark (BBOB). By comparing the results from the benchmark, mirrored orthogonal sampling is found to outperform both the standard CMA-ES and its variant using mirrored sampling.

Bioinspired nonequilibrium search for novel materials

Abstract

Large-scale parallel similarity search with Product Quantization for online multimedia services

The similarity search in high-dimensional spaces is a core operation found in several online multimedia retrieval applications. With the popularity of these applications, they are required to handle very large and increasing datasets, while keeping the response time low. This problem is worsened in the context of online applications, mostly due to the fact that load on these systems vary during the execution according to the users demands. Those variations require the application to adapt during the execution in order to minimize the response times. In this paper, we address these challenges with an efficient parallelization of the Product Quantization Approximate Nearest Neighbor Search (PQANNS) indexing. This method is capable of answering queries with a reduced memory demand and, coupled with a distributed memory parallelization proposed here, can efficiently handle very large datasets. We have also proposed mechanisms to minimize the query response times in online scenarios in which the query rates vary at run-time. For this sake, our strategies tune the parallelism configurations and task granularity during the execution. The parallelism and granularity tuning approaches (ADAPT and ADAPT+G) have shown, for instance, to reduce the query response times by a factor of 6.4× in comparison with the best static configuration of parallelism and task granularity. Further, the distributed memory execution using 128 nodes/3584 CPU cores has attained a parallel efficiency of 0.97 with a dataset of 256 billion SIFT vectors.