Free Lunch Theorem Research Articles

Evaluating Overfit and Underfit in Models of Network Community Structure

A common data mining task on networks is community detection, which seeks an unsupervised decomposition of a network into structural groups based on statistical regularities in the network's connectivity. Although many methods exist, the No Free Lunch theorem for community detection implies that each makes some kind of tradeoff, and no algorithm can be optimal on all inputs. Thus, different algorithms will over or underfit on different inputs, finding more, fewer, or just different communities than is optimal, and evaluation methods that use a metadata partition as a ground truth will produce misleading conclusions about general accuracy. Here, we present a broad evaluation of over and underfitting in community detection, comparing the behavior of 16 state-of-the-art community detection algorithms on a novel and structurally diverse corpus of 406 real-world networks. We find that (i) algorithms vary widely both in the number of communities they find and in their corresponding composition, given the same input, (ii) algorithms can be clustered into distinct high-level groups based on similarities of their outputs on real-world networks, and (iii) these differences induce wide variation in accuracy on link prediction and link description tasks. We introduce a new diagnostic for evaluating overfitting and underfitting in practice, and use it to roughly divide community detection methods into general and specialized learning algorithms. Across methods and inputs, Bayesian techniques based on the stochastic block model and a minimum description length approach to regularization represent the best general learning approach, but can be outperformed under specific circumstances. These results introduce both a theoretically principled approach to evaluate over and underfitting in models of network community structure and a realistic benchmark by which new methods may be evaluated and compared.

A Recommender System for Metaheuristic Algorithms for Continuous Optimization Based on Deep Recurrent Neural Networks

As revealed by the no free lunch theorem, no single algorithm can outperform any others on all classes of optimization problems. To tackle this issue, methods for recommending an existing algorithm for solving given problems have been proposed. However, existing recommendation methods for continuous optimization suffer from low practicability and transferability, mainly due to the difficulty in extracting features that can effectively describe the problem structure and lack of data for training a recommendation model. This work proposes a generic recommender system to address the above two challenges. First, a novel method is proposed to represent an analytic objective function of a continuous optimization problem as a tree, which is directly used as the features of the problem. For black-box optimization problems whose objective function is unknown, a symbolic regressor is adopted to estimate the tree structure. Second, a large number of benchmark problems are randomly created based on the proposed tree representation, providing an abundant amount of training data with various levels of difficulty. By employing a deep recurrent neural network, a recommendation model is trained to recommend a most suitable metaheuristic algorithm for white- or black-box optimization, making a significant step forward towards fully automated algorithm recommendation for continuous optimization. Experimental results on 100 000 benchmark problems show that the proposed recommendation model achieves considerably better performance than existing ones, and exhibits high transferability to real-world problems.

Settings-Free Hybrid Metaheuristic General Optimization Methods

Several population-based metaheuristic optimization algorithms have been proposed in the last decades, none of which are able either to outperform all existing algorithms or to solve all optimization problems according to the No Free Lunch (NFL) theorem. Many of these algorithms behave effectively, under a correct setting of the control parameter(s), when solving different engineering problems. The optimization behavior of these algorithms is boosted by applying various strategies, which include the hybridization technique and the use of chaotic maps instead of the pseudo-random number generators (PRNGs). The hybrid algorithms are suitable for a large number of engineering applications in which they behave more effectively than the thoroughbred optimization algorithms. However, they increase the difficulty of correctly setting control parameters, and sometimes they are designed to solve particular problems. This paper presents three hybridizations dubbed HYBPOP, HYBSUBPOP, and HYBIND of up to seven algorithms free of control parameters. Each hybrid proposal uses a different strategy to switch the algorithm charged with generating each new individual. These algorithms are Jaya, sine cosine algorithm (SCA), Rao’s algorithms, teaching-learning-based optimization (TLBO), and chaotic Jaya. The experimental results show that the proposed algorithms perform better than the original algorithms, which implies the optimal use of these algorithms according to the problem to be solved. One more advantage of the hybrid algorithms is that no prior process of control parameter tuning is needed.

Who Should See the Patient? On Discretionary Patient-Provider Assignments in Hospitals

Traditional principles in operations management and organizational theory suggest that standardizing assignments of tasks to individuals can have substantial benefits and boost performance. However, in various organizations including hospitals, individuals are not forced to follow recommended assignments, and thus, deviations from routine task assignments are often observed. This is due to the conventional wisdom that professionals should be given the opportunity to use their discretionary judgement and deviate from routine assignments when they perceive it to be advantageous. It is unclear, however, whether and when this conventional wisdom is true. We use evidence on the assignments of generalist and specialists to patients in a children’s hospital, and generate insights into whether and when hospital administrators should allow providers to use their discretion to deviate from routine assignments. To perform our analyses, we identify 73 top medical diagnoses and use detailed patient-level electronic medical record (EMR) data of more than 4,700 hospitalizations. In parallel, we conduct a carefully-designed survey of physicians and utilize it to identify the routine provider type that should have been assigned to each patient. Using these two sources of data, we examine the consequence of discretionary deviations from routine provider assignments on three sets of performance measures: operational efficiency (measured by length of stay), quality of care (measured by 30-day readmission rates and rate of adverse events), and cost (measured by total charges). Taken together, our findings suggest that allowing providers to use their discretion in order to deviate from routine assignments is beneficial for task types (patients’ diagnosis in our setting) that are either (a) well-defined (improving operational efficiency and costs), or (b) require high contact (improving costs and adverse events, though at the expense of lower operational efficiency). For other task types (e.g., highly complex or resource-intensive tasks), we find that routines should be enforced: deviations are either detrimental or yield no tangible benefits. Our findings also establish a no free lunch theorem: while for some task types deviations are beneficial regarding some performance measures, they simultaneously degrade performance in terms of other dimensions. Finally, by comparing deviations during weekdays and weekends, early shifts and late shifts, and high congestion and low congestion periods, our results shed light on some environmental conditions under which discretionary deviations are invoked more in practice.

A holistic review on artificial intelligence techniques for well placement optimization problem

Well placement optimization is one of the major challenging factors in the field development process of oil and gas industry. The objective function of well placement optimization is considered as high dimensional, discontinuous and multi-model. Over the last decade, both gradient-based and gradient-free optimization methods have been implemented to tackle this problem. Nature-inspired gradient-free optimization algorithms like particle swarm optimization, genetic algorithm, covariance matrix adaptation evolution strategy and differential evolution have been utilized in this area. These optimization techniques are implemented as stand-alone or as hybrid form to maximize the economic factors. In this paper, several nature-inspired metaheuristic optimization techniques and their application to maximize the economic factors are reviewed. Newly developed optimization algorithms are very efficient and favorable when compared to other established optimization algorithms and in all cases, it has been noticed that hybridization of two or more algorithms works better than stand-alone algorithms. Furthermore, none of the single optimization techniques can be established as being universally superior which aligns with no free lunch theorem. For future endeavor, combining optimization methods and exploiting multiple optimization processes for faster convergence and developing efficient proxy model is expected.

An empirical demonstration of the no free lunch theorem

An empirical demonstration of the no free lunch theorem

Combinatorial Test Suites Generation Strategy Utilizing the Whale Optimization Algorithm

The potentially many software system input combinations make exhaustive testing practically impossible. To address this issue, combinatorial t-way testing (where t indicates the interaction strength, i.e. the number of interacting parameters (input)) was adopted to minimize the number of cases for testing. Complimentary to existing testing techniques (e.g. boundary value, equivalence partitioning, cause and effect graphing), combinatorial testing helps to detect faults caused by the faulty interaction between input parameters. In the last 15 years, applications of meta-heuristics as the backbone of t-way test suite generation have shown promising results (e.g. Particle Swarm Optimization, Cuckoo Search, Flower Pollination Algorithm, and Hyper-Heuristics (HHH), to name a few). Supporting the No Free Lunch theorem, as well as potentially offering new insights into the whole process of t-way generation, this article proposes a new strategy with constraint support based on the Whale Optimization Algorithm (WOA). Our work is the first attempt to adopt the WOA as part of a search-based software engineering (SBSE) initiative for t-way test suite generation with constraint support. The experimental results of the test-suite generation indicate that WOA produces competitive outcomes compared to some selected single-based and population-based meta-heuristic algorithms.

The Issue of Bias. The Framing Powers of Ml

From a computer science perspective, bias may refer to the productive bias that enables ML, both at the level of picking the training set and at the level of training the algorithms. It reminds one of David Wolpert’s ‘no free lunch theorem’, if not of Humean scepticism or Gadamer’s acknowledgment of constitutive presumptions. This e.g. relates to the trade-off between the size of a training set, its relevance, the types of algorithms used, and the accuracy and/or speed of the results. From a societal perspective, bias may refer to unfair treatment or even unlawful discrimination. It is crucial to distinguish inherent computational bias from the unwarranted impact of unfair or wrongful bias, while teasing out where they meet and how they interact. This includes an inquiry into the ethical assessments of ML bias, based on the fact that ML applications are reconfiguring the ‘choice architectures’ of our online and offline environments. Finally, I will briefly argue that ethics will not do when confronting the framing powers of machine bias, highlighting the need to bring the design choice that determine these framing powers under the Rule of Law.

A survey of regularization strategies for deep models

The most critical concern in machine learning is how to make an algorithm that performs well both on training data and new data. No free lunch theorem implies that each specific task needs its own tailored machine learning algorithm to be designed. A set of strategies and preferences are built into learning machines to tune them for the problem at hand. These strategies and preferences, with the core concern of generalization improvement, are collectively known as regularization. In deep learning, because of a considerable number of parameters, a great many forms of regularization methods are available to the deep learning community. Developing more effective regularization strategies has been the subject of significant research efforts in recent years. However, it is difficult for developers to choose the most suitable strategy for their problem at hand, because there is no comparative study regarding the performance of different strategies. In this paper, at the first step, the most effective regularization methods and their variants are presented and analyzed in a systematic approach. At the second step, comparative research on regularization techniques is presented in which the testing errors and computational costs are evaluated in a convolutional neural network, using CIFAR-10 ( https://www.cs.toronto.edu/~kriz/cifar.html ) dataset. In the end, different regularization methods are compared in terms of accuracy of the network, the number of epochs for the network to be trained and the number of operations per input sample. Also, the results are discussed and interpreted based on the employed strategy. The experiment results showed that weight decay and data augmentation regularizations have little computational side effects so can be used in most applications. In the case of enough computational resources, Dropout family methods are rational to be used. Moreover, in the case of abundant computational resources, batch normalization family and ensemble methods are reasonable strategies to be employed.

A Sparse-Modeling Based Approach for Class Specific Feature Selection.

In this work, we propose a novel Feature Selection framework called Sparse-Modeling Based Approach for Class Specific Feature Selection (SMBA-CSFS), that simultaneously exploits the idea of Sparse Modeling and Class-Specific Feature Selection. Feature selection plays a key role in several fields (e.g., computational biology), making it possible to treat models with fewer variables which, in turn, are easier to explain, by providing valuable insights on the importance of their role, and likely speeding up the experimental validation. Unfortunately, also corroborated by the no free lunch theorems, none of the approaches in literature is the most apt to detect the optimal feature subset for building a final model, thus it still represents a challenge. The proposed feature selection procedure conceives a two-step approach: (a) a sparse modeling-based learning technique is first used to find the best subset of features, for each class of a training set; (b) the discovered feature subsets are then fed to a class-specific feature selection scheme, in order to assess the effectiveness of the selected features in classification tasks. To this end, an ensemble of classifiers is built, where each classifier is trained on its own feature subset discovered in the previous phase, and a proper decision rule is adopted to compute the ensemble responses. In order to evaluate the performance of the proposed method, extensive experiments have been performed on publicly available datasets, in particular belonging to the computational biology field where feature selection is indispensable: the acute lymphoblastic leukemia and acute myeloid leukemia, the human carcinomas, the human lung carcinomas, the diffuse large B-cell lymphoma, and the malignant glioma. SMBA-CSFS is able to identify/retrieve the most representative features that maximize the classification accuracy. With top 20 and 80 features, SMBA-CSFS exhibits a promising performance when compared to its competitors from literature, on all considered datasets, especially those with a higher number of features. Experiments show that the proposed approach may outperform the state-of-the-art methods when the number of features is high. For this reason, the introduced approach proposes itself for selection and classification of data with a large number of features and classes.

A Generalized Parallel Quantum Inspired Evolutionary Algorithm Framework for Hard Subset Selection Problems

Quantum-inspired evolutionary algorithms (QIEAs) like all evolutionary algorithms (EAs) perform well on many problems but cannot perform equally better than random for all problems due to the No Free Lunch theorem. However, a framework providing near-optimal solutions on reasonably hard instances of a large variety of problems is feasible. It has an effective general strategy for easy incorporation of domain information along with effective control on the randomness in the search process to balance the exploration and exploitation. Moreover, its effective parallel implementation is desired in the current age. Such a Generalized Parallel QIEA framework designed for the solution of Subset Selection Problems is presented here. The computational performance results demonstrate its effectiveness in the solution of different large-sized hard SSPs like the Difficult Knapsack Problem, the Quadratic Knapsack Problem and the Multiple Knapsack problem. This is the first such a generalized framework and is a major step towards creating an adaptive search framework for combinatorial optimization problems.

No Free Lunch Theorem for concept drift detection in streaming data classification: A review

AbstractMany real‐world data mining applications have to deal with unlabeled streaming data. They are unlabeled because the sheer volume of the stream makes it impractical to label a significant portion of the data. The data streams can evolve over time and these changes are called concept drifts. Concept drifts have different characteristics, which can be used to categorize them into different types. A trade‐off between performance and cost exists among many concept drift detection approaches. On the one hand, high accuracy detection approach usually requires labeled data, possibly involving high cost for labeling. On the other hand, a variety of methods have been devoted to the topic of concept drift detection with unlabeled data, but these approaches often are most suited for only a subset of the concept drift types. The objective of this survey is to present these methods, categorize them and give recommendations of usage based on their behaviors under different types of concept drift.This article is categorized under: Fundamental Concepts of Data and Knowledge > Data Concepts Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Explainable AI > Classification

A Metahierarchical Rule Decision System to Design Robust Fuzzy Classifiers Based on Data Complexity

There is a wide variety of studies that propose different classifiers to solve a large amount of problems in distinct classification scenarios. The no free lunch theorem states that if we use a big enough set of varied problems, all classifiers would be equivalent in performance. From another point of view, the performance of the classifiers is dependant of the scope and properties of the datasets. In this sense, new proposals on the topic often focus on a given context, aiming at improving the related state-of-the-art approaches. Data complexity metrics have been traditionally used to determine the inner characteristics of datasets. This way, researchers are able to categorize the problems in different scenarios. Then, this taxonomy can be applied to determine inner characteristics of the datasets in order to determine intervals of good and bad behavior for a given classifier. In this paper, we will take advantage of the data complexity metrics in order to design a fuzzy metaclassifier. The final goal is to create decision rules based on the inner characteristics of the data to apply a different version of the fuzzy classifier for a given problem. To do so, we will make use of the FARC-HD classifier, an evolutionary fuzzy system that has led to different extensions in the specialized literature. Experimental results show the goodness of this novel approach as it is able to outperform all versions of FARC-HD on a wide set of problems, and obtain competitive results (in terms of performance and interpretability) versus two selected state-of-the-art rule-based classification system, C4.5 and FURIA.

When and why metaheuristics researchers can ignore “No Free Lunch” theorems

The No Free Lunch (NFL) theorem for search and optimisation states that averaged across all possible objective functions on a fixed search space, all search algorithms perform equally well. Several refined versions of the theorem find a similar outcome when averaging across smaller sets of functions. This paper argues that NFL results continue to be misunderstood by many researchers, and addresses this issue in several ways. Existing arguments against real-world implications of NFL results are collected and re-stated for accessibility, and new ones are added. Specific misunderstandings extant in the literature are identified, with speculation as to how they may have arisen. This paper presents an argument against a common paraphrase of NFL findings -- that algorithms must be specialised to problem domains in order to do well -- after problematising the usually undefined term "domain". It provides novel concrete counter-examples illustrating cases where NFL theorems do not apply. In conclusion it offers a novel view of the real meaning of NFL, incorporating the anthropic principle and justifying the position that in many common situations researchers can ignore NFL.

THE NO FREE LUNCH THEOREM: BAD NEWS FOR (WHITE'S ACCOUNT OF) THE PROBLEM OF INDUCTION

AbstractWhite (2015) proposes an a priori justification of the reliability of inductive prediction methods based on his thesis of induction-friendliness. It asserts that there are by far more induction-friendly event sequences than induction-unfriendly event sequences. In this paper I contrast White's thesis with the famous no free lunch (NFL) theorem. I explain two versions of this theorem, the strong NFL theorem applying to binary and the weak NFL theorem applying to real-valued predictions. I show that both versions refute the thesis of induction-friendliness. In the conclusion I argue that an a priori justification of the reliability of induction based on a uniform probability distribution over possible event sequences is impossible. In the outlook I consider two alternative approaches: (i) justification externalism and (ii) optimality justifications.

Solving non-convex economic load dispatch problem via artificial cooperative search algorithm

According to no free lunch (NFL) theorem, a metaheuristic optimization method is superior to other metaheuristic optimization methods when it has focused on specific class of optimization problems. Thus, this paper focuses on developing artificial cooperative search (ACS) optimization algorithm to solve economic dispatch (ED) problems more precisely with less complexity than other metaheuristic optimization methods. ACS is a recently developed two population search algorithm based on coevolution process with high probability of finding optimal solution in complex and non-convex optimization problems. This merit is provided by balancing exploration of the problem's search space and exploitation of better results through use of two advanced evolutionary operators and only one control parameter. The constraint handling strategy of the proposed method for solving economic power load dispatch problems is to generate and work with feasible solutions along all the optimization iterations without any mismatch between the total amount of electric power generation and electricity demand plus network transmission loss. Unlike the penalty method, this strategy is unaffected by parameter setting of applied optimization method that complicates its applicability for solving economic power load dispatch problems. The feasibility of ACS for solving ED problem is tested on different lossy non-convex test systems of various sizes and complexities. The practical aspects such as satisfaction of power demand constraint, generation limits of generators and value-point loading effect are undertaken to solve ED problem in medium to relatively large-scale electric power systems. Obtained results confirm the ACS's capability in converging to a better solution highly robust within the reasonable computational time in all independent trials; all these as compared with other optimization algorithms reported in the literature for solving lossy non-convex ED problems. The results are analyzed statistically in terms of solution quality and computational efficiency. The statistical analyses reveal that ACS is a potential method to solve economic power load dispatch problems as it provides higher quality solution in comparison with other optimization algorithms.

On the Economic Significance of Stock Market Prediction and the No Free Lunch Theorem

Forecasting of stock market returns is a challenging research activity that is now expanding with the availability of new data sources, markets, financial instruments, and algorithms. At its core, the predictability of prices still raises important questions. Here, we discuss the economic significance of the prediction accuracy. To develop this question, we collect the daily series prices of almost half of the publicly traded companies around the world over a period of ten years and formulate some trading strategies based on their prediction. Proper visualization of these data together with the use of the No Free Lunch theoretical framework gives some unexpected results that show how the a priori less accurate algorithms and inefficient strategies can offer better results than the a priori best alternatives in some particular subsets of data that have a clear interpretation in terms of economic sectors and regions.

Ameliorated grey wolf optimization for economic load dispatch problem

Owing to no free lunch theorem, meta-heuristic search algorithms, behaviour and performance are different for solving specific class of optimization problems than another. Further, search algorithms are conceptualized of exploration and exploitation aspects. Ameliorated grey wolf optimization algorithm is proposed to solve economic power load dispatch problem that synergizes between exploration and exploitation aspects. Ameliorated grey wolf optimization procedure coordinates the behaviour of grey wolves, random exploratory search, local random search and opposition learning heuristics. Grey wolf optimization algorithm is considered to perform global search and random exploratory search applied to perform a search in the neighbourhood of already visited search locations. To maintain good solutions and diversity among solutions, opposition based learning has been implemented. The practical aspects i.e. value-point loading effect, avoidance of prohibited operating zones and ramp rate limits, generation limits of generators and satisfaction of power demand constraint are undertaken to solve economic power load dispatch problem. Heuristic search is exploited to handle equality constraints. The proposed algorithm is validated on standard benchmark functions and medium to large electric power systems. Results reveal that proposed technique is a potential method to solve economic load dispatch problems as it competes with recent algorithms undertaken for comparison.

Machine Learning for Structured Finance

Machine learning and artificial intelligence have evolved beyond simple hype and have integrated themselves in business and in popular conversation as an increasing number of smart applications profoundly transform the way we work and live. This article defines machine learning in terms of potential benefits and pitfalls for a nontechnical audience, and gives examples of popular and powerful machine learning algorithms: k-means clustering, principal component analysis, and artificial neural networks. Three important philosophical challenges of machine learning are introduced: the no free lunch theorem, the curse of dimensionality, and the bias–variance trade-off.

Ensemble strategies for population-based optimization algorithms – A survey

In population-based optimization algorithms (POAs), given an optimization problem, the quality of the solutions depends heavily on the selection of algorithms, strategies and associated parameter combinations, constraint handling method, local search method, surrogate model, niching method, etc. In the literature, there exist several alternatives corresponding to each aspect of configuring a population-based algorithm such as one-point/two-points/uniform crossover operators, tournament/ranking/stochastic uniform sampling selection methods, Gaussian/Levy/Cauchy mutation operators, clearing/crowding/sharing based niching algorithms, adaptive penalty/epsilon/superiority of feasible constraint handling approaches, associated parameter values and so on. In POA literature, No Free Lunch (NFL) theorem has been well-documented and therefore, to effectively solve a given optimization problem, an appropriate configuration is necessary. But, the trial and error approach for the appropriate configuration may be impractical because at different stages of evolution, the most appropriate configurations could be different depending on the characteristics of the current search region for a given problem. Recently, the concept of incorporating ensemble strategies into POAs has become popular so that the process of configuring an optimization algorithm can benefit from both the availability of diverse approaches at different stages and alleviate the computationally intensive offline tuning. In addition, algorithmic components of different advantages could support one another during the optimization process, such that the ensemble of them could potentially result in a versatile POA. This paper provides a survey on the use of ensemble strategies in POAs. In addition, we also provide an overview of similar methods in the literature such as hyper-heuristics, island models, adaptive operator selection, etc. and compare them with the ensemble strategies in the context of POAs.