Black-box Methods Research Articles

Black-Box String Test Case Generation through a Multi-Objective Optimization

String test cases are required by many real-world applications to identify defects and security risks. Random Testing (RT) is a low cost and easy to implement testing approach to generate strings. However, its effectiveness is not satisfactory. In this research, black-box string test case generation methods are investigated. Two objective functions are introduced to produce effective test cases. The diversity of the test cases is the first objective, where it can be measured through string distance functions. The second objective is guiding the string length distribution into a Benford distribution based on the hypothesis that the population of strings is right-skewed within its range. When both objectives are applied via a multi-objective optimization algorithm, superior string test sets are produced. An empirical study is performed with several real-world programs indicating that the generated string test cases outperform test cases generated by other methods.

Blackbox and derivative-free optimization: theory, algorithms and applications

Blackbox optimization refers to problems where the structure of the objective and constraint functions cannot be exploited. This is often the case when their evaluation requires the execution of a (usually time-consuming) simulation using computational models that are typically inaccessible by the user. The term Derivative-Free Optimization refers to the use of algorithms that utilize only function values because their partial derivatives are either not defined or not available; gradient approximations may sometimes be obtained, but the amount of work required to ensure they are dependable may not be worth the effort. Both blackbox and derivative-free optimization have attracted significant, and still increasing, interest from researchers over the last decade. Thus, we felt that it was time to dedicate a special issue of OPTE to this topic. Blackbox and derivative-free optimization methods are often the only realistic and practical tools available to engineers working on simulation-based design. It is obvious that if the design optimization problem at hand allows an evaluation or reliable approximation of the gradients, then efficient gradient-based methods should be used. Blackbox and derivative-free algorithms are not competitors of gradient-based methods; they are a fallback when gradient-based algorithms cannot be used. The design engineering community is increasingly becoming aware that

Turning statistical physics models into materials design engines

Despite the success statistical physics has enjoyed at predicting the properties of materials for given parameters, the inverse problem, identifying which material parameters produce given, desired properties, is only beginning to be addressed. Recently, several methods have emerged across disciplines that draw upon optimization and simulation to create computer programs that tailor material responses to specified behaviors. However, so far the methods developed either involve black-box techniques, in which the optimizer operates without explicit knowledge of the material's configuration space, or require carefully tuned algorithms with applicability limited to a narrow subclass of materials. Here we introduce a formalism that can generate optimizers automatically by extending statistical mechanics into the realm of design. The strength of this approach lies in its capability to transform statistical models that describe materials into optimizers to tailor them. By comparing against standard black-box optimization methods, we demonstrate how optimizers generated by this formalism can be faster and more effective, while remaining straightforward to implement. The scope of our approach includes possibilities for solving a variety of complex optimization and design problems concerning materials both in and out of equilibrium.

Web browsing automation for applications quality control

Context: Quality control comprises the set of activities aimed to evaluate that software meets its specification and delivers the functionality expected by the consumers. These activities are often removed in the development process and, as a result, the final software product usually lacks quality. Objective: We propose a set of techniques to automate the quality control for web applications from the client-side, guiding the process by functional and nonfunctional requirements (performance, security, compatibility, usability and accessibility). Method: The first step to achieve automation is to define the structure of the web navigation. Existing software artifacts in the phase of analysis and design are reused. Then, the independent paths of navigation are found, and each path is traversed automatically using real browsers while different kinds of assessments are carried out. Results: The processes and methods proposed in this paper have been implemented by means of a reference architecture and open source tools. A laboratory experiment and an industrial case study have been performed in order to validate the proposal. Conclusion: The definition of navigation paths is a rich approach to model web applications. Grey-box (black-box and white-box) methods have been proved to be very valuable for web assessment. The Chinese Postman Problem (CPP) is an optimal way to find the independent paths in a web navigation modeled as a directed graph.

Bayesian optimization for learning gaits under uncertainty

Designing gaits and corresponding control policies is a key challenge in robot locomotion. Even with a viable controller parametrization, finding near-optimal parameters can be daunting. Typically, this kind of parameter optimization requires specific expert knowledge and extensive robot experiments. Automatic black-box gait optimization methods greatly reduce the need for human expertise and time-consuming design processes. Many different approaches for automatic gait optimization have been suggested to date. However, no extensive comparison among them has yet been performed. In this article, we thoroughly discuss multiple automatic optimization methods in the context of gait optimization. We extensively evaluate Bayesian optimization, a model-based approach to black-box optimization under uncertainty, on both simulated problems and real robots. This evaluation demonstrates that Bayesian optimization is particularly suited for robotic applications, where it is crucial to find a good set of gait parameters in a small number of experiments.

Modeling nonlinear dynamic biological systems with human-readable fuzzy rules optimized by convergent heterogeneous particle swarm

It is an important issue to model the dynamic biological networks from their time-course response datasets, which is critical to understand the system behaviors. This task includes two sub-tasks: network structure identification and associated parameters estimation. In most existing methods, the two sub-tasks are dealt with step by step, which may result in inconsistence between them and hence inaccuracy of the final model. Another challenge is how to transparently understand the derived model, which cannot be achieved by the traditional black-box methods. A human readable fuzzy rule-based model, denoted as MoPath, is developed for the identification of both structure topology and associated parameters, simultaneously, of a biological network within an optimization framework. MoPath encodes the fuzzy rules into particles of heterogeneous particle swarm optimization (CHPSO) algorithm to generate the optimal model. Theoretically, we demonstrate that the cooperation in CHPSO can maintain a balance between exploration and exploitation to guarantee the particles converge to stable points, which is greatly helpful for finding the optimal model consisting of both the network topology and parameters. We demonstrate MoPath on two dynamic biological networks, and successfully generate a few human readable rules that can well represent the network with high accuracy and good robustness.

Black-box modeling of residential HVAC system and comparison of gray-box and black-box modeling methods

In this article, black-box models of the residential heating, ventilation and air conditioning (HVAC) system are developed. The data of the input and output of the system is measured and the models of the energy recovery ventilator (ERV), air handling unit (AHU), buffer tank (BT), radiant floor heating (RFH) and ground source heat pump (GSHP) are developed using the system identification techniques in Matlab®. The developed models include models based on multiple-input and multiple-output (MIMO) artificial neural network (ANN), transfer function (TF), process, state-space (SS) and autoregressive exogenous (ARX) ones of each HVAC subsystem (ERV, AHU, BT and RFH). The gray-box models of the same HVAC subsystems were developed in [1] which are also compared with the black-box models developed in this article. The models were compared visually and analytically. Ranks of the models were calculated based on their relative performance. It was found that the ANN outperforms the other modeling methods followed by the ARX, TF, SS, process and gray-box models respectively.

Perancangan Aplikasi Permainan Let’s Say dengan Interaksi Pengenalan Ucapan

Computer games has been widespread. Currently, almost all devices that have a screen can play the game with the experience of interactivity to entertain the public. The game does not always have a negative effect. Game application that has a learning function will be useful and easily accepted by society. This study was conducted to determine how computer games can be a fun medium of information and learning by making use of speech recognition interaction. The application development of this "Let's Say" game is done by using the method of multimedia development consisting stages of concept, design, material collection, implementation, testing, and distribution. The design of the application is done using UML modeling and ERD. Application development is done using Visual Basic.NET programming language, WPF system interface, and SQLite database. Application utilizes Windows Speech Recognition as a speech recognition engine. Functional testing of the application is done using black box methods. Applications testing to users is done through a questionnaire. The test results showed that the application works well according to functions that has been specified. Application testing to users indicate that the application is informative and easy to use with a view that is quite interesting. Tests also showed that the use of speech recognition has been going well and helpful.

Combining macula clinical signs and patient characteristics for age-related macular degeneration diagnosis: a machine learning approach.

BackgroundTo investigate machine learning methods, ranging from simpler interpretable techniques to complex (non-linear) “black-box” approaches, for automated diagnosis of Age-related Macular Degeneration (AMD).MethodsData from healthy subjects and patients diagnosed with AMD or other retinal diseases were collected during routine visits via an Electronic Health Record (EHR) system. Patients’ attributes included demographics and, for each eye, presence/absence of major AMD-related clinical signs (soft drusen, retinal pigment epitelium, defects/pigment mottling, depigmentation area, subretinal haemorrhage, subretinal fluid, macula thickness, macular scar, subretinal fibrosis). Interpretable techniques known as white box methods including logistic regression and decision trees as well as less interpreitable techniques known as black box methods, such as support vector machines (SVM), random forests and AdaBoost, were used to develop models (trained and validated on unseen data) to diagnose AMD. The gold standard was confirmed diagnosis of AMD by physicians. Sensitivity, specificity and area under the receiver operating characteristic (AUC) were used to assess performance.ResultsStudy population included 487 patients (912 eyes). In terms of AUC, random forests, logistic regression and adaboost showed a mean performance of (0.92), followed by SVM and decision trees (0.90). All machine learning models identified soft drusen and age as the most discriminating variables in clinicians’ decision pathways to diagnose AMD.ConclusionsBoth black-box and white box methods performed well in identifying diagnoses of AMD and their decision pathways. Machine learning models developed through the proposed approach, relying on clinical signs identified by retinal specialists, could be embedded into EHR to provide physicians with real time (interpretable) support.Electronic supplementary materialThe online version of this article (doi:10.1186/1471-2415-15-10) contains supplementary material, which is available to authorized users.

Using Fuzzy Logic Techniques for Assertion-Based Software Testing Metrics.

Software testing is a very labor intensive and costly task. Therefore, many software testing techniques to automate the process of software testing have been reported in the literature. Assertion-Based automated software testing has been shown to be effective in detecting program faults as compared to traditional black-box and white-box software testing methods. However, the applicability of this approach in the presence of large numbers of assertions may be very costly. Therefore, software developers need assistance while making decision to apply Assertion-Based testing in order for them to get the benefits of this approach at an acceptable level of costs. In this paper, we present an Assertion-Based testing metrics technique that is based on fuzzy logic. The main goal of the proposed technique is to enhance the performance of Assertion-Based software testing in the presence of large numbers of assertions. To evaluate the proposed technique, an experimental study was performed in which the proposed technique is applied on programs with assertions. The result of this experiment shows that the effectiveness and performance of Assertion-Based software testing have improved when applying the proposed testing metrics technique.

Interactive chemical reactivity exploration.

Elucidating chemical reactivity in complex molecular assemblies of a few hundred atoms is, despite the remarkable progress in quantum chemistry, still a major challenge. Black-box search methods to find intermediates and transition-state structures might fail in such situations because of the high-dimensionality of the potential energy surface. Here, we propose the concept of interactive chemical reactivity exploration to effectively introduce the chemist's intuition into the search process. We employ a haptic pointer device with force feedback to allow the operator the direct manipulation of structures in three dimensions along with simultaneous perception of the quantum mechanical response upon structure modification as forces. We elaborate on the details of how such an interactive exploration should proceed and which technical difficulties need to be overcome. All reactivity-exploration concepts developed for this purpose have been implemented in the samson programming environment.

Driving as a human: a track learning based adaptable architecture for a car racing controller

We present the evolution and current state of the Mr. Racer car racing controller that excelled at the corresponding TORCS competitions of the last years. Although several heuristics and black-box optimization methods are employed, the basic idea of the controller architecture has been to take over many of the mechanisms human racing drivers apply. They learn the track geometry, plan ahead, and wherever necessary, adapt their plans to the current circumstances quickly. Mr. Racer consists of several modules that have partly been adapted and optimized separately, where the final tuning is usually done with respect to a certain racing track during the warmup phase of the TORCS competitions. We also undertake an experimental evaluation that investigates how the controller profits from adding some of the modules to a basic configuration and which modules are most important for reaching the best possible performance.

The combined use of wavelet transform and black box models in reservoir inflow modeling

Abstract In the study presented, different hybrid model approaches are proposed for reservoir inflow modeling from the meteorological data (monthly precipitation, one-month-ahead precipitation and monthly mean temperature data) by the combined use of discrete wavelet transform (DWT) and different black box techniques. Multiple linear regression (MLR), feed forward neural networks (FFNN) and least square support vector machines (LSSVM) were considered as the black box methods. In the modeling strategy, meteorological input data were decomposed into wavelet sub-time series at three resolution levels and ineffective sub-time series were eliminated by Mallows’ Cp based all possible regression method. As a result of all possible regression analyses, 2-months mode of time series of monthly temperature (D1_Tt), 8-months mode of time series (D3_Tt) of monthly temperature and approximation mode of time series (A3_Tt) of monthly temperature were eliminated. Remained effective sub-time series were used as the inputs of MLR, FFNN and LSSVM. When the performances of the training and testing periods were compared, it was observed that the DWTFFNN conjunction model has better results in terms of mean square errors (MSE) and determination coefficients (R2) statistics. The discrete wavelet transform approach also increased the accuracy of multiple linear regression and least squares support vector machines.

Statistical Learning Algorithm for in situ and invasive breast carcinoma segmentation

Dynamic Contrast Enhanced MRI (DCE-MRI) has proven to be a highly sensitive imaging modality in diagnosing breast cancers. However, analyzing the DCE-MRI is time-consuming and prone to errors due to the large volume of data. Mathematical models to quantify contrast perfusion, such as the black box methods and pharmacokinetic analysis, are inaccurate, sensitive to noise and depend on a large number of external factors such as imaging parameters, patient physiology, arterial input function, and fitting algorithms, leading to inaccurate diagnosis. In this paper, we have developed a novel Statistical Learning Algorithm for Tumor Segmentation (SLATS) based on Hidden Markov Models to auto-segment regions of angiogenesis, corresponding to tumor. The SLATS algorithm has been trained to identify voxels belonging to the tumor class using the time–intensity curve, first and second derivatives of the intensity curves (“velocity” and “acceleration” respectively) and a composite vector consisting of a concatenation of the intensity, velocity and acceleration vectors. The results of SLATS trained for the four vectors has been shown for 22 Invasive Ductal Carcinoma (IDC) and 19 Ductal Carcinoma In Situ (DCIS) cases. The SLATS trained for the velocity tuple shows the best performance in delineating the tumors when compared with the segmentation performed by an expert radiologist and the output of a commercially available software, CADstream.

Comparison of fast boundary element methods on parametric surfaces

We compare fast black-box boundary element methods on parametric surfaces in R3. These are the adaptive cross approximation, the multipole method based on interpolation, and the wavelet Galerkin scheme. The surface representation by a piecewise smooth parameterization is in contrast to the common approximation of surfaces by panels. Nonetheless, parametric surface representations are easily accessible from computer aided design (CAD) and are recently topic of the studies in isogeometric analysis. Especially, we can apply two-dimensional interpolation in the multipole method. A main feature of this approach is that the cluster bases and the respective moment matrices are independent of the geometry. This results in a superior compression of the far-field compared to other cluster methods.

Performance of first-order methods for smooth convex minimization: a novel approach

We introduce a novel approach for analyzing the worst-case performance of first-order black-box optimization methods. We focus on smooth unconstrained convex minimization over the Euclidean space. Our approach relies on the observation that by definition, the worst-case behavior of a black-box optimization method is by itself an optimization problem, which we call the performance estimation problem (PEP). We formulate and analyze the PEP for two classes of first-order algorithms. We first apply this approach on the classical gradient method and derive a new and tight analytical bound on its performance. We then consider a broader class of first-order black-box methods, which among others, include the so-called heavy-ball method and the fast gradient schemes. We show that for this broader class, it is possible to derive new bounds on the performance of these methods by solving an adequately relaxed convex semidefinite PEP. Finally, we show an efficient procedure for finding optimal step sizes which results in a first-order black-box method that achieves best worst-case performance.

FRKAS: Knowledge Acquisition Using a Fuzzy Rule Base Approach to Insight of DNA-Binding Domains/Proteins

Numerous prediction methods of DNA-binding domains/proteins were proposed by identifying informative features and designing effective classifiers. These researches reveal that the DNA-protein binding mechanism is complicated and existing accurate predictors such as support vector machine (SVM) with position specific scoring matrices (PSSMs) are regarded as black-box methods which are not easily interpretable for biologists. In this study, we propose an ensemble fuzzy rule base classifier consisting of a set of interpretable fuzzy rule classifiers (iFRCs) using informative physicochemical properties as features. In designing iFRCs, feature selection, membership function design, and fuzzy rule base generation are all simultaneously optimized using an intelligent genetic algorithm (IGA). IGA maximizes prediction accuracy, minimizes the number of features selected, and minimizes the number of fuzzy rules to generate an accurate and concise fuzzy rule base. Benchmark datasets of DNA-binding domains are used to evaluate the proposed ensemble classifier of 30 iFRCs. Each iFRC has a mean test accuracy of 77.46%, and the ensemble classifier has a test accuracy of 83.33%, where the method of SVM with PSSMs has the accuracy of 82.81%. The physicochemical properties of the first two ranks according to their contribution are positive charge and Van Der Waals volume. Charge complementarity between protein and DNA is thought to be important in the first step of recognition between protein and DNA. The amino acid residues of binding peptides have larger Van Der Waals volumes and positive charges than those of non-binding ones. The proposed knowledge acquisition method by establishing a fuzzy rule-based classifier can also be applicable to predict and analyze other protein functions from sequences.

Classification via Bayesian Nonparametric Learning of Affine Subspaces

It has become common for datasets to contain large numbers of variables in studies conducted in areas such as genetics, machine vision, image analysis, and many others. When analyzing such data, parametric models are often too inflexible while nonparametric procedures tend to be nonrobust because of insufficient data on these high-dimensional spaces. This is particularly true when interest lies in building efficient classifiers in the presence of many predictor variables. When dealing with these types of data, it is often the case that most of the variability tends to lie along a few directions, or more generally along a much smaller dimensional submanifold of the data space. In this article, we propose a class of models that flexibly learn about this submanifold while simultaneously performing dimension reduction in classification. This methodology allows the cell probabilities to vary nonparametrically based on a few coordinates expressed as linear combinations of the predictors. Also, as opposed to many black-box methods for dimensionality reduction, the proposed model is appealing in having clearly interpretable and identifiable parameters that provide insight into which predictors are important in determining accurate classification boundaries. Gibbs sampling methods are developed for posterior computation, and the methods are illustrated using simulated and real data applications.

FRKAS: Knowledge Acquisition Using a Fuzzy Rule Base Approach to Insight of DNA-Binding Domains/Proteins

Numerous prediction methods of DNA-binding domains/proteins were proposed by identifying informative features and designing effective classifiers. These researches reveal that the DNA-protein binding mechanism is complicated and existing accurate predictors such as support vector machine (SVM) with position specific scoring matrices (PSSMs) are regarded as black-box methods which are not easily interpretable for biologists. In this study, we propose an ensemble fuzzy rule base classifier consisting of a set of interpretable fuzzy rule classifiers (iFRCs) using informative physicochemical properties as features. In designing iFRCs, feature selection, membership function design, and fuzzy rule base generation are all simultaneously optimized using an intelligent genetic algorithm (IGA). IGA maximizes prediction accuracy, minimizes the number of features selected, and minimizes the number of fuzzy rules to generate an accurate and concise fuzzy rule base. Benchmark datasets of DNA-binding domains are used to evaluate the proposed ensemble classifier of 30 iFRCs. Each iFRC has a mean test accuracy of 77.46%, and the ensemble classifier has a test accuracy of 83.33%, where the method of SVM with PSSMs has the accuracy of 82.81%. The physicochemical properties of the first two ranks according to their contribution are positive charge and Van Der Waals volume. Charge complementarity between protein and DNA is thought to be important in the first step of recognition between protein and DNA. The amino acid residues of binding peptides have larger Van Der Waals volumes and positive charges than those of non-binding ones. The proposed knowledge acquisition method by establishing a fuzzy rule-based classifier can also be applicable to predict and analyze other protein functions from sequences.

Static test case prioritization using topic models

Software development teams use test suites to test changes to their source code. In many situations, the test suites are so large that executing every test for every source code change is infeasible, due to time and resource constraints. Development teams need to prioritize their test suite so that as many distinct faults as possible are detected early in the execution of the test suite. We consider the problem of static black-box test case prioritization (TCP), where test suites are prioritized without the availability of the source code of the system under test (SUT). We propose a new static black-box TCP technique that represents test cases using a previously unused data source in the test suite: the linguistic data of the test cases, i.e., their identifier names, comments, and string literals. Our technique applies a text analysis algorithm called topic modeling to the linguistic data to approximate the functionality of each test case, allowing our technique to give high priority to test cases that test different functionalities of the SUT. We compare our proposed technique with existing static black-box TCP techniques in a case study of multiple real-world open source systems: several versions of Apache Ant and Apache Derby. We find that our static black-box TCP technique outperforms existing static black-box TCP techniques, and has comparable or better performance than two existing execution-based TCP techniques. Static black-box TCP methods are widely applicable because the only input they require is the source code of the test cases themselves. This contrasts with other TCP techniques which require access to the SUT runtime behavior, to the SUT specification models, or to the SUT source code.