Benchmark Suite Research Articles

Applying genetic programming to PSB2: the next generation program synthesis benchmark suite

Applying genetic programming to PSB2: the next generation program synthesis benchmark suite

Using spatial neighborhoods for parameter adaptation: An improved success history based differential evolution

Differential Evolution (DE) has been widely appraised as a simple yet robust population-based, non-convex optimization algorithm primarily designed for continuous optimization. Two important control parameters of DE are the scale factor F, which controls the amplitude of a perturbation step on the current solutions and the crossover rate Cr, which limits the mixing of components of the parent and the mutant individuals during recombination. We propose a very simple, yet effective, nearest spatial neighborhood-based modification to the adaptation process of the aforesaid parameters in the Success-History based adaptive DE (SHADE) algorithm. SHADE uses a historical archive of the successful F and Cr values to update these parameters and stands out as a very competitive DE variant of current interest. Our proposed modifications can be extended to any SHADE-based DE algorithm like L-SHADE (SHADE with linear population size reduction), jSO (L-SHADE with modified mutation) etc. The enhanced performance of the modified SHADE algorithm is showcased on the IEEE CEC (Congress on Evolutionary Computation) 2013, 2014, 2015, and 2017 benchmark suites by comparing against the DE-based winners of the corresponding competitions. Furthermore, the effectiveness of the proposed neighborhood-based parameter adaptation strategy is demonstrated by using the real-life problems from the IEEE CEC 2011 competition on testing evolutionary algorithms on real-world numerical optimization problems.

SMT-based verification of program changes through summary repair

This article provides an innovative approach for verification by model checking of programs that undergo continuous changes. To tackle the problem of repeating the entire model checking for each new version of the program, our approach verifies programs incrementally. It reuses computational history of the previous program version, namely function summaries. In particular, the summaries are over-approximations of the bounded program behaviors. Whenever reusing of summaries is not possible straight away, our algorithm repairs the summaries to maximize the chance of reusability of them for subsequent runs. We base our approach on satisfiability modulo theories (SMT) to take full advantage of lightweight modeling approach and at the same time the ability to provide concise function summarization. Our approach leverages pre-computed function summaries in SMT to localize the checks of changed functions. Furthermore, to exploit the trade-off between precision and performance, our approach relies on the use of an SMT solver, not only for underlying reasoning, but also for program modeling and the adjustment of its precision. On the benchmark suite of primarily Linux device drivers versions, we demonstrate that our algorithm achieves an order of magnitude speedup compared to prior approaches.

GAMORRA: An API-level workload model for rasterization-based graphics pipeline architecture

The performance of applications that require frame rendering time estimation or dynamic frequency scaling, rely on the accuracy of the workload model that is utilized within these applications. Existing models lack sufficient accuracy in their core model. Hence, they require changes to the target application or the hardware to produce accurate results. This paper introduces a mathematical workload model for a rasterization-based graphics Application Programming Interface (API) pipeline, named GAMORRA, which works based on the load of each stage of the pipeline. Firstly, GAMORRA models the load of each stage of the pipeline based on their input data size. Then, the calculated workloads of the stages are fed to a Multiple Linear Regression (MLR) model as explanatory variables. A hybrid offline/online training scheme is proposed as well to train the model. A suite of benchmarks is also designed to tune the model parameters based on the performance of the target system. The experiments were performed on Direct3D 11 and on two different rendering platforms comparing GAMORRA to an AutoRegressive (AR) model, a Frame Complexity Model (FCM) and a frequency-based (FRQ) model. The experiments show an average of 1.27 ms frame rendering time estimation error (9.4%) compared to an average of 1.87 ms error (13.2%) for FCM which is the best method among the three chosen methods. However, this comes at the cost of 0.54 ms (4.6%) increase in time complexity compared to FCM. Furthermore, GAMMORA improves frametime underestimations by 1.1% compared to FCM.

SCD: A Stacked Carton Dataset for Detection and Segmentation.

Carton detection is an important technique in the automatic logistics system and can be applied to many applications such as the stacking and unstacking of cartons and the unloading of cartons in the containers. However, there is no public large-scale carton dataset for the research community to train and evaluate the carton detection models up to now, which hinders the development of carton detection. In this article, we present a large-scale carton dataset named Stacked Carton Dataset (SCD) with the goal of advancing the state-of-the-art in carton detection. Images were collected from the Internet and several warehouses, and objects were labeled for precise localization using instance mask annotation. There were a total of 250,000 instance masks from 16,136 images. Naturally, a suite of benchmarks was established with several popular detectors and instance segmentation models. In addition, we designed a carton detector based on RetinaNet by embedding our proposed Offset Prediction between the Classification and Localization module (OPCL) and the Boundary Guided Supervision module (BGS). OPCL alleviates the imbalance problem between classification and localization quality, which boosts AP by ∼ on SCD at the model level, while BGS guides the detector to pay more attention to the boundary information of cartons and decouple repeated carton textures at the task level. To demonstrate the generalization of OPCL for other datasets, we conducted extensive experiments on MS COCO and PASCAL VOC. The improvements in AP on MS COCO and PASCAL VOC were ∼ and ∼, respectively.

FPGA Architecture Exploration for DNN Acceleration

Recent years have seen an explosion of machine learning applications implemented on Field-Programmable Gate Arrays (FPGAs) . FPGA vendors and researchers have responded by updating their fabrics to more efficiently implement machine learning accelerators, including innovations such as enhanced Digital Signal Processing (DSP) blocks and hardened systolic arrays. Evaluating architectural proposals is difficult, however, due to the lack of publicly available benchmark circuits. This paper addresses this problem by presenting an open-source benchmark circuit generator that creates realistic DNN-oriented circuits for use in FPGA architecture studies. Unlike previous generators, which create circuits that are agnostic of the underlying FPGA, our circuits explicitly instantiate embedded blocks, allowing for meaningful comparison of recent architectural proposals without the need for a complete inference computer-aided design (CAD) flow. Our circuits are compatible with the VTR CAD suite, allowing for architecture studies that investigate routing congestion and other low-level architectural implications. In addition to addressing the lack of machine learning benchmark circuits, the architecture exploration flow that we propose allows for a more comprehensive evaluation of FPGA architectures than traditional static benchmark suites. We demonstrate this through three case studies which illustrate how realistic benchmark circuits can be generated to target different heterogeneous FPGAs.

A reinforcement learning based RMOEA/D for bi-objective fuzzy flexible job shop scheduling

The flexible job shop scheduling problem (FJSP) is significant for realistic manufacturing. However, the job processing time usually is uncertain and changeable during manufacturing. This paper presents a multi-objective FJSP with fuzzy processing time (MOFFJSP) for optimizing the makespan and total machine workload as objectives. To solve the MOFFJSP, a MOEA/D based on reinforcement learning named RMOEA/D is proposed. RMOEA/D can be featured as: (i) an initial strategy with three rules is used to get a high-quality initial population; (ii) a parameter adaption strategy based on Q-learning is proposed to guide the population choose the best parameter to increase diversity; (iii) a variable neighborhood search based on reinforcement learning is designed to lead the solution to choose the right local search method; and (iv) an elite archive is used to improve the usage rate of the abandoned historical solution. RMOEA/D is compared with five well-known realted methods, i.e., MOEA/D, NSGA-II, MOEA/D-M2M, NSGA-III and IAIS on three benchmark suites. The results show that RMOEA/D outperforms these five state-of-art algorithms.

Scientific machine learning benchmarks

Deep learning has transformed the use of machine learning technologies for the analysis of large experimental datasets. In science, such datasets are typically generated by large-scale experimental facilities, and machine learning focuses on the identification of patterns, trends and anomalies to extract meaningful scientific insights from the data. In upcoming experimental facilities, such as the Extreme Photonics Application Centre (EPAC) in the UK or the international Square Kilometre Array (SKA), the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. However, at present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is a challenge due to the potential applicability of many different machine learning frameworks, computer architectures and machine learning models. Historically, for modelling and simulation on high-performance computing systems, these issues have been addressed through benchmarking computer applications, algorithms and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to open, curated scientific datasets is a new challenge for both scientists and computer scientists. Here, we introduce the concept of machine learning benchmarks for science and review existing approaches. As an example, we describe the SciMLBench suite of scientific machine learning benchmarks. Finding the most appropriate machine learning algorithm for the analysis of any given scientific dataset is currently challenging, but new machine learning benchmarks for science are being developed to help.

Indicator-Based Evolutionary Algorithm for Solving Constrained Multiobjective Optimization Problems

To prevent the population from getting stuck in local areas and then missing the constrained Pareto front fragments in dealing with constrained multiobjective optimization problems (CMOPs), it is important to guide the population to evenly explore the promising areas that are not dominated by all examined feasible solutions. To this end, we first introduce a cost value-based distance into the objective space, and then use this distance and the constraints to define an indicator to evaluate the contribution of each individual to exploring the promising areas. Theoretical studies show that the proposed indicator can effectively guide population to focus on exploring the promising areas without crowding in local areas. Accordingly, we propose a new constraint handling technique (CHT) based on this indicator. To further improve the diversity of population in the promising areas, the proposed indicator-based CHT divides the promising areas into multiple subregions, and then gives priority to removing the individuals with the worst fitness values in the densest subregions. We embed the indicator-based CHT in evolutionary algorithm and propose an indicator-based constrained multiobjective algorithm for solving CMOPs. Numerical experiments on several benchmark suites show the effectiveness of the proposed algorithm. Compared with six state-of-the-art constrained evolutionary multiobjective optimization algorithms, the proposed algorithm performs better in dealing with different types of CMOPs, especially in those problems that the individuals are easy to appear in the local infeasible areas that dominate the constrained Pareto front fragments.

PMU-Events-Driven DVFS Techniques for Improving Energy Efficiency of Modern Processors

This paper describes the results of our measurement-based study, conducted on an Intel Core i7 processor running the SPEC CPU2017 benchmark suites, that evaluates the impact of dynamic voltage frequency scaling (DVFS) on performance (P), energy efficiency (EE) , and their product (PxEE) . The results indicate that the default DVFS-based power management techniques heavily favor performance, resulting in poor energy efficiency. To remedy this problem, we introduce, implement, and evaluate four DVFS-based power management techniques driven by the following metrics derived from the processor's performance monitoring unit: (i) the total pipeline slot stall ratio (FS-PS), (ii) the total cycle stall ratio (FS-TS), (iii) the total memory-related cycle stall ratio (FS-MS), and (iv) the number of last level cache misses per kilo instructions (FS-LLCM). The proposed techniques linearly map these metrics onto the available processor clock frequencies. The experimental evaluation results show that the proposed techniques significantly improve EE and PxEE metrics compared to the existing approaches. Specifically, EE improves from 44% to 92%, and PxEE improves from 31% to 48% when all the benchmarks are considered together. Furthermore, we find that the proposed techniques are particularly effective for a class of memory-intensive benchmarks – they improve EE from 121% to 183% and PxEE from 100% to 141%. Finally, we elucidate the advantages and disadvantages of each of the proposed techniques and offer recommendations on using them.

Non-Dominated Sorting Manta Ray Foraging Optimization for Multi-Objective Optimal Power Flow with Wind/Solar/Small- Hydro Energy Sources

This present study describes a novel manta ray foraging optimization approach based non-dominated sorting strategy, namely (NSMRFO), for solving the multi-objective optimization problems (MOPs). The proposed powerful optimizer can efficiently achieve good convergence and distribution in both the search and objective spaces. In the NSMRFO algorithm, the elitist non-dominated sorting mechanism is followed. Afterwards, a crowding distance with a non-dominated ranking method is integrated for the purpose of archiving the Pareto front and improving the optimal solutions coverage. To judge the NSMRFO performances, a bunch of test functions are carried out including classical unconstrained and constrained functions, a recent benchmark suite known as the completions on evolutionary computation 2020 (CEC2020) that contains twenty-four multimodal optimization problems (MMOPs), some engineering design problems, and also the modified real-world issue known as IEEE 30-bus optimal power flow involving the wind/solar/small-hydro power generations. Comparison findings with multimodal multi-objective evolutionary algorithms (MMMOEAs) and other existing multi-objective approaches with respect to performance indicators reveal the NSMRFO ability to balance between the coverage and convergence towards the true Pareto front (PF) and Pareto optimal sets (PSs). Thus, the competing algorithms fail in providing better solutions while the proposed NSMRFO optimizer is able to attain almost all the Pareto optimal solutions.

Comparative Analysis of K-Means Variants Implemented in R

One of the ways of acquiring new knowledge or underlying patterns in data is by means of clustering algorithms or techniques for creating groups of objects or individuals with similar characteristics in each group and at the same time different from the other groups. There is a consensus in the scientific community that the most widely used clustering algorithm is K-means, mainly because its results are easy to interpret and there are different implementations. In this paper we present an exploratory analysis of the behavior of the main variants of the K-means algorithm (HartiganWong, Lloyd, Forgy and MacQueen) when solving some of the difficult sets of instances from the Fundamental Clustering Problems Suite (FCPS) benchmark. These variants are implemented in the R language and allow finding the minimum and maximum intra-cluster distance of the final clustering. The different scenarios are shown with the results obtained.

A Novel Memetic Algorithm Based on Multiparent Evolution and Adaptive Local Search for Large-Scale Global Optimization.

In many fields, including management, computer, and communication, Large-Scale Global Optimization (LSGO) plays a critical role. It has been applied to various applications and domains. At the same time, it is one of the most challenging optimization problems. This paper proposes a novel memetic algorithm (called MPCE & SSALS) based on multiparent evolution and adaptive local search to address the LSGO problems. In MPCE & SSALS, a multiparent crossover operation is used for global exploration, while a step-size adaptive local search is utilized for local exploitation. A new offspring is generated by recombining four parents. In the early stage of the algorithm execution, global search and local search are performed alternately, and the population size gradually decreases to 1. In the later stage, only local searches are performed for the last individual. Experiments were conducted on 15 benchmark functions of the CEC′2013 benchmark suite for LSGO. The results were compared with four state-of-the-art algorithms, demonstrating that the proposed MPCE & SSALS algorithm is more effective.

A Partial-Reconfiguration-Enabled HW/SW Co-Design Benchmark for LTE Applications

Rapid and continuous evolution in telecommunication standards and applications has increased the demand for a platform with high parallelization capability, high flexibility, and low power consumption. FPGAs are known platforms that can provide all these requirements. However, the evaluation of approaches, architectures, and scheduling policies in this era requires a suitable and open-source benchmark suite that runs on FPGA. This paper harnesses high-level synthesis tools to implement high-performance, resource-efficient, and easy-maintenance kernels for FPGAs. We provide various implementations of each kernel of PHY-Bench and WiBench, which are the most well-known benchmark suites for telecommunication applications on FPGAs. We analyze the execution time and power consumption of different kernels on ARM processors and FPGA. We have made all sources and documentation public for the benefit of the research community. The codes are flexible, and all kernels can easily be regenerated for different sizes. The results show that the FPGA can increase the speed by up to 19.4 times. Furthermore, we show that the power consumption of the FPGA can be reduced by up to 45% by partially reconfiguring a kernel that fits the size of the input data instead of using a large kernel that supports all inputs. We also show that partial reconfiguration can improve the execution time for processing a sub-frame in the uplink application by 33% compared to an FPGA-based approach without partial reconfiguration.

A Case for Precise, Fine-Grained Pointer Synthesis in High-Level Synthesis

This article combines two practical approaches to improve pointer synthesis within HLS tools. Both approaches focus on inefficiencies in how HLS tools treat thepoints-to graph—a mapping that connects each instruction to the memory locations that it might access at runtime. HLS pointer synthesis first computes the points-to graph via pointer analysis and then implements its connections in hardware, which gives rise to two inefficiencies. First, HLS tools typically favour pointer analysis that is fast, sacrificing precision. Second, they also favour centralising memory connections in hardware for instructions that can point to more than one location.In this article, we demonstrate that a more precise pointer analysis coupled with decentralised memory connections in hardware can substantially reduce the unnecessary sharing of memory resources. We implement both flow- and context-sensitive pointer analysis and fine-grained memory connections in two modern HLS tools, LegUp and Vitis HLS. An evaluation on three benchmark suites, ranging from non-trivial pointer use to standard HLS benchmarks, indicates that when we improve both precision and granularity of pointer synthesis, on average, we can reduce area and latency by around 42% and 37%, respectively.

A powerful meta-heuristic search algorithm for solving global optimization and real-world solar photovoltaic parameter estimation problems

The teaching-learning-based artificial bee colony (TLABC) is a new hybrid swarm-based metaheuristic search algorithm. It combines the exploitation of the teaching learning-based optimization (TLBO) with the exploration of the artificial bee colony (ABC). With the hybridization of these two nature-inspired swarm intelligence algorithms, a robust method has been proposed to solve global optimization problems. However, as with swarm-based algorithms, with the TLABC method, it is a great challenge to effectively simulate the selection process. Fitness-distance balance (FDB) is a powerful recently developed method to effectively imitate the selection process in nature. In this study, the three search phases of the TLABC algorithm were redesigned using the FDB method. In this way, the FDB-TLABC algorithm, which imitates nature more effectively and has a robust search performance, was developed. To investigate the exploitation, exploration, and balanced search capabilities of the proposed algorithm, it was tested on standard and complex benchmark suites (Classic, IEEE CEC 2014, IEEE CEC 2017, and IEEE CEC 2020). In order to verify the performance of the proposed FDB-TLABC for global optimization problems and in the photovoltaic parameter estimation problem (a constrained real-world engineering problem) a very comprehensive and qualified experimental study was carried out according to IEEE CEC standards. Statistical analysis results confirmed that the proposed FDB-TLABC provided the best optimum solution and yielded a superior performance compared to other optimization methods.

Lithography Hotspot Detection Method Based on Transfer Learning Using Pre-Trained Deep Convolutional Neural Network

As the designed feature size of integrated circuits (ICs) continues to shrink, the lithographic printability of the design has become one of the important issues in IC design and manufacturing. There are patterns that cause lithography hotspots in the IC layout. Hotspot detection affects the turn-around time and the yield of IC manufacturing. The precision and F1 score of available machine-learning-based hotspot-detection methods are still insufficient. In this paper, a lithography hotspot detection method based on transfer learning using pre-trained deep convolutional neural network is proposed. The proposed method uses the VGG13 network trained with the ImageNet dataset as the pre-trained model. In order to obtain a model suitable for hotspot detection, the pre-trained model is trained with some down-sampled layout pattern data and takes cross entropy as the loss function. ICCAD 2012 benchmark suite is used for model training and model verification. The proposed method performs well in accuracy, recall, precision, and F1 score. There is significant improvement in the precision and F1 score. The results show that updating the weights of partial convolutional layers has little effect on the results of this method.

Diversified Position Update Equation-Based SSA with Refreshing-Gap Strategy for global optimization

This study, incorporating three improvements into the structure of the original SSA algorithm to increase its exploration and exploitation capabilities, suggests an SSA variant algorithm named ”Diversified Position Update Equation-Based SSA with Refreshing-Gap Strategy (DSSA-R)”. Unlike the original SSA, DSSA-R accepts half of the population as the leader instead of using a single leader salp strategy. Nevertheless, the suggested algorithm includes two different equations to update the positions of the leader salps and follower salps. Furthermore, Refreshing Gap strategy was included in DSSA-R to avoid the algorithm from being stuck in the local optimum. The final improvement made on the original SSA was the addition of ”Linear Population Reduction Strategy” to strengthen the exploitation aspect of the algorithm. The performance of the DSSA-R algorithm was tested using the functions of the CEC 2014 and CEC 2015 benchmark suites and compared with five SSA variants and four state-of-the-art algorithms. Test results show that DSSA-R obtained better and competitive results compared to the algorithms it was compared with.

CARL-D: A vision benchmark suite and large scale dataset for vehicle detection and scene segmentation

Vision-based object detection and scene understanding are becoming key features of environment perception and autonomous driving. In the past couple of years, numerous large-scale datasets for visual object detection and semantic understanding have been released which has enormously benefited the environment perception in self-driving cars. However, these datasets like Kitti and Cityscapes only focus on well organized and urban road traffic scenarios of European countries while ignoring the dense and unpattern traffic conditions of sub-continent countries like Pakistan, India, Bangladesh, and Sri Lanka. Consequently, the environment perception system developed on these datasets cannot efficiently assist self-driving cars in traffic scenarios of sub-continent countries. To this end, we present CARL-D, a large-scale dataset and benchmark suite for develop 2D object detection and instance/pixel-level segmentation methods for self-driving cars. CARL-D comprises large-scale stereo vision-based driving videos captured from more than 100 cities of Pakistan, including motorways, dense and unpattern traffic scenarios of urban, rural, and hilly areas. As a benchmark selection, 15,000 suitable images are labeled for 2D object detection and recognition. Whereas semantic segmentation benchmark contains 2500 images with pixel-level high-quality fine annotations and 5000 coarse-annotated images which could help in enabling the deep neural networks to leverage the weakly labeled data. Alongside the dataset, we also present transfer learning-based 2D vehicle detection and scene segmentation methods to evaluate the performance of existing state-of-the-art deep neural networks on our dataset. Lastly, an extensive experimental evaluation along with the comparative study has been carried out which demonstrates the upper edge of our dataset in terms of interclass-diversity, scene-variability, and annotations richness. The proposed benchmark suite is available at https://carl-dataset.github.io/index/.

A Hardware Trojan Detection and Diagnosis Method for Gate-Level Netlists Based on Different Machine Learning Algorithms

The design complexity and outsourcing trend of modern integrated circuits (ICs) have increased the chance for adversaries to implant hardware Trojans (HTs) in the development process. To effectively defend against this hardware-based security threat, many solutions have been reported in the literature, including dynamic and static techniques. However, there is still a lack of methods that can simultaneously detect and diagnose HT circuits with high accuracy and low time complexity. Therefore, to overcome these limitations, this paper presents an HT detection and diagnosis method for gate-level netlists (GLNs) based on different machine learning (ML) algorithms. Given a GLN, the proposed method first partitions it into several circuit cones and extracts seven HT-related features from each cone. Then, we repeat this process for the sample GLN to construct a dataset for the next step. After that, we use K-Nearest Neighbor (KNN), Decision Tree (DT) and Naive Bayes (NB) to classify all circuit cones of the target GLN. Finally, we determine whether each circuit cone is HT-implanted through the label, completing the HT detection and diagnosis for target GLN. We have applied our method to 11 GLNs from ISCAS’85 and ISCAS’89 benchmark suites. As shown in experimental results of the three ML algorithms used in our method: (1) NB costs shortest time and achieves the highest average true positive rate (ATPR) of 100%; (2) DT costs longest time but achieve the highest average true negative rate (ATNR) of 98.61%; (3) Compared to NB and DT, KNN costs a slightly longer time than NB but the ATPR and ATNR values are approximately close to DT. Moreover, it can also report the possible implantation location of a Trojan instance according to the detecting results.