Benchmarking Framework Research Articles

Benchmarking framework for anomaly localization: Towards real-world deployment of automated visual inspection

Localizing defects in products is a critical component of industrial pipelines in manufacturing, retail, and many other industries to ensure consistent delivery of high quality products. Automated anomaly localization systems leveraging computer vision have the potential to replace laborious and subjective manual inspection of products. Recently, there have been tremendous efforts in this research domain investigating deep learning-based anomaly localization methods. However, such proposed methods, mainly considering product-specific evaluation, cannot be directly implemented for real-world use. Therefore, despite the advancements, there is still a gap between research and deployment of those methods to real-world production environments. Implementing any automated solution for manufacturing can involve a steep upfront investment. It is important to develop an industry-friendly benchmarking framework to ensure the feasibility and robustness of an automated quality control system. In this paper, we present a new anomaly localization benchmarking framework considering different aspects - (1) understand the performance of models in a generalizable product-agnostic manner, (2) explore pros and cons to find the most optimal modeling approach, (3) develop an efficient training and inference scheme with defect-free training samples and very few defective samples for evaluation, and (4) perform an ablation study of threshold estimation techniques to determine optimal threshold level for segmentation. We release a newly-labeled dataset for the research community with product-agnostic categorization of defective product images. To the best of our knowledge, this is the first anomaly localization work on developing a benchmarking framework focusing on real-world use. We believe domain experts from different industries will find this useful and can gain valuable insights to deploy automated visual inspection in production pipelines.

A Bin-Picking Benchmark for Systematic Evaluation of Robotic-Assisted Food Handling for Line Production

Robotic manipulation and automation have gained increasing popularity in the food manufacturing industry due to their potential benefits for enhancing hygiene standards, enforcing quality consistency, promoting product traceability, and reducing labor costs. As a majority of robotic manipulation, the pick-and-place operation plays a crucial role in food handling applications. However, the reproducibility and comparability of results have put a dilemma that hinders further advancement in this field, especially for those unstructured scenarios. To tackle such thorny issues, this article proposes a benchmarking framework for system-level evaluation of robotic-assisted food handling under the line production environment. A typical food handling scenario, including a pick-and-place operation and a packing operation, is presented as the benchmark task, where food items are supposed to be picked from the tray and placed in the serving dish. A robotic system incorporating a high-speed Delta robot, vision system, conveyor belt, and end-effector is developed as the testbed for the benchmarking implementation. Finally, five variants of the robotic system with different end-effectors are evaluated using the proposed benchmarking framework. Comparative studies illustrate the performance of various benchmarked systems and validate the applicability of the benchmarking strategy for the food handling context. Videos of our experiments are available at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/SBAOoswnjWM</i> .

Fine-Grained Benchmarking and Targeted Optimization: Enabling Green IoT-Oriented Blockchain in the 6G Era

For 6G-empowered IoT, blockchain is widely regarded as a fundamental technique for addressing the security and privacy issues. Nonetheless, the current blockchains suffer from several flaws, especially the low performance and scalability. They seriously hinder the blockchain deployments in 6G-empowered IoT with high concurrency and complexity. To address this problem, we need to better understand blockchains and try to optimize them. However, we still lack a standard tool for comprehensively benchmarking the IoT-oriented blockchains in 6G era. As to blockchain optimization, the current general-purpose strategies can hardly hold efficiency all the time due to the heterogeneity of 6G-empowered IoT. In this paper, we present a novel pipeline for facilitating the green IoT-oriented blockchains in 6G era, called &#x201C;fine-grained benchmarking + targeted optimization". Specifically, we first design a benchmarking framework which realizes the whole workflow of data acquisition, log parsing, and metric visualization, offering quantitative blockchain performance reports. Moreover, our framework provides ten metrics divided into four dimensions, leading blockchain benchmarking to the fine-grained level. Based on the acquired insights, we further present the concept of targeted optimization, i.e., directly designing optimization strategies based on the metrics that perform poorly in benchmarking. To better illustrate this pipeline, we take IOTA, the most potential IoT-oriented blockchain in 6G era, as a case study. We observe two defects of IOTA, i.e., low power efficiency and slow ledger synchronization. Following our pipeline, we design two targeted optimization mechanisms, namely Proof-of-Resource Efficiency and Dynamic Local Snapshot Protocol. Extensive experiments demonstrate that our pipeline can maximize the optimization efficiency, successfully addressing the issues hindering IOTA. Last but not least, we emphasize that the proposed pipeline can also be applied by other IoT-oriented blockchains in the 6G era.

DPUBench: An application-driven scalable benchmark suite for comprehensive DPU evaluation

With the development of data centers, network bandwidth has rapidly increased, reaching hundreds of Gbps. However, the network I/O processing performance of CPU improvement has not kept pace with this growth in recent years, which leads to the CPU being increasingly burdened by network applications in data centers. To address this issue, Data Processing Unit (DPU) has emerged as a hardware accelerator designed to offload network applications from the CPU. As a new hardware device, the DPU architecture design is still in the exploration stage. Previous DPU benchmarks are not neutral and comprehensive, making them unsuitable as general benchmarks. To showcase the advantages of their specific architectural features, DPU vendors tend to provide some particular architecture-dependent evaluation programs. Moreover, they fail to provide comprehensive coverage and cannot adequately represent the full range of network applications. To address this gap, we propose an application-driven scalable benchmark suite called DPUBench. DPUBench classifies DPU applications into three typical scenarios — network, storage, and security, and includes a scalable benchmark framework that contains essential Operator Set in these scenarios and End-to-end Evaluation Programs in real data center scenarios. DPUBench can easily incorporate new operators and end-to-end evaluation programs as DPU evolves. We present the results of evaluating the NVIDIA BlueField-2 using DPUBench and provide optimization recommendations. DPUBench are publicly available from https://www.benchcouncil.org/DPUBench.

IoT Traffic Analyzer Tool with Automated and Holistic Feature Extraction Capability.

The Internet of Things (IoT) is an emerging technology that attracted considerable attention in the last decade to become one of the most researched topics in computer science studies. This research aims to develop a benchmark framework for a public multi-task IoT traffic analyzer tool that holistically extracts network traffic features from an IoT device in a smart home environment that researchers in various IoT industries can implement to collect information about IoT network behavior. A custom testbed with four IoT devices is created to collect real-time network traffic data based on seventeen comprehensive scenarios of these devices' possible interactions. The output data is fed into the IoT traffic analyzer tool for both flow and packet levels analysis to extract all possible features. Such features are ultimately classified into five categories: IoT device type, IoT device behavior, Human interaction type, IoT behavior within the network, and Abnormal behavior. The tool is then evaluated by 20 users considering three variables: usefulness, accuracy of information being extracted, performance and usability. Users in three groups were highly satisfied with the interface and ease of use of the tool, with scores ranging from 90.5% to 93.8% and with an average score between 4.52 and 4.69 with a low standard deviation range, indicating that most of the data revolve around the mean.

Blending of energy benchmarks models for residential buildings

Building energy consumption forecasting is crucial for energy managers and policymakers. However, acutely predicting the energy consumption of any building is more important in energy efficiency benchmarking of buildings. Energy benchmarking is a precise technique for measuring, tracking, and reducing end-use energy usage of buildings by employing comparative scenarios. Bottom-Up energy consumption assessment has been used in several studies to evaluate a collection of buildings' energy performance to a standard. Traditionally, researchers have used a single approach to predict the energy use of buildings and to conduct benchmarking. However, it has been found that doing so frequently compromises the accuracy of the predicted energy use of buildings. It also has an adverse effect on the buildings' energy benchmarking framework results. This paper will provide an energy benchmarking framework for residential buildings based on the blending techniques of multiple approaches. Over 3000 datasets were gathered from the Indian city of Jaipur. In order to analyse the trends in energy consumption, the data were divided into categories based on economic levels. The data is then subjected to some fundamental statistical analysis, along with several approaches are blended to forecast the energy consumption of the buildings with the highest accuracy. These benchmark levels set guidelines for assessing and recognising strong performance while also identifying underperforming buildings and giving them more priority for energy efficiency improvement. The technique may be used for different metropolitan levels and climatic conditions.

A benchmarking framework for performance evaluation of statistical wind power forecasting models

Lack of benchmarks for wind power forecasting models undermine their potential and consequently their implementation for industry applications. Despite the extensive existing literature in the field, a unified framework has not been created yet. Therefore, we propose a benchmark set-up where statistical wind power forecasting models can be tested under standardized criteria with respect to data, time resolution, and prediction horizon, while evaluating them under varied representative operational conditions of wind farms. The utility of this framework is shown with an example case applied to mode decomposition models, which have shown a higher performance compared to other statistical models in recent times. Data collected from two Irish wind farms are used to calculate the accuracy of statistical wind power forecasting models. Their robustness is also examined by providing an assessment of such models under several scenarios of power generation. In the example case of this benchmark, results indicate that the use of variational mode decomposition as decomposition algorithm together with advanced recurrent networks provide the best performance among the evaluated forecasting models.

A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support

ObjectiveThe study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. Materials and methodsOur methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods—research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. ResultsSix research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DiscussionThe proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. ConclusionWe posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Development of soil health benchmarks for managed and semi-natural landscapes

Efforts to improve soil health require that target values of key soil properties are established. No agreed targets exist but providing population data as benchmarks is a useful step to standardise soil health comparison between landscapes. We exploited nationally representative topsoil (0–15 cm) measurements to derive soil health benchmarks for managed and semi-natural environments across Great Britain. In total, 4587 soil organic matter (SOM), 3860 pH, 2908 bulk density (BD), and 465 earthworm abundance (EA) datapoints were used. As soil properties are sensitive to site-specific characteristics, data were stratified by habitat, soil type, and mean annual precipitation, with benchmarks defined as the middle 80 % of values in each distribution – yielding 135 benchmarks. BD and pH decreased with land management intensity (agriculture > semi-natural grasslands > woodlands > heathlands > wetlands), and vice versa for SOM and EA. Normalising benchmark ranges by medians revealed soil health indicator benchmark widths increased in the order: pH < BD < SOM < EA, while width increased with decreasing land management intensity. Arable and horticulture and improved grassland exhibited narrow benchmarks for SOM, pH and BD, yet the widest EA benchmark, suggesting additional drivers impact EA patterns. Upland wetlands had the widest BD benchmarks, important when determining carbon stocks. East Anglia currently possesses the largest proportions of atypical soils, including below typical SOM (19.2 %), above typical BD (17.4 %) and pH (39.1 %), and the smallest proportions of above typical SOM (2.4 %), and below typical BD (5.8 %) and pH (2.3 %). This is found even after land use, soil type and rainfall have been considered, underscoring how urgently soil health should be addressed here. Our benchmarking framework allows landowners to compare where their measured soil health indicators fall within expected ranges and is applicable to other biomes, national and multinational contexts.

Towards measuring national railways’ safety through a benchmarking framework of transparency and published data

Safety is paramount in railway transportation. Learning best practices of risk data reporting and documentation from different railway systems help to better identify safety issues. However, literature shows that comparing statistics between different systems or countries is usually unreliable without comparable data. Globally, railway risk data is not universally collected or published, so the lack of transparency prevents appropriate analyses. In this paper, the authors set out to assess the availability of railway risk data around the world and develop an evaluation framework to support cohesive benchmarks. A total of 148 countries that operate railways were analysed firstly by the public availability of safety statistics, and secondly by their performance against five KPIs in the years between 2015 and 2020. On average, 40 out of 148 countries consistently reported at least three of the five KPIs. It was found that more countries reported data on accident types than on their effects. More importantly, findings highlight the vast disparity of reporting and availability between the Global North and South and demonstrate how applications of the framework showed how the framework can be used to improve overall railway safety. Further work may focus on establishing narrower scope work where countries may benchmark themselves against high-performing counterparts and identify structures and processes that lead to better safety conditions. Moreover, it is anticipated that the framework may also encourage the culture of publishing data to identify whether safety is improving, shifting the stigma of reporting negative events towards a more open approach to benchmarking.

VeriBench: Analyzing the Performance of Database Systems with Verifiability

Database systems are paying more attention to data security in recent years. Immutable systems such as blockchains, verifiable databases, and ledger databases are equipped with various verifiability mechanisms to protect data. Such systems often adopt different threat models, and techniques, therefore, have different performance implications compared to traditional database systems. So far, there is no uniform benchmarking tool for evaluating the performance of these systems, especially at the level of verification functions. In this paper, we first survey the design space of the verifiability-enabled database systems along five dimensions: threat model, authenticated data structure (ADS), query processing, verification, and auditing. Based on this survey, we design and implement VeriBench, a benchmark framework for verifiability-enabled database systems. VeriBench enables a fair comparison of systems designed with different underlying technologies that share the client-side verification scheme, and focuses on design space exploration to provide a deeper understanding of different system design choices. VeriBench incorporates micro- and macro-benchmarks to provide a comprehensive evaluation. Further, VeriBench is designed to enable easy extension for benchmarking new systems and workloads. We run VeriBench to conduct a comprehensive analysis of state-of-the-art systems comprising blockchains, ledger databases, and log transparency technologies. The results expose the weaknesses and strengths of each underlying design choice, and the insights should serve as guidance for future development.

Proposal for an objective binary benchmarking framework that validates each other for comparing MCDM methods through data analytics.

When it comes to choosing the best option among multiple alternatives with criteria of different importance, it makes sense to use multi criteria decision making (MCDM) methods with more than 200 variations. However, because the algorithms of MCDM methods are different, they do not always produce the same best option or the same hierarchical ranking. At this point, it is important how and according to which MCDM methods will be compared, and the lack of an objective evaluation framework still continues. The mathematical robustness of the computational procedures, which are the inputs of MCDM methods, is of course important. But their output dimensions, such as their capacity to generate well-established real-life relationships and rank reversal (RR) performance, must also be taken into account. In this study, we propose for the first time two criteria that confirm each other. For this purpose, the financial performance (FP) of 140 listed manufacturing companies was calculated using nine different MCDM methods integrated with step-wise weight assessment ratio analysis (SWARA). İn the next stage, the statistical relationship between the MCDM-based FP final results and the simultaneous stock returns of the same companies in the stock market was compared. Finally, for the first time, the RR performance of MCDM methods was revealed with a statistical procedure proposed in this study. According to the findings obtained entirely through data analytics, Faire Un Choix Adéquat (FUCA) and (which is a fairly new method) the compromise ranking of alternatives from distance to ideal solution (CRADIS) were determined as the most appropriate methods by the joint agreement of both criteria.

International benchmarking of railroad safety data systems and performance – a cross-continental case study

Rail safety is a universal goal that every railroad system pursues. Comparing rail safety performance allows the identification of relative system strengths and weaknesses and potential adaptation of risk mitigation strategies from one railroad system to another. Achieving this requires comparable data from the railroad systems to be analyzed that are available and with high resolution so that fair comparisons can be established. This paper presented an international benchmarking framework for railroad safety-related data system and safety performance. A novel and standardized methodology was developed to collect railroad safety-related data sources among different countries and compare their data completeness and resolution. Six countries with high data availability and transparency were selected to demonstrate the benchmarking framework. High-level rail safety performance measures were derived and compared among these countries. The results showed that there are inconsistencies in the resolution of different types of rail safety data among the six countries. The countries that had the lowest and highest overall accident rate, grade crossing incident rate, and other safety performance metrices were identified. This research provided valuable insights into how railroad operators can improve their railroad safety-related data system and safety performance by benchmarking with other railroad operators to make the most efficient use of risk mitigation resources. The study also highlighted the importance of providing publicly available railroad safety-related data for the mutual benefits of overall safety of railroad systems around the world.

Minimizing energy consumption for NOMA multi-drone communications in automotive-industry 5.0

The forthcoming era of the automotive industry, known as Automotive-Industry 5.0, will leverage the latest advancements in 6G communications technology to enable reliable, computationally advanced, and energy-efficient exchange of data between diverse onboard sensors, drones and other vehicles. We propose a non-orthogonal multiple access (NOMA) multi-drone communications network in order to address the requirements of enormous connections, various quality of services (QoS), ultra-reliability, and low latency in upcoming sixth-generation (6G) drone communications. Through the use of a power optimization framework, one of our goals is to evaluate the energy efficiency of the system. In particular, we define a non-convex power optimization problem while considering the possibility of imperfect successive interference cancellation (SIC) detection. Therefore, the goal is to reduce the total energy consumption of NOMA drone communications while guaranteeing the lowest possible rate for wireless devices. We use a novel method based on iterative sequential quadratic programming (SQP) to get the best possible solution to the non-convex optimization problem so that we may move on to the next step and solve it. The standard OMA framework, the Karush–Kuhn–Tucker (KKT)-based NOMA framework, and the average power NOMA framework are compared with the newly proposed optimization framework. The results of the Monte Carlo simulation demonstrate the accuracy of our derivations. The results that have been presented also demonstrate that the optimization framework that has been proposed is superior to previous benchmark frameworks in terms of system-achievable energy efficiency.

Fuzzy Decision-Making Framework for Sensitively Prioritizing Autism Patients with Moderate Emergency Level

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder that requires careful assessment and management. The prioritization of ASD patients involves navigating through complexities such as conflicts, trade-offs, and the importance of different criteria. Therefore, this study focuses on prioritizing patients with ASD in the healthcare setting through an evaluation and benchmarking framework. The aim of this study is to develop a framework that utilizes Multi-Criteria Decision Making (MCDM) methods to assist healthcare professionals in prioritizing ASD patients, particularly those with moderate injury levels. The methodology of the framework outlines several phases, including dataset identification, development of a decision matrix, weighting of 19 ASD criteria using the FWZIC method, ranking 432 patients using the VIKOR method, and evaluating the proposed framework using four sensitivity analysis scenarios. Among the 19 ASD criteria, the criterion 'verbal communication' obtained the highest weight. Additionally, criteria such as 'laughing for no reason', 'nodding', 'patient movement at home', and 'pointing with the index finger' obtained similar higher weights, indicating their potential impact on ASD patients. The experimental results highlight the significance of adjusting ASD weights in influencing the final rankings obtained through the VIKOR method. This emphasizes the need for careful consideration when assigning weights to the 19 ASD criteria to ensure accurate prioritization. Moreover, the framework provides valuable insights into improving the care and support provided to individuals with autism in Iraq. The findings contribute to the existing body of knowledge in the field of autism care prioritization and pave the way for future research and interventions aimed at enhancing the quality of care for individuals with autism in Iraq.

Benchmarking machine learning robustness in Covid-19 genome sequence classification

The rapid spread of the COVID-19 pandemic has resulted in an unprecedented amount of sequence data of the SARS-CoV-2 genome—millions of sequences and counting. This amount of data, while being orders of magnitude beyond the capacity of traditional approaches to understanding the diversity, dynamics, and evolution of viruses, is nonetheless a rich resource for machine learning (ML) approaches as alternatives for extracting such important information from these data. It is of hence utmost importance to design a framework for testing and benchmarking the robustness of these ML models. This paper makes the first effort (to our knowledge) to benchmark the robustness of ML models by simulating biological sequences with errors. In this paper, we introduce several ways to perturb SARS-CoV-2 genome sequences to mimic the error profiles of common sequencing platforms such as Illumina and PacBio. We show from experiments on a wide array of ML models that some simulation-based approaches with different perturbation budgets are more robust (and accurate) than others for specific embedding methods to certain noise simulations on the input sequences. Our benchmarking framework may assist researchers in properly assessing different ML models and help them understand the behavior of the SARS-CoV-2 virus or avoid possible future pandemics.

Benchmarking framework for machine learning classification from fNIRS data.

While efforts to establish best practices with functional near infrared spectroscopy (fNIRS) signal processing have been published, there are still no community standards for applying machine learning to fNIRS data. Moreover, the lack of open source benchmarks and standard expectations for reporting means that published works often claim high generalisation capabilities, but with poor practices or missing details in the paper. These issues make it hard to evaluate the performance of models when it comes to choosing them for brain-computer interfaces. We present an open-source benchmarking framework, BenchNIRS, to establish a best practice machine learning methodology to evaluate models applied to fNIRS data, using five open access datasets for brain-computer interface (BCI) applications. The BenchNIRS framework, using a robust methodology with nested cross-validation, enables researchers to optimise models and evaluate them without bias. The framework also enables us to produce useful metrics and figures to detail the performance of new models for comparison. To demonstrate the utility of the framework, we present a benchmarking of six baseline models [linear discriminant analysis (LDA), support-vector machine (SVM), k-nearest neighbours (kNN), artificial neural network (ANN), convolutional neural network (CNN), and long short-term memory (LSTM)] on the five datasets and investigate the influence of different factors on the classification performance, including: number of training examples and size of the time window of each fNIRS sample used for classification. We also present results with a sliding window as opposed to simple classification of epochs, and with a personalised approach (within subject data classification) as opposed to a generalised approach (unseen subject data classification). Results show that the performance is typically lower than the scores often reported in literature, and without great differences between models, highlighting that predicting unseen data remains a difficult task. Our benchmarking framework provides future authors, who are achieving significant high classification scores, with a tool to demonstrate the advances in a comparable way. To complement our framework, we contribute a set of recommendations for methodology decisions and writing papers, when applying machine learning to fNIRS data.

Comparison of energy-efficiency benchmarking methodologies for residential buildings

Benchmarking with defined metrics is an excellent tool to track progress and obtain the desired goals. Benchmarking end-use energy in the residential sector has been a topic of great interest among researchers in recent times. This research analyses the buildings’ energy performance based on a devised benchmarking procedure for residential buildings in the Indian city of Jaipur. Creation of benchmarking system involves collection, analysis and verification of the results obtained from a similar set of data into consideration. This study explores the relationship between the Energy Performance Index (EPI) and with other influencing factors like; Area, end-use appliances (Fridge, AC, Cooler, etc.) to ensure the contribution of each variable towards the buildings’ end-use energy consumption. In this study, information has been collected and analysed from over 2700 houses of Jaipur City. The independent and dependent variables have been identified, and a five-star benchmarking framework has been developed. There are very few studies available investigating the effectiveness of various benchmarking techniques, the domain still lacks a comparative performance study of black-box and gray-box benchmarking techniques. This study has selected two black box approaches and one gray box approach to design a benchmarking model and has compared the energy performance of sample buildings. According to the Pearson and Spearman correlation coefficients, Multiple Linear Regression (MLR) and Bayesian ranking scores are the most consistent for energy benchmarking of the residential building sector. This study proposes a novel implementation of the composite indicator (C.I.) as a platform for designing energy benchmarking tables. Finally, the study concludes with recommendations for future work to employ numerous or hybrid combinations of benchmarking methodologies that are expected to provide a more accurate depiction of the energy performance of buildings.

An Automated Framework for Board-Level Trojan Benchmarking

Economic and operational advantages have led the supply chain of printed circuit boards (PCBs) to incorporate various untrusted entities. Any of the untrusted entities are capable of introducing malicious alterations to facilitate a functional failure or leakage of secret information during field operation. While researchers have been investigating the threat of malicious modification within the scale of individual microelectronic components, the possibility of a board-level malicious manipulation has essentially been unexplored. In the absence of standard benchmarking solutions, prospective countermeasures for PCB trust assurance are likely to utilize homegrown representation of the attacks that undermine their evaluation and do not provide scope for comparison with other techniques. In this article, we have developed a benchmarking solution to facilitate an unbiased and comparable evaluation of countermeasures applicable to PCB trust assurance. Based on a taxonomy tailored for PCB-level alterations, we have developed a toolflow for the automatic generation of Trojan benchmarks to facilitate a comprehensive evaluation against a large number of diverse Trojan implementations and application of data mining for trust verification. Using the toolflow, we have developed a suite of custom “Trojan benchmarks” (i.e., PCB designs with Trojans) containing representative examples of Trojans in the taxonomy inserted in different PCB designs of varying complexity and functionality. Finally, with experimental measurements from a fabricated PCB and structural analysis of netlist, we analyze the stealthiness of the Trojan designs and present the runtime of the tool for a large number of PCB designs.

3-D Heterogeneous Integration of RRAM-Based Compute-In-Memory: Impact of Integration Parameters on Inference Accuracy

Three-dimensional heterogeneous integration (3-D-HI) has been proposed as a potential method to stack a large amount of embedded memory required in state-of-the-art compute-in-memory (CIM) artificial intelligence (AI) accelerators. While embedded nonvolatile memory, such as resistive RAM (RRAM), is a promising alternative to static random access memory (SRAM)/dynamic random access memory (DRAM) as a CIM synaptic device owing to high density, low leakage, and nondestructive read, thermal-induced device conductance drift remains a challenge. High-temperature-driven lower retention can be more significant in dense memory-logic 3-D integration due to increased volumetric power, which has not been studied in prior work. The scope of this work is to quantify the thermal impact of different 3-D-HI architectures on the reliability of 3-D-integrated binary RRAM devices for CIM applications. A device-integration-application reliability evaluation methodology is proposed, using which 3-D integration architectures and logic-memory partitioning configurations are benchmarked. Due to higher junction temperatures for memory tier in both five-tier monolithic 3-D (M3D) and five-tier through silicon via (TSV)-based 3-D compared to the 2-D baseline, the drop in inference accuracy at ten years is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\approx 80$ </tex-math></inline-formula> %. For our assumed device, integration, and application parameters, a three-tier configuration provides a balanced design option between thermal and application performance. The integrated benchmark framework is released on GitHub ( <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/i3dsystems/3D_CIM_thermal_v1.0</uri> ) as an open-source tool for the research community.