Industry Standard Benchmark Research Articles

Investigation into the Mechanical Properties of Commercial Sandcrete Blocks Produced in Nigeria: A Case Study of Warri Metropolis

This study investigated the extent to which the mechanical strength of commercially produced sandcrete blocks in Delta State, Nigeria, conforms to recognized international standards. Fifty sandcrete blocks (28 curing days) were sampled from ten known sandcrete block manufacturers in Warri municipal area, five blocks from each manufacturer. The compressive strength of the blocks was determined in accordance with the American Society for Testing and Materials (ASTM) International guidelines. The results showed that the compressive strength of all the blocks failed to meet the Nigeria Industrial Standard (NIS) benchmark for load loading sandcrete block, while only 20% of the blocks attained NIS requirement for non-load bearing walls. Field observation revealed that poor mixing ratio of the sandcrete; inconsistencies in the batching method and sand used in the production of the sandcrete blocks were responsible for the poor quality (low compressive strength) of the sandcrete blocks manufactured in the region. The findings of this study underscore the importance for the Nigerian Institution of Structural Engineers, to rigorously monitor the strength of sandcrete blocks made in Nigeria to ensure that their quality meets internationally recognized standards. This will reduce the occurrence of structural damage in Nigeria, due to the use of low quality blocks for wall constriction.

Flexible Resource Allocation for Relational Database-as-a-Service

Oversubscription is an essential cost management strategy for cloud database providers, and its importance is magnified by the emerging paradigm of serverless databases. In contrast to general purpose techniques used for oversubscription in hypervisors, operating systems and cluster managers, we develop techniques that leverage our understanding of how DBMSs use resources and how resource allocations impact database performance. Our techniques are designed to flexibly redistribute resources across database tenants at the node and cluster levels with low overhead. We have implemented our techniques in a commercial cloud database service: Azure SQL Database. Experiments using microbenchmarks, industry-standard benchmarks and real-world resource usage traces show that using our approach, it is possible to tightly control the impact on database performance even with a relatively high degree of oversubscription.

TPCx-AI - An Industry Standard Benchmark for Artificial Intelligence and Machine Learning Systems

Artificial intelligence (AI) and machine learning (ML) techniques have existed for years, but new hardware trends and advances in model training and inference have radically improved their performance. With an ever increasing amount of algorithms, systems, and hardware solutions, it is challenging to identify good deployments even for experts. Researchers and industry experts have observed this challenge and have created several benchmark suites for AI and ML applications and systems. While they are helpful in comparing several aspects of AI applications, none of the existing benchmarks measures end-to-end performance of ML deployments. Many have been rigorously developed in collaboration between academia and industry, but no existing benchmark is standardized. In this paper, we introduce the TPC Express Benchmark for Artificial Intelligence (TPCx-AI), the first industry standard benchmark for end-to-end machine learning deployments. TPCx-AI is the first AI benchmark that represents the pipelines typically found in common ML and AI workloads. TPCx-AI provides a full software kit, which includes data generator, driver, and two full workload implementations, one based on Python libraries and one based on Apache Spark. We describe the complete benchmark and show benchmark results for various scale factors. TPCx-AI's core contributions are a novel unified data set covering structured and unstructured data; a fully scalable data generator that can generate realistic data from GB up to PB scale; and a diverse and representative workload using different data types and algorithms, covering a wide range of aspects of real ML workloads such as data integration, data processing, training, and inference.

Big data in oncology: Challenges and solutions.

6513 Background: Oncology care is complex and often multimodal. With recent technological advances, only a fraction of data is structured feasibly for research. Here we present a step-by-step method of building a novel comprehensive pan-cancer oncology data model using standard data definitions and industry-standard benchmarks. Methods: A team of 133 members was assembled including a project manager, bioinformatic engineers, business analysts, biostatisticians, data stewardship experts, clinical curators, and quality assurance (QA) managers. We first identified data domains that capture a comprehensive patient journey, leveraging existing oncology data models as a starting point, including NAACCR, PRISSMM (Deb Schrag & Eva Lepisto), and mCODE (ASCO). A common data model was developed using standard terms plus 5-10 disease specific elements (DSE). REDCap was used as the database platform, as it is HIPAA compliant and allows customizations. The data was stored in AuroraDB using an architecture and products that provide scalability from both an integration and consumption perspective. Results: We identified 10 data domains, including 186 distinct elements: demographics (20), comorbidities (2); cancer diagnosis & staging (27), pathology (45), imaging (18), medications (11), oncology responses (11), radiation treatments (14), cancer surgeries (11), cancer genomic (19), tumor markers (8), and vitals (8). Standard ontologies were used, including ICD-0-3 histology codes, ICD-10 comorbidities codes, CPT cancer surgeries codes, and CTCAE 5.0 for toxicities. We identified a data steward for each tumor type across medical oncology, surgery, pathology, radiology, and radiation oncology domains who aided curator training and the identification of DSE. QA managers and analysts performed 20% source data verification. In addition, we built REDCap rules (applicable across a form), and complex queries (applicable across multiple forms). To support QA and clinical engagement, interactive Tableau dashboards were constructed. In addition, timing and quality errors were monitored via Tableau dashboards at the individual curator level to provide timely feedback, leading to improved data quality and curation efficiency in real time. The Medical oncology and radiology domains were the most time-consuming, whereas Cancer diagnosis was the most difficult to curate. Conclusions: We collected genomic and phenomic data for 15,579 patients across six tumor types to date. Collecting comprehensive oncology data across tumor types is possible but requires institutional support, collaboration between clinical & informatics teams, and a dedicated QA team. [Table: see text]

Quantification of Chemical Composition Distribution of Polyolefin Materials for Improved Accuracy and Speed

Crystallization-based separation techniques are widely used for chemical composition distribution (CCD, also often referred as short chain branching distribution SCBD) measurement in the polyolefin industry, in which CCD is among the most critical structural parameters to define their properties and applications. The desire to improve the sample throughput rate by shortening the analysis time in the current CCD techniques would normally sacrifice measurement accuracy due to the cocrystallization phenomena, in which the polymer chains with similar molecular architectures will cocrystallize or coelute enigmatically in the CCD analysis. Herein, we reported the investigation by exploring various separation packing materials on the cocrystallization phenomena. In comparison with the commercially available Crystallization Elution Fractionation (CEF) technique, an improved technique for CCD measurement (iCCD) was developed using the columns packed with gold-coated nickel substrate or spherical gold particles with or without the dynamic cooling process. With iCCD, it was observed with equivalent cocrystallization degree to the industrial gold standard benchmark Wild-TREF method but with over 70+ times improved sample throughput rate (62 vs 4500 min). The experimental CCD chromatogram for a polyolefin blend similarly assembled to the mathematically constructed result further demonstrated the minimized cocrystallization and measurement accuracy in the iCCD method. In addition, the superior mechanical properties of gold-coated nickel particles enabled the long-term robustness and enhanced measurement precision of the iCCD technique during the repeated thermal treatment cycles and pressure variations.

Explaining the Unseen: Leveraging XAI to Enhance the Trustworthiness of Black-Box Models in Performance Testing

This paper presents a novel approach to enhance the trustworthiness of black-box and interpretable models in performance testing through the application of Explainable Artificial Intelligence (XAI). The proposed approach utilizes the Shapley Additive Explanation (SHAP) algorithm as a surrogate model, enabling performance analysts to gain insights into the decision-making process of black-box machine learning models. By incorporating SHAP around black-box models, analysts can obtain explanations regarding the pass or fail status of a test, as well as identify the relative significance of performance data to the machine learning models. To evaluate the effectiveness of the proposed approach, load testing experiments were conducted on a real testbed, employing industry-standard benchmarks and manually injected performance bugs. The results demonstrate that the proposed approach significantly enhances the trustworthiness of machine learning models by providing interpretable explanations for their decision-making. Furthermore, the approach showcases its versatility across domains and requires minimal effort to operate, making it highly applicable in various contexts.

A modeling approach for estimating performance and energy consumption of storage systems

Improvements in data storage may be constrained by the lower performance of hard disk drives (HDD) and the higher cost per gigabyte of solid-state drives (SSD). To mitigate these issues, hybrid storage architectures have been conceived. Some works evaluate the performance of storage architectures, but energy consumption is usually neglected and not simultaneously evaluated with performance. This paper presents an approach based on generalized stochastic Petri nets (GSPN) for performance and energy consumption evaluation of individual and hybrid storage systems. The proposed models can represent distinct workloads and also estimate throughput, response time and energy consumption of storage systems. Experiments based on industry-standard benchmarks are adopted to demonstrate the feasibility of the proposed approach.

A Comparative Study on minimum skew Clock tree distribution algorithms for high-speed Digital Integrated Circuits

In this paper an extensive analysis on clock tree synthesis algorithms are carried out with conventional cts, Mesh based cts, Meshless Multi-source and flexible H-tree multisource cts with mesh. The algorithms are analysed with industry standard placement benchmarks. Latency, clock power, wirelength, number of buffers, and clock skew are observed. The clock distribution algorithm which provides a single digit least skew is picked, the pros and cons of that algorithm is elaborately discussed.

MONet: heterogeneous Memory over Optical Network for large-scale data center resource disaggregation

The memory over optical network (MONet) system is a disaggregated data center architecture where serial (HMC)/parallel (DDR4) memory resources can be accessed over optically switched interconnects within and between racks. An FPGA/ASIC-based custom hardware IP (ReMAT) supports heterogeneous memory pools, accommodates optical-to-electrical conversion for remote access, performs the required serial/parallel conversion, and hosts the necessary local memory controller. An optically interconnected HMC-based (serial I/O type) memory card is accessed by a memory controller embedded in the compute card, simplifying the hardware near the memory modules. This substantially reduces overheads on latency, cost, power consumption, and space. We characterize CPU–memory performance by experimentally demonstrating the impact of distance, number of switching hops, transceivers, channel bonding, and bit rate per transceiver on the bit error rate, power consumption, additional latency, sustained remote memory bandwidth/throughput (using industry standard benchmark STREAMS), and cloud workload performance (such as operations per second, average added latency, and retired instructions per second memcached with YCSB cloud workloads). MONet pushes the CPU–memory operational limit from a few centimeters to tens of meters, yet applications can experience as low as 10% performance penalty (at 36 m) compared to a direct-attached equivalent. Using the proposed parallel topology, a system can support up to 100,000 disaggregated cards.

Perceptron based filtering of futile prefetches in embedded VLIW DSPs

Abstract Prefetching has been widely used in general purpose computers, especially in high performance CPUs, but has been much less used in DSPs and embedded processors. The goal of this paper is to investigate adding a perceptron, a single layer neural network [1], to straight-forward hardware prefetching techniques for embedded DSPs, and assess their performance improvement. By using industry standard benchmarks we come to the conclusion that the use of a perceptron to improve the prefetch decision making significantly reduces the number of accesses from external memory via shared buses without any significant performance impact. We show that using a perceptron in addition to a prefetch mechanism results in a reduction of 70% on average in the number of accesses to external memory relative to accesses performed using the prefetch mechanism alone. The bandwidth reduction is achieved without any significant performance loss.

Speeding up AutoTuning of the Memory Management Options in Data Analytics

Many solutions used towards building autonomous (or, self-driving) data processing systems today are trying to leverage the “black box” algorithm of Bayesian Optimization (BO) both due to its wider applicability and the theoretical guarantees provided on the quality of results produced. The black-box approach, however, could be time and labor-intensive; or otherwise get stuck in a local minima. We study an important problem of auto-tuning the memory allocation for applications running on modern distributed data processing systems. A simple “white-box” model is developed which can quickly separate good configurations from bad ones. To combine the benefits of the two approaches to tuning, we build a framework called Guided Bayesian Optimization (GBO) that uses the white-box model as a guide during the Bayesian Optimization exploration process. An evaluation carried out on Apache Spark using industry-standard benchmark applications shows that GBO consistently provides performance speedups across the application workload with the magnitude of savings being close to 2x.

CLOTHO: directed test generation for weakly consistent database systems

Relational database applications are notoriously difficult to test and debug. Concurrent execution of database transactions may violate complex structural invariants that constraint how changes to the contents of one (shared) table affect the contents of another. Simplifying the underlying concurrency model is one way to ameliorate the difficulty of understanding how concurrent accesses and updates can affect database state with respect to these sophisticated properties. Enforcing serializable execution of all transactions achieves this simplification, but it comes at a significant price in performance, especially at scale, where database state is often replicated to improve latency and availability. To address these challenges, this paper presents a novel testing framework for detecting serializability violations in (SQL) database-backed Java applications executing on weakly-consistent storage systems. We manifest our approach in a tool, CLOTHO, that combines a static analyzer and model checker to generate abstract executions, discover serializability violations in these executions, and translate them back into concrete test inputs suitable for deployment in a test environment. To the best of our knowledge, CLOTHO, is the first automated test generation facility for identifying serializability anomalies of Java applications intended to operate in geo-replicated distributed environments. An experimental evaluation on a set of industry-standard benchmarks demonstrates the utility of our approach.

Efficient suspect selection in unreachable state diagnosis

In the modern hardware design cycle, correcting the design when verification reveals a state to be erroneously unreachable can be a time-consuming manual process. Recently-developed algorithms aid the engineer in finding the root cause of the failure in these cases. However, they exhaustively examine every design location to determine a set of possible root causes, potentially requiring substantial runtime. This work develops a novel approach that is applicable to practical diagnosis problems. In contrast to previous approaches, it considers only a portion of the design locations but still finds the complete solution set to the problem. The presented approach proceeds through a series of iterations, each considering a strategically-chosen subset of the design locations (a suspect set) to determine if they are root causes. The results of each iteration inform the choice of suspect set for the next iteration. By choosing the first iteration’s suspect set appropriately, the algorithm is able to find the complete solution set to the problem. Empirical results on industrial designs and standard benchmark designs demonstrate a 15x speedup compared to the previous approach, while considering only 18.7% of the design locations as suspects.

On-line Optimization of Power Efficiency in 3D Multicore Processors

This paper considers the problem of maximizing power efficiency in multi-core processors by dynamic voltage frequency scaling. We formulate the problem in an on-line setting and apply to it a gradient-based stochastic approximation algorithm. We use a cycle-level, full-system simulation platform capable of realistic modeling of state-of-the-art microarchitectures executing industry-standard benchmarks (Splash 2). The optimization algorithm computes the gradients by the Infinitesimal Perturbation Analysis (IPA) sample-based sensitivity analysis technique. Despite the fact that the system’s characteristics vary widely with time and hence the optimum point varies as well, the simulation experiments exhibit a tracking of the optimal values. To our knowledge this is the first application of IPA to co-optimizing throughput performance and power in computer processors.

WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction\u2014a model-driven approach for session-based application systems

The specification of workloads is required in order to evaluate performance characteristics of application systems using load testing and model-based performance prediction. Defining workload specifications that represent the real workload as accurately as possible is one of the biggest challenges in both areas. To overcome this challenge, this paper presents an approach that aims to automate the extraction and transformation of workload specifications for load testing and model-based performance prediction of session-based application systems. The approach (WESSBAS) comprises three main components. First, a system- and tool-agnostic domain-specific language (DSL) allows the layered modeling of workload specifications of session-based systems. Second, instances of this DSL are automatically extracted from recorded session logs of production systems. Third, these instances are transformed into executable workload specifications of load generation tools and model-based performance evaluation tools. We present transformations to the common load testing tool Apache JMeter and to the Palladio Component Model. Our approach is evaluated using the industry-standard benchmark SPECjEnterprise2010 and the World Cup 1998 access logs. Workload-specific characteristics (e.g., session lengths and arrival rates) and performance characteristics (e.g., response times and CPU utilizations) show that the extracted workloads match the measured workloads with high accuracy.

GMU: Genuine Multiversion Update-Serializable Partial Data Replication

In this article we introduce GMU, a genuine partial replication protocol for transactional systems, which exploits an innovative, highly scalable, distributed multiversioning scheme. Unlike existing multiversion-based solutions, GMU does not rely on any global logical clock, which may represent a contention point and a major impairment to system scalability. Also, GMU never aborts read-only transactions and spares them from undergoing distributed validation schemes. This makes GMU particularly efficient in presence of read-intensive workloads, as typical of a wide range of real-world applications. GMU guarantees the Extended Update Serializability (EUS) isolation level. This consistency criterion is particularly attractive as it is sufficiently strong to ensure correctness even for very demanding applications (such as TPC-C), but is also weak enough to allow efficient and scalable implementations, such as GMU. Further, unlike several relaxed consistency models proposed in literature, EUS shows simple and intuitive semantics, thus being an attractive consistency model for ordinary programmers. We integrated GMU in a popular open source in-memory transactional data grid, namely Infinispan. On the basis of a wide experimental study performed on heterogeneous platforms and using industry standard benchmarks (namely TPC-C and YCSB), we show that GMU achieves almost linear scalability and that it introduces reduced overhead, with respect to solutions ensuring non-serializable semantics, in a wide range of workloads.

Greener Data Exchange in the Cloud: A Coding-Based Optimization for Big Data Processing

The rise of the cloud and distributed data-intensive (big data) applications puts pressure on data center networks due to the movement of massive volumes of data. Reducing the volume of communication is pivotal for embracing greener data exchange by efficient utilization of network resources. This paper proposes the use of mixing technique, spate coding , working in tandem with software-defined network control as a means of dynamically-controlled reduction in volume of communication. We introduce motivating real-world use-cases, and present a novel spate coding algorithm for the data center networks. We also analyze the computational complexity of the general problem of minimizing the volume of communication in a distributed data center application without degrading the rate of information exchange, and provide theoretical limits of such schemes. Moreover, we proceed to bridge the gap between theory and practice by performing a proof-of-concept implementation of the proposed system in a real world data center. We use Hadoop MapReduce, the most widely used big data processing framework, as our target. The experimental results employing two of industry standard benchmarks show the advantage of our proposed system compared to a vanilla Hadoop implementation , an in-network combiner , and Combine-N-Code . The proposed coding-based scheme shows performance improvement in terms of volume of communication (up to 62%), goodput (up to 76%), disk utilization (up to 38%), and the number of bits that can be transmitted per Joule of energy (up to 200%).

Scalable transactions in cloud data stores

Cloud Computing is a successful paradigm for deploying scalable and highly available web applications at low cost. In real life scenarios, the applications are expected to be scalable and consistent. Data partitioning is a commonly used technique for improving scalability. Traditional horizontal partitioning techniques are not capable of tracking the data access patterns of web applications. The development of novel, scalable workload-driven data partitioning is a requirement for improving scalability. This paper proposes a novel workload-aware approach, with scalable workload-driven data partitioning based on data access patterns of web applications for transaction processing. It is specially designed to scale out using NoSQL data stores. In contrast to the existing static approaches, this approach offers high throughput, lower response time, and a less number of distributed transactions. Further, implementation and validation of scalable workload-driven partitioning scheme is carried out through experimentation over cloud data stores such as Hadoop HBase and Amazon SimpleDB. An experimental results of the concerned partitioning scheme is conducted using the industry standard TPC-C benchmark. Analytical and experimental results are observed and it shows that scalable workload-driven data partitioning outperforms the schema level and graph partitioning in terms of throughput, response time and distributed transactions.

Incorporating hardware and software features into a prediction model for processor‐system throughput

ABSTRACTThe cycle‐accurate simulation is a method for design space study of a processor system before it goes for the hardware implementation. Even though the simulations provide precise results about the system performance, the simulation times are exorbitantly high for practical systems. Therefore, an alternative is to use experimentally‐ developed models that tend to be faster than the aforementioned simulations. This is an extension to our previous work done in the area of neural network models for assessing the processor system throughput. In addition to hardware‐related parameters, the current work includes multiple software‐related parameters to better represent the dynamic behavior of programs. Consequently, the model provides higher accuracy estimates of a widely used performance metric (instructions per cycle) when tested with industry standard CPU benchmark programs. Potential uses of the model are compiler design and computer architecture research and teaching. © 2015 Wiley Periodicals, Inc. Comput Appl Eng Educ 24:169–179, 2016; View this article online at wileyonlinelibrary.com/journal/cae; DOI 10.1002/cae.21681.

Optimization of common table expressions in MPP database systems

Big Data analytics often include complex queries with similar or identical expressions, usually referred to as Common Table Expressions (CTEs). CTEs may be explicitly defined by users to simplify query formulations, or implicitly included in queries generated by business intelligence tools, financial applications and decision support systems. In Massively Parallel Processing (MPP) database systems, CTEs pose new challenges due to the distributed nature of query processing, the overwhelming volume of underlying data and the scalability criteria that systems are required to meet. In these settings, the effective optimization and efficient execution of CTEs are crucial for the timely processing of analytical queries over Big Data. In this paper, we present a comprehensive framework for the representation, optimization and execution of CTEs in the context ofOrca-- Pivotal's query optimizer for Big Data. We demonstrate experimentally the benefits of our techniques using industry standard decision support benchmark.