Database Benchmark Research Articles

Assessment of Skimmed Milk Flocculation for Bacterial Enrichment from Water Samples, and Benchmarking of DNA Extraction and 16S rRNA Databases for Metagenomics.

Water samples for bacterial microbiome studies undergo biomass concentration, DNA extraction, and taxonomic identification steps. Through benchmarking, we studied the applicability of skimmed milk flocculation (SMF) for bacterial enrichment, an adapted in-house DNA extraction protocol, and six 16S rRNA databases (16S-DBs). Surface water samples from two rivers were treated with SMF and vacuum filtration (VF) and subjected to amplicon or shotgun metagenomics. A microbial community standard underwent five DNA extraction protocols, taxonomical identification with six different 16S-DBs, and evaluation by the Measurement Integrity Quotient (MIQ) score. In SMF samples, the skimmed milk was metabolized by members of lactic acid bacteria or genera such as Polaromonas, Macrococcus, and Agitococcus, resulting in increased relative abundance (p < 0.5) up to 5.0 log fold change compared to VF, rendering SMF inapplicable for bacterial microbiome studies. The best-performing DNA extraction protocols were FastSpin Soil, the in-house method, and EurX. All 16S-DBs yielded comparable MIQ scores within each DNA extraction kit, ranging from 61-66 (ZymoBIOMICs) up to 80-82 (FastSpin). DNA extraction kits exert more bias toward the composition than 16S-DBs. This benchmarking study provided valuable information to inform future water metagenomic study designs.

Developing a Performance Evaluation Benchmark for Event Sourcing Databases

In the domain of software architecture, Event Sourcing (ES) has emerged as a significant paradigm, especially for systems requiring high levels of auditability, traceability, and intricate state management. Systems such as financial transaction platforms, inventory management systems, customer relationship management (CRM) software, and any application requiring a detailed audit trail can significantly benefit from this approach. Numerous aspects of ES remain unexplored, as they have yet to be thoroughly investigated by scientific research. The unique demands of such systems, particularly in terms of database performance and functionality, are not adequately addressed by existing database benchmarks. By establishing benchmarks, organizations can compare different databases to determine which best meets their needs for applications. This aids in selecting the most appropriate technology based on empirical data rather than assumptions or marketing claims.This paper introduces a novel benchmarking framework specifically designed for evaluating databases in the context of event sourcing. The framework addresses critical aspects unique to ES, including event append performance, efficient handling of Projections (separate databases for read operations), strong consistency, ordered data insertion, and robust versioning controls. Through rigorous testing and analysis, this framework aims to fill the gap in existing database benchmarking tools, providing a more accurate and relevant assessment for ES systems. We also conducted experiments that not only demonstrated the effectiveness of our approach but also yielded meaningful results, substantiating its practicality and applicability.

Deduplication Methods Using Levenshtein Distance Algorithm

The study aimed to propose methods to improve the data integrity of the Relational databases such as MS SQL, MySQL and PostgreSQL via record duplication detection. The FODORS and ZAGAT Restaurant database benchmark datasets have been utilized to facilitate the processes involved in preparing and delivering high-quality data. Furthermore, the Levenshtein distance algorithm was used to propose three (3) methods namely: default, eliminating equal string, and knowledge-based libraries to cut duplicate records in the database. In the 70% selected threshold, the average detected duplicate records of 88 out of 112 records in the restaurant dataset. Finally, to efficiently detect duplicate records in the database, depend on the data being analyzed and threshold selected.

FEBench: A Benchmark for Real-Time Relational Data Feature Extraction

As the use of online AI inference services rapidly expands in various applications (e.g., fraud detection in banking, product recommendation in e-commerce), real-time feature extraction (RTFE) systems have been developed to compute the requested features from incoming data tuples in ultra-low latency. Similar to relational databases, these RTFE procedures can be expressed using SQL-like languages. However, there is a lack of research on the workload characteristics and specialized benchmarks for RTFE, especially in comparison with existing database workloads and benchmarks (e.g., concurrent transactions in TPC-C). In this paper, we study the RTFE workload characteristics using over one hundred real datasets from open repositories (e.g. Kaggle, Tianchi, UCI ML, KiltHub) and those from 4Paradigm. The study highlights the significant differences between RTFE workloads and existing database benchmarks in terms of application scenarios, operator distributions, and query structures. Based on these findings, we propose to develop a realtime feature extraction benchmark named FEBench based on the four important criteria for a domain-specific benchmark proposed by Jim Gray. FEBench consists of selected representative datasets, query templates, and an online request simulator. We use FEBench to evaluate the effectiveness of feature extraction systems including OpenMLDB and Flink and find that each system exhibits distinct advantages and limitations in terms of overall latency, tail latency, and concurrency performance.

Benchmarking of protein interaction databases for integration with manually reconstructed signalling network models.

Protein interaction databases are critical resources for network bioinformatics and integrating molecular experimental data. Interaction databases may also enable construction of predictive computational models of biological networks, although their fidelity for this purpose is not clear. Here, we benchmark protein interaction databases X2K, Reactome, Pathway Commons, Omnipath and Signor for their ability to recover manually curated edges from three logic-based network models of cardiac hypertrophy, mechano-signalling and fibrosis. Pathway Commons performed best at recovering interactions from manually reconstructed hypertrophy (137 of 193 interactions, 71%), mechano-signalling (85 of 125 interactions, 68%) and fibroblast networks (98 of 142 interactions, 69%). While protein interaction databases successfully recovered central, well-conserved pathways, they performed worse at recovering tissue-specific and transcriptional regulation. This highlights a knowledge gap where manual curation is critical. Finally, we tested the ability of Signor and Pathway Commons to identify new edges that improve model predictions, revealing important roles of protein kinase C autophosphorylation and Ca2+/calmodulin-dependent protein kinase II phosphorylation of CREB in cardiomyocyte hypertrophy. This study provides a platform for benchmarking protein interaction databases for their utility in network model construction, as well as providing new insights into cardiac hypertrophy signalling. KEY POINTS: Protein interaction databases are used to recover signalling interactions from previously developed network models. The five protein interaction databases benchmarked recovered well-conserved pathways, but did poorly at recovering tissue-specific pathways and transcriptional regulation, indicating the importance of manual curation. We identify new signalling interactions not previously used in the network models, including a role for Ca2+/calmodulin-dependent protein kinase II phosphorylation of CREB in cardiomyocyte hypertrophy.

National Performance Benchmarks for Screening Digital Breast Tomosynthesis: Update from the Breast Cancer Surveillance Consortium.

Background It is important to establish screening mammography performance benchmarks for quality improvement efforts. Purpose To establish performance benchmarks for digital breast tomosynthesis (DBT) screening and evaluate performance trends over time in U.S. community practice. Materials and Methods In this retrospective study, DBT screening examinations were collected from five Breast Cancer Surveillance Consortium (BCSC) registries between 2011 and 2018. Performance measures included abnormal interpretation rate (AIR), cancer detection rate (CDR), sensitivity, specificity, and false-negative rate (FNR) and were calculated based on the American College of Radiology Breast Imaging Reporting and Data System, fifth edition, and compared with concurrent BCSC DM screening examinations, previously published BCSC and National Mammography Database benchmarks, and expert opinion acceptable performance ranges. Benchmarks were derived from the distribution of performance measures across radiologists (n = 84 or n = 73 depending on metric) and were presented as percentiles. Results A total of 896 101 women undergoing 2 301 766 screening examinations (458 175 DBT examinations [median age, 58 years; age range, 18-111 years] and 1 843 591 DM examinations [median age, 58 years; age range, 18-109 years]) were included in this study. DBT screening performance measures were as follows: AIR, 8.3% (95% CI: 7.5, 9.3); CDR per 1000 screens, 5.8 (95% CI: 5.4, 6.1); sensitivity, 87.4% (95% CI: 85.2, 89.4); specificity, 92.2% (95% CI: 91.3, 93.0); and FNR per 1000 screens, 0.8 (95% CI: 0.7, 1.0). When compared with BCSC DM screening examinations from the same time period and previously published BCSC and National Mammography Database performance benchmarks, all performance measures were higher for DBT except sensitivity and FNR, which were similar to concurrent and prior DM performance measures. The following proportions of radiologists achieved acceptable performance ranges with DBT: 97.6% for CDR, 91.8% for sensitivity, 75.0% for AIR, and 74.0% for specificity. Conclusion In U.S. community practice, large proportions of radiologists met acceptable performance ranges for screening performance metrics with DBT. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Lee and Moy in this issue.

A critical examination of robustness and generalizability of machine learning prediction of materials properties

Recent advances in machine learning (ML) have led to substantial performance improvement in material database benchmarks, but an excellent benchmark score may not imply good generalization performance. Here we show that ML models trained on Materials Project 2018 can have severely degraded performance on new compounds in Materials Project 2021 due to the distribution shift. We discuss how to foresee the issue with a few simple tools. Firstly, the uniform manifold approximation and projection (UMAP) can be used to investigate the relation between the training and test data within the feature space. Secondly, the disagreement between multiple ML models on the test data can illuminate out-of-distribution samples. We demonstrate that the UMAP-guided and query by committee acquisition strategies can greatly improve prediction accuracy by adding only 1% of the test data. We believe this work provides valuable insights for building databases and models that enable better robustness and generalizability.

Spintronic leaky-integrate-fire spiking neurons with self-reset and winner-takes-all for neuromorphic computing

Neuromorphic computing using nonvolatile memories is expected to tackle the memory wall and energy efficiency bottleneck in the von Neumann system and to mitigate the stagnation of Moore’s law. However, an ideal artificial neuron possessing bio-inspired behaviors as exemplified by the requisite leaky-integrate-fire and self-reset (LIFT) functionalities within a single device is still lacking. Here, we report a new type of spiking neuron with LIFT characteristics by manipulating the magnetic domain wall motion in a synthetic antiferromagnetic (SAF) heterostructure. We validate the mechanism of Joule heating modulated competition between the Ruderman–Kittel–Kasuya–Yosida interaction and the built-in field in the SAF device, enabling it with a firing rate up to 17 MHz and energy consumption of 486 fJ/spike. A spiking neuron circuit is implemented with a latency of 170 ps and power consumption of 90.99 μW. Moreover, the winner-takes-all is executed with a current ratio >104 between activated and inhibited neurons. We further establish a two-layer spiking neural network based on the developed spintronic LIFT neurons. The architecture achieves 88.5% accuracy on the handwritten digit database benchmark. Our studies corroborate the circuit compatibility of the spintronic neurons and their great potential in the field of intelligent devices and neuromorphic computing.

Hybrid Graph Partitioning with OLB Approach in Distributed Transactions

Online Transaction Processing (OLTP) gets support from data partitioning to achieve better performance and scalability. The primary objective of database and application developers is to provide scalable and reliable database systems. This research presents a novel method for data partitioning and load balancing for scalable transactions. Data is efficiently partitioned using the hybrid graph partitioning method. Optimized load balancing (OLB) approach is applied to calculate the weight factor, average workload, and partition efficiency. The presented approach is appropriate for various online data transaction applications. The quality of the proposed approach is examined using OLTP database benchmark. The performance of the proposed methodology significantly outperformed with respect to metrics like throughput, response time, and CPU utilization.

Exploring Fine-Grained In-Memory Database Performance for Modern CPUs

Modern CPUs keep integrating more cores and large size cache, which is beneficial for in-memory databases to improve parallel processing power and cache locality. While state-of-the-art CPUs have diverse architectures and roadmaps such as large core count and large cache size (AMD x86), moderate core count and cache size (intel x86), large core count and moderate cache size (ARM), exploring in-memory databases performance characteristics for different CPU architectures is important for in-memory database designs and optimizations. In this paper, we develop a fine-grained in-memory database benchmark to evaluate the performance of each operator on different CPUs to explore how CPU hardware architectures influence performance. Different from well known conclusions that more cores and larger cache size can achieve higher performance, we find out that the micro cache architectures play an important role opposite to core count and cache size, the shared monolithic L3 cache with moderate size beats large disaggregated L3 cache. The experiments also show that predicting operator performance on different CPUs is difficult according to diverse CPU architectures and micro cache architectures, and different implementations of each operator are not always high or low with interleaved strong and weak performance regions influenced by CPU hardware architectures. Intel x86 CPUs represent cache-centric processor design, while AMD x86 and ARM CPUs represent computing-centric processor design, the OLAP benchmark experiments of SSB discover that OmniSciDB and OLAP Accelerator with vector-wise processing model performs well on intel x86 CPUs compared to AMD x86 CPUs and the JIT compliant based Hyper prefers to AMD x86 CPUs rather than intel x86 CPUs. The CPU roadmaps of increasing cores or improving cache locality should be considered for in-memory database algorithm design and platform selection.

The LDBC Social Network Benchmark

The Social Network Benchmark's Business Intelligence workload (SNB BI) is a comprehensive graph OLAP benchmark targeting analytical data systems capable of supporting graph workloads. This paper marks the finalization of almost a decade of research in academia and industry via the Linked Data Benchmark Council (LDBC). SNB BI advances the state-of-the art in synthetic and scalable analytical database benchmarks in many aspects. Its base is a sophisticated data generator, implemented on a scalable distributed infrastructure, that produces a social graph with small-world phenomena, whose value properties follow skewed and correlated distributions and where values correlate with structure. This is a temporal graph where all nodes and edges follow lifespan-based rules with temporal skew enabling realistic and consistent temporal inserts and (recursive) deletes. The query workload exploiting this skew and correlation is based on LDBC's "choke point"-driven design methodology and will entice technical and scientific improvements in future (graph) database systems. SNB BI includes the first adoption of "parameter curation" in an analytical benchmark, a technique that ensures stable runtimes of query variants across different parameter values. Two performance metrics characterize peak single-query performance (power) and sustained concurrent query throughput. To demonstrate the portability of the benchmark, we present experimental results on a relational and a graph DBMS. Note that these do not constitute an official LDBC Benchmark Result - only audited results can use this trademarked term.

Towards a Benchmark for Shared Databases [Vision Paper

Traditionally, data has been held in silos and was rarely shared with other organizations. However, recently data sharing across organizations is becoming more and more important as evidenced by governmental and industrial initiatives such as the EU data strategy. As a result, both academia and industry have been proposing new systems for shared databases, that allow multiple organizations to collaboratively insert and manage data in a common database. Yet, each new system seems to come with its own architectural choices and custom guarantees that make it hard for users to navigate the plethora of shared database systems. While standard benchmarks like the TPC‑C database benchmark have been a well-established tool to compare and analyze traditional database systems, they seem to be unsuited to evaluate shared database systems. This is because these systems are built with fundamentally different assumptions in mind, such as a different threat/trust model since multiple (untrusted) parties access and modify the same data. In this paper, we present a vision and initial ideas for a new benchmark to evaluate shared databases and capture their unique characteristics.

An automated, modular system for organic waste utilization using Hermetia illucens larvae: Design, sustainability, and economics

Large amounts of food are wasted, and valuable contents are not utilized completely. Processing such wastes into biomass of defined composition is possible using insects. However, automation and decentralization of insect-based processes are necessary for certain applications. This study presents a modular design for rearing larvae of the black soldier fly, Hermetia illucens. A life cycle and economic assessment are carried out to check whether the process may be viable. A sales price of 3.55 € per kg of the product would make it profitable within five years. Production of 1 kg of dried larvae would be associated with 2.77 kg CO2 eq emitted, 55.24 MJ of non-renewable energy use, and occupation of 0.68 m2 of organic arable land. Per kg protein, the insect biomass appears more sustainable than database benchmarks. These results indicate that even small-scale insect production processes have sustainability benefits when using food waste.

Optical disk segmentation in human retina images with golden eagle optimizer

In ophthalmology, examination of the optic nerve and disc of the retina is a common way to diagnose a variety of eye diseases such as aging, diabetes, optic neuritis, glaucoma, papilledema, and ischemic optic neuropathy. This study presents the implementation of three combined methods based on the Golden Eagle algorithm, backup vector machine and geometrically active contour method and image segmentation. These methods have been tested using the well-known retina database benchmark, RIM-ONE. This study uses preprocessing based on the geometrically active Shannon contour guided by the Golden Eagle algorithm and post-processing based on regular spaced cross-section segmentation. The Golden Eagle method is more efficient than approaches such as the geometrically active contour and the SVM method. The proposed method, compared to other methods, shows higher mean values in terms of image similarity and statistical index. It is confirmed that the proposed Golden Eagle method could be used in the future to assess the clinical fit of retinal images.

A mapping algorithm for International Classification of Diseases 10th Revision codes for congenital heart surgery benchmark procedures.

Administrative billing data are critical to many initiatives in congenital heart surgery. Mapping algorithms for International Classification of Disease, 10th Revision diagnosis and procedure codes to clinical registry procedure definitions will allow identification of surgical cases and account for patient and procedural factors within administrative data. Our objectives were to develop mapping logic to crosswalk International Classification of Disease, 10th Revision procedure codes to 10 Society of Thoracic Surgeons Congenital Heart Surgery Database benchmark and beta-test the algorithm. Patients undergoing Society of Thoracic Surgeons Congenital Heart Surgery Database benchmark procedures from 2015 to 2019 were identified and served as the gold standard. Cases were linked on direct identifiers to cases from the Pediatric Health Information System Database. Two independent teams developed International Classification of Disease, 10th Revision-based algorithms for cases capture. Algorithms were compared and iteratively refined to optimize sensitivity and specificity. Operative mortalities for cases identified in the administrative versus registry data were compared. Overall sensitivity was 91% and specificity was 99% for capture of benchmark operations using International Classification of Diseases 10th Revision codes. Sensitivity was more than 90% in identifying 6 of the 10 individual benchmark procedures and more than 98% sensitive in identifying Fontan, Glenn, and arterial switch with ventricular septal defect procedures. Specificity was more than 98% for all benchmark operations. There were no statistical differences in operative mortality between cases identified in the administrative versus the registry data. Novel mapping algorithm for International Classification of Disease, 10th Revision procedure codes enables identification of congenital heart benchmark procedures within administrative billing data. This crosswalk facilitates population-based congenital heart surgical research and quality assessment.

An Evasion Attack against Stacked Capsule Autoencoder

Capsule networks are a type of neural network that use the spatial relationship between features to classify images. By capturing the poses and relative positions between features, this network is better able to recognize affine transformation and surpass traditional convolutional neural networks (CNNs) when handling translation, rotation, and scaling. The stacked capsule autoencoder (SCAE) is a state-of-the-art capsule network that encodes an image in capsules which each contain poses of features and their correlations. The encoded contents are then input into the downstream classifier to predict the image categories. Existing research has mainly focused on the security of capsule networks with dynamic routing or expectation maximization (EM) routing, while little attention has been given to the security and robustness of SCAEs. In this paper, we propose an evasion attack against SCAEs. After a perturbation is generated based on the output of the object capsules in the model, it is added to an image to reduce the contribution of the object capsules related to the original category of the image so that the perturbed image will be misclassified. We evaluate the attack using an image classification experiment on the Mixed National Institute of Standards and Technology Database (MNIST), Fashion-MNIST, and German Traffic Sign Recognition Benchmark (GTSRB) datasets, and the average attack success rate can reach 98.6%. The experimental results indicate that the attack can achieve high success rates and stealthiness. This finding confirms that the SCAE has a security vulnerability that allows for the generation of adversarial samples. Our work seeks to highlight the threat of this attack and focus attention on SCAE’s security.

Next-Generation Database Benchmark for Financial Scenarios

As the major financial entity in China, banks have high performance and security requirements for databases and data service solutions. With the progression of data application services in banking, the data types and business scenarios become more diverse, and it is difficult for users to make optimal choices among a wide diversity of database products and data service solutions. In combination with the data application demands of the financial industry, this study comprehensively analyzes the current status of applications of databases in banking, particularly the status and challenges of database localization in recent years, by using literature research and theoretical analysis. In addition, we systematically investigate the database benchmarks of China and other countries, and further prospect the necessity and importance of constructing next-generation database benchmarks for financial scenarios. We find that current database benchmarks have many deficiencies and face various challenges in dealing with the database testing in financial scenarios owing to the complex business logic, diverse data patterns, and high security requirements. Therefore, to build a next-generation database benchmark that can meet the requirements of financial scenarios, we propose several suggestions to address these challenges, which involve the aspects of workloads, data schemes, metrics, and technical architecture.

Improving the Ambient Intelligence Living Using Deep Learning Classifier

Over the last decade, there is a surge of attention in establishing ambient assisted living (AAL) solutions to assist individuals live independently. With a social and economic perspective, the demographic shift toward an elderly population has brought new challenges to today’s society. AAL can offer a variety of solutions for increasing people’s quality of life, allowing them to live healthier and more independently for longer. In this paper, we have proposed a novel AAL solution using a hybrid bidirectional long-term and short-term memory networks (BiLSTM) and convolutional neural network (CNN) classifier. We first pre-processed the signal data, then used time-frequency features such as signal energy, signal variance, signal frequency, empirical mode, and empirical mode decomposition. The convolutional neural network-bidirectional long-term and short-term memory (CNN-biLSTM) classifier with dimensional reduction isomap algorithm was then used to select ideal features. We assessed the performance of our proposed system on the publicly accessible human gait database (HuGaDB) benchmark dataset and achieved an accuracy rates of 93.95 percent, respectively. Experiments reveal that hybrid method gives more accuracy than single classifier in AAL model. The suggested system can assists persons with impairments, assisting carers and medical personnel.

CIMAR, NIMAR, and LMMA: Novel algorithms for thread and memory migrations in user space on NUMA systems using hardware counters

This paper introduces two novel algorithms for thread migrations, named CIMAR (Core-aware Interchange and Migration Algorithm with performance Record –IMAR–) and NIMAR (Node-aware IMAR), and a new algorithm for the migration of memory pages, LMMA (Latency-based Memory pages Migration Algorithm), in the context of Non-Uniform Memory Access (NUMA) systems. This kind of system has complex memory hierarchies that present a challenging problem in extracting the best possible performance, where thread and memory mapping play a critical role. The presented algorithms gather and process the information provided by hardware counters to make decisions about the migrations to be performed, trying to find the optimal mapping. They have been implemented as a user space tool that looks for improving the system performance, particularly in, but not restricted to, scenarios where multiple programs with different characteristics are running. This approach has the advantage of not requiring any modification on the target programs or the Linux kernel while keeping a low overhead.Two different benchmark suites have been used to validate our algorithms: The NAS parallel benchmark, mainly devoted to computational routines, and the LevelDB database benchmark focused on read–write operations. These benchmarks allow us to illustrate the influence of our proposal in these two important types of codes. Note that those codes are state-of-the-art implementations of the routines, so few improvements could be initially expected. Experiments have been designed and conducted to emulate three different scenarios: a single program running in the system with full resources, an interactive server where multiple programs run concurrently varying the availability of resources, and a queue of tasks where granted resources are limited. The proposed algorithms have been able to produce significant benefits, especially in systems with higher latency penalties for remote accesses. When more than one benchmark is executed simultaneously, performance improvements have been obtained, reducing execution times up to 60%. In this kind of situation, the behaviour of the system is more critical, and the NUMA topology plays a more relevant role. Even in the worst case, when isolated benchmarks are executed using the whole system, that is, just one task at a time, the performance is not degraded.

Copy-and-patch compilation: a fast compilation algorithm for high-level languages and bytecode

Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers. We present copy-and-patch, an extremely fast compilation technique that also produces good quality code. It is capable of lowering both high-level languages and low-level bytecode programs to binary code, by stitching together code from a large library of binary implementation variants. We call these binary implementations stencils because they have holes where missing values must be inserted during code generation. We show how to construct a stencil library and describe the copy-and-patch algorithm that generates optimized binary code. We demonstrate two use cases of copy-and-patch: a compiler for a high-level C-like language intended for metaprogramming and a compiler for WebAssembly. Our high-level language compiler has negligible compilation cost: it produces code from an AST in less time than it takes to construct the AST. We have implemented an SQL database query compiler on top of this metaprogramming system and show that on TPC-H database benchmarks, copy-and-patch generates code two orders of magnitude faster than LLVM -O0 and three orders of magnitude faster than higher optimization levels. The generated code runs an order of magnitude faster than interpretation and 14% faster than LLVM -O0. Our WebAssembly compiler generates code 4.9X-6.5X faster than Liftoff, the WebAssembly baseline compiler in Google Chrome. The generated code also outperforms Liftoff's by 39%-63% on the Coremark and PolyBenchC WebAssembly benchmarks.