Comprehensive Benchmark Research Articles

Fastlin: an ultra-fast program for Mycobacterium tuberculosis complex lineage typing.

Fastlin is a bioinformatics tool designed for rapid Mycobacterium tuberculosis complex (MTBC) lineage typing. It utilizes an ultra-fast alignment-free approach to detect previously identified barcode single nucleotide polymorphisms associated with specific MTBC lineages. In a comprehensive benchmarking against existing tools, fastlin demonstrated high accuracy and significantly faster running times. fastlin is freely available at https://github.com/rderelle/fastlin and can easily be installed using Conda.

Fast and accurate electromagnetic field calculation for substrate-supported metasurfaces using the discrete dipole approximation

Abstract Metasurface design tends to be tedious and time-consuming based on sweeping geometric parameters. Common numerical simulation techniques are slow for large areas, ultra-fine grids, and/or three-dimensional simulations. Simulation time can be reduced by combining the principle of the discrete dipole approximation (DDA) with analytical solutions for light scattered by a dipole near a flat surface. The DDA has rarely been used in metasurface design, and comprehensive benchmarking comparisons are lacking. Here, we compare the accuracy and speed of three DDA methods—substrate discretization, two-dimensional Cartesian Green’s functions, and one-dimensional (1D) cylindrical Green’s functions—against the finite difference time domain (FDTD) method. We find that the 1D cylindrical approach performs best. For example, the s-polarized field scattered from a silica-substrate-supported 600 × 180 × 60 nm gold elliptic nanocylinder discretized into 642 dipoles is computed with 0.78 % pattern error and 6.54 % net power error within 294 s, which is 6 times faster than FDTD. Our 1D cylindrical approach takes advantage of parallel processing and also gives transmitted field solutions, which, to the best of our knowledge, is not found in existing tools. We also examine the differences among four polarizability models: Clausius–Mossotti, radiation reaction, lattice dispersion relation, and digitized Green’s function, finding that the radiation reaction dipole model performs best in terms of pattern error, while the digitized Green’s function has the lowest power error.

Benchmarking a new 2.5D shallow water model for lava flows

Lava flows associated with effusive volcanic eruptions require accurate modelling in order to forecast potential paths of destruction.This study presents a new depth-averaged model that overcomes the classical shallow water hypothesis by incorporating several enhancements, allowing for a more precise representation of the flow dynamics and behaviour: (i) a parabolic profile which captures the vertical variations in velocity within the flow; (ii) a non-constant vertical profile for temperature, enabling a more realistic representation of thermal gradients within the flowing lava; (iii) a viscoplastic temperature-dependent viscosity model to account for the non-Newtonian behaviour of lava; (iv) a transport equation for temperature accounting for the thermal heat exchanges with the environment and the soil. The first two modifications allow us to describe, under reasonable assumptions, the vertical structure of the flow, and for this reason, we put our model in the class of 2.5D models.To assess the performance of our modified model, comprehensive benchmark tests are conducted using both laboratory experiments and real-world lava flow data related to the 2014–2015 Pico do Fogo, Cape Verde, effusive eruption. The benchmarking analysis demonstrates that this model accurately reproduces, with short execution times, essential flow features such as flow front advancement and cooling processes.

Catalyst: Fast and flexible modeling of reaction networks.

We introduce Catalyst.jl, a flexible and feature-filled Julia library for modeling and high-performance simulation of chemical reaction networks (CRNs). Catalyst supports simulating stochastic chemical kinetics (jump process), chemical Langevin equation (stochastic differential equation), and reaction rate equation (ordinary differential equation) representations for CRNs. Through comprehensive benchmarks, we demonstrate that Catalyst simulation runtimes are often one to two orders of magnitude faster than other popular tools. More broadly, Catalyst acts as both a domain-specific language and an intermediate representation for symbolically encoding CRN models as Julia-native objects. This enables a pipeline of symbolically specifying, analyzing, and modifying CRNs; converting Catalyst models to symbolic representations of concrete mathematical models; and generating compiled code for numerical solvers. Leveraging ModelingToolkit.jl and Symbolics.jl, Catalyst models can be analyzed, simplified, and compiled into optimized representations for use in numerical solvers. Finally, we demonstrate Catalyst's broad extensibility and composability by highlighting how it can compose with a variety of Julia libraries, and how existing open-source biological modeling projects have extended its intermediate representation.

Optimizing polypharmacy management in the elderly: a comprehensive European benchmarking survey and the development of an innovative online benchmarking application.

Background: Polypharmacy, defined as the simultaneous use of multiple medications by a patient, is a worldwide problem of rising prevalence. Paving the way for drug interactions, adverse drug reactions and non-adherence, it leads to negative health outcomes, increased use of healthcare services and rising costs. Since it is closely related to multimorbidity, it peaks in older adults. So far, not many polypharmacy management programs in the elderly have been introduced in practice. However, due to the rapid ageing of European societies, there is an urgent need to implement them more widely. Objective: The aim of this study was to benchmark polypharmacy management programs in the elderly available in Europe and creating a dedicated benchmarking application. Methods: It was a cross-sectional study based on an online survey targeting healthcare professionals and other stakeholders across European countries. Data collected in the survey were reused to design an online benchmarking application. Results: As many as 911 respondents from all but two EU countries took part in this study. Out of the survey participants, 496 (54.4%) reported availability of various activities or formal programs targeting polypharmacy in the elderly that were known to them. These programs had multiple goals, of which improved patient safety was indicated as the most common objective (65.1% of the cases). The most typical settings for such programs was primary care (49.4%), with pharmacists and primary care doctors being indicated most often as those providing the programs (61.7% and 35.5% of cases, respectively). Vast majority of programs applied diverse forms of drug reviews. The identified programs were assessed against four predefined dimensions of effectiveness, applicability, scalability and cost-effectiveness. The lowest scores were obtained within the last of these categories, due to unavailability of relevant data. Based on the survey results, a benchmarking application was constructed. It allows for comparing an individual polypharmacy management program targeting the elderly against the other ones, and particularly, against the national and European context. Conclusion: By providing strong evidence, the findings of this study, coupled with the benchmarking application, can prove valuable in aiding clinicians and policymakers in the implementation and expansion of polypharmacy management programs for the elderly.

Comprehensive benchmarking and guidelines of mosaic variant calling strategies

Rapid advances in sequencing and analysis technologies have enabled the accurate detection of diverse forms of genomic variants represented as heterozygous, homozygous and mosaic mutations. However, the best practices for mosaic variant calling remain disorganized owing to the technical and conceptual difficulties faced in evaluation. Here we present our benchmark of 11 feasible mosaic variant detection approaches based on a systematically designed whole-exome-level reference standard that mimics mosaic samples, supported by 354,258 control positive mosaic single-nucleotide variants and insertion-deletion mutations and 33,111,725 control negatives. We identified not only the best practice for mosaic variant detection but also the condition-dependent strengths and weaknesses of the current methods. Furthermore, feature-level evaluation and their combinatorial usage across multiple algorithms direct the way for immediate to prolonged improvements in mosaic variant detection. Our results will guide researchers in selecting suitable calling algorithms and suggest future strategies for developers.

Simulation of undiagnosed patients with novel genetic conditions

Rare Mendelian disorders pose a major diagnostic challenge and collectively affect 300–400 million patients worldwide. Many automated tools aim to uncover causal genes in patients with suspected genetic disorders, but evaluation of these tools is limited due to the lack of comprehensive benchmark datasets that include previously unpublished conditions. Here, we present a computational pipeline that simulates realistic clinical datasets to address this deficit. Our framework jointly simulates complex phenotypes and challenging candidate genes and produces patients with novel genetic conditions. We demonstrate the similarity of our simulated patients to real patients from the Undiagnosed Diseases Network and evaluate common gene prioritization methods on the simulated cohort. These prioritization methods recover known gene-disease associations but perform poorly on diagnosing patients with novel genetic disorders. Our publicly-available dataset and codebase can be utilized by medical genetics researchers to evaluate, compare, and improve tools that aid in the diagnostic process.

ScIBD: a self-supervised iterative-optimizing model for boosting the detection of heterotypic doublets in single-cell chromatin accessibility data

Application of the widely used droplet-based microfluidic technologies in single-cell sequencing often yields doublets, introducing bias to downstream analyses. Especially, doublet-detection methods for single-cell chromatin accessibility sequencing (scCAS) data have multiple assay-specific challenges. Therefore, we propose scIBD, a self-supervised iterative-optimizing model for boosting heterotypic doublet detection in scCAS data. scIBD introduces an adaptive strategy to simulate high-confident heterotypic doublets and self-supervise for doublet-detection in an iteratively optimizing manner. Comprehensive benchmarking on various simulated and real datasets demonstrates the outperformance and robustness of scIBD. Moreover, the downstream biological analyses suggest the efficacy of doublet-removal by scIBD.

Comprehensive benchmarking of laser welding technologies including novel beam shapes and wavelengths for e-drive copper hairpins

Laser welding is the industrially accepted method for the joining of Cu hairpin windings in the production of electric drives. High brilliance laser beams are scanned over the bare ends of the Cu wires producing a rapid connection through deep penetration remote welding. Despite being an accepted manufacturing method, laser welding of Cu hairpins still requires detailed studies concerning manufacturing productivity and quality. As the availability of novel laser sources with higher power levels, new wavelengths, and beam shaping capabilities increase, the need for benchmarking studies emerges. In this work, six different laser welding systems were compared in terms of process productivity and quality during the welding of Cu hairpins used for automotive traction. The different solutions presented power levels from 3 to 6 kW, with wavelengths from near infrared (NIR) to visible, including in source dynamic beam shaping. The weld bead formation was observed through high-speed imaging. The welds were analyzed in terms of their geometry, internal defects, and most relevantly for their mechanical strength. The results showed advantages of each of the employed system while the laser systems providing the highest irradiance profile produced the fastest weld with more elevated mechanical strength independently from the wavelength.

Beyond Human Detection: A Benchmark for Detecting Common Human Posture.

Human detection is the task of locating all instances of human beings present in an image, which has a wide range of applications across various fields, including search and rescue, surveillance, and autonomous driving. The rapid advancement of computer vision and deep learning technologies has brought significant improvements in human detection. However, for more advanced applications like healthcare, human-computer interaction, and scene understanding, it is crucial to obtain information beyond just the localization of humans. These applications require a deeper understanding of human behavior and state to enable effective and safe interactions with humans and the environment. This study presents a comprehensive benchmark, the Common Human Postures (CHP) dataset, aimed at promoting a more informative and more encouraging task beyond mere human detection. The benchmark dataset comprises a diverse collection of images, featuring individuals in different environments, clothing, and occlusions, performing a wide range of postures and activities. The benchmark aims to enhance research in this challenging task by designing novel and precise methods specifically for it. The CHP dataset consists of 5250 human images collected from different scenes, annotated with bounding boxes for seven common human poses. Using this well-annotated dataset, we have developed two baseline detectors, namely CHP-YOLOF and CHP-YOLOX, building upon two identity-preserved human posture detectors: IPH-YOLOF and IPH-YOLOX. We evaluate the performance of these baseline detectors through extensive experiments. The results demonstrate that these baseline detectors effectively detect human postures on the CHP dataset. By releasing the CHP dataset, we aim to facilitate further research on human pose estimation and to attract more researchers to focus on this challenging task.

Efficient prediction of peptide self-assembly through sequential and graphical encoding.

In recent years, there has been an explosion of research on the application of deep learning to the prediction of various peptide properties, due to the significant development and market potential of peptides. Molecular dynamics has enabled the efficient collection of large peptide datasets, providing reliable training data for deep learning. However, the lack of systematic analysis of the peptide encoding, which is essential for artificial intelligence-assisted peptide-related tasks, makes it an urgent problem to be solved for the improvement of prediction accuracy. To address this issue, we first collect a high-quality, colossal simulation dataset of peptide self-assembly containing over 62 000 samples generated by coarse-grained molecular dynamics. Then, we systematically investigate the effect of peptide encoding of amino acids into sequences and molecular graphs using state-of-the-art sequential (i.e. recurrent neural network, long short-term memory and Transformer) and structural deep learning models (i.e. graph convolutional network, graph attention network and GraphSAGE), on the accuracy of peptide self-assembly prediction, an essential physiochemical process prior to any peptide-related applications. Extensive benchmarking studies have proven Transformer to be the most powerful sequence-encoding-based deep learning model, pushing the limit of peptide self-assembly prediction to decapeptides. In summary, this work provides a comprehensive benchmark analysis of peptide encoding with advanced deep learning models, serving as a guide for a wide range of peptide-related predictions such as isoelectric points, hydration free energy, etc.

SpaRx: elucidate single-cell spatial heterogeneity of drug responses for personalized treatment.

Spatial cellular authors heterogeneity contributes to differential drug responses in a tumor lesion and potential therapeutic resistance. Recent emerging spatial technologies such as CosMx, MERSCOPE and Xenium delineate the spatial gene expression patterns at the single cell resolution. This provides unprecedented opportunities to identify spatially localized cellular resistance and to optimize the treatment for individual patients. In this work, we present a graph-based domain adaptation model, SpaRx, to reveal the heterogeneity of spatial cellular response to drugs. SpaRx transfers the knowledge from pharmacogenomics profiles to single-cell spatial transcriptomics data, through hybrid learning with dynamic adversarial adaption. Comprehensive benchmarking demonstrates the superior and robust performance of SpaRx at different dropout rates, noise levels and transcriptomics coverage. Further application of SpaRx to the state-of-the-art single-cell spatial transcriptomics data reveals that tumor cells in different locations of a tumor lesion present heterogenous sensitivity or resistance to drugs. Moreover, resistant tumor cells interact with themselves or the surrounding constituents to form an ecosystem for drug resistance. Collectively, SpaRx characterizes the spatial therapeutic variability, unveils the molecular mechanisms underpinning drug resistance and identifies personalized drug targets and effective drug combinations.

Neutronic and burn up analysis of VVER 1000 benchmark using EXCEL and TRIHEX-FA code system

Benchmarking and validation of the neutronic codes is an important aspect of the reactor design. This paper presents VVER-1000 X2 Benchmark steady state core physics analysis carried out using the lattice level burn-up code, EXCEL and core physics analysis using multi-group diffusion code, TRIHEXFA developed at BARC. These codes have been augmented with new features for modeling of LWR Cores. The VVER-1000 X2 Benchmark with modern fuel assembly (FA) design is a comprehensive benchmark proposed for validating and verifying package of codes and data libraries for reactor physics calculations of LWRs, which includes fuel assembly modeling, generation of homogenized cross section data base for all envisaged core conditions, and reactor core modeling. The trends of k∞ (k-inf) values as a function of various parametric variations and as a function of burn-up for all the fuel assemblies considered in benchmark are analyzed in detail and the deviations have been estimated.The VVER-1000 core follow-up simulations are done by using core diffusion code TRIHEXFA using the few group data generated by EXCEL code. Important core physical characteristics like critical soluble boron, 2-D radial power distribution, axial power distribution, radial peaking factors and volumetric power peaking factors, FA burn-up values obtained from the TRIHEXFA simulations are compared with the experimental core follow up data. All these parameters are found to be reasonably in good agreement with the values provided in benchmark.

Real-Time Dynamics and Detailed Balance in Ring Polymer Surface Hopping: The Impact of Frustrated Hops.

Ring polymer surface hopping (RPSH) has been recently introduced as a well-tailored method for incorporating nuclear quantum effects, such as zero-point energy and tunneling, into nonadiabatic molecular dynamics simulations. The practical widespread usage of RPSH demands a comprehensive benchmarking of different reaction regimes and conditions with equal emphasis on demonstrating both the cons and the pros of the method. Here, we investigate the fundamental questions related to the conservation of energy and detailed balance in the context of RPSH. Using Tully's avoided crossing model as well as a 2-state quantum system coupled to a classical bath undergoing Langevin dynamics, we probe the critical problem of the proper treatment of the classically forbidden transitions stemming from the surface hopping algorithm. We show that proper treatment of these frustrated hops is key to the accurate description of real-time dynamics as well as reproducing the correct quantum Boltzmann populations.

Intersection of policy and Immunization Information Systems (IIS)

BackgroundImmunization information systems (IIS) are confidential, population-based computerized databases that record vaccination doses administered to persons residing within a given geopolitical area. We sought to highlight the evolution of IIS policy over the last two decades, as IIS play a pivotal role in achieving equitable and high vaccine uptake.MethodsLegal epidemiological research methods were used to assess relevant IIS statutes and administrative codes across all 50 states, the District of Columbia, Philadelphia, and New York City. Where relevant, laws were cross-checked or supplemented with state and local health department resources. Comparisons to previous legal studies enabled evaluation of trends in IIS laws over time.ResultsThe compilation of current laws provides an updated overview of the diverse interstate and intrastate policies within the US that govern the capabilities and implementation of IIS. The findings of this study show the progress that has been made in the past decade in improving policies that enable IIS to be utilized across the life-course. Conversely, gaps in IIS data collection, limited interoperability with local and national health information systems, and inconsistent access to view or utilize IIS records due to existing policies, continue to limit the full potential of IIS.ConclusionsIn the United States (US), IIS are implemented and managed at the state and local level, creating variability in IIS policies and implementation. Findings from this study serve as a comprehensive benchmark of current IIS laws that may aid policy stakeholders who are exploring amendments to jurisdictional IIS laws.

Trace encoding in process mining: A survey and benchmarking

Encoding methods are employed across several process mining tasks, including predictive process monitoring, anomalous case detection, trace clustering, etc. These methods are usually performed as preprocessing steps and are responsible for mapping complex event data information into a numerical feature space. Most papers choose existing encoding methods arbitrarily or employ a strategy based on expert domain knowledge. Moreover, existing methods are employed by using their default parameters without evaluating other options. This practice can lead to several drawbacks, such as suboptimal performance and unfair comparisons with the state-of-the-art. Therefore, this work aims at providing a comprehensive survey and benchmark on event log encoding by comparing 27 methods, from different natures, in terms of expressivity, scalability, correlation, and domain agnosticism. To the best of our knowledge, this is the most comprehensive study so far focusing on trace encoding in process mining. It contributes to maturing awareness about the role of trace encoding in process mining pipelines and sheds light on issues, concerns, and future research directions regarding the use of encoding methods to bridge the gap between machine learning models and process mining.

Spectral-infinite-element simulations of seismic wave propagation in self-gravitating, rotating 3-D Earth models

SUMMARY Although observation of gravity perturbations induced by earthquakes is possible, simulation of seismic wave propagation in a self-gravitating, rotating Earth model with 3-D heterogeneity is challenging due to the numerical complexities associated with the unbounded Poisson/Laplace equation that governs gravity perturbations. Therefore, gravity perturbations are generally omitted, and only the background gravity is taken into account using the so-called Cowling approximation. However, gravity perturbations may be significant for large earthquakes (Mw ≥ 6.0) and long-period responses. In this study, we develop a time-domain solver based on the spectral-infinite-element approach, which combines the spectral element method inside the Earth domain with a mapped-infinite-element method in the infinite space outside. This combination allows us to solve the complete, coupled momentum-gravitational equations in a fully discretized domain while accommodating complex 3-D Earth models. We compute displacement and gravity perturbations considering various Earth models, including Preliminary Reference Earth Model and S40RTS and conduct comprehensive benchmarks of our method against the spherical harmonics normal-mode approach and the direct radial integration method. Our 3-D simulations accommodate topography, bathymetry, rotation, ellipticity and oceans. Results show that our technique is accurate and stable for long simulations. Our method provides a new scope for incorporating earthquake-induced gravity perturbations into source and adjoint tomographic inversions.

New Analytical Solutions for Elastoplastic Buckling of Non-Lévy‐Type Rectangular Plates

Analytical solutions for elastoplastic buckling of plates play a crucial role in providing benchmark results and facilitating fast structural analyses for preliminary designs. However, the analytical solutions for elastoplastic buckling of plates remain incomplete due to the inherent mathematical difficulties associated with higher-order partial differential equations and material nonlinearity. Consequently, the existing analytical solutions for rectangular plates are only applicable to those with Lévy‐type boundary conditions. To address the limitation, this study extends a novel symplectic superposition method to obtain new analytical elastoplastic buckling solutions of non-Lévy‐type rectangular plates, where both the incremental theory (IT) and deformation theory (DT) are adopted. Comprehensive benchmark elastoplastic buckling loads are presented and validated by the modified differential quadrature method. The plastic buckling paradox is explicitly observed, which highlights a significant disparity between the IT and DT in predicting buckling loads for relatively thick plates. Furthermore, our analysis reveals that the DT provides an optimal load ratio for buckling resistance, while no such finding is observed with the IT. The stability criterion curves are plotted using the elastic theory and plastic theories (IT and DT) to further reveal the importance of incorporating the effect of plasticity as well as to provide a useful guideline for the relevant analyses and designs.

Straightforward Free Vibration Solutions of Open Cylindrical Shells by the Finite Integral Transform Method

Straightforward analytical free vibration solutions of non-Lévy-type open cylindrical shells are obtained in this study by the finite integral transform method, which was not achieved by conventional semi-inverse methods. Three double finite integral transforms are imposed on the high-order partial differential equations (PDEs) for the problems, with some boundary conditions input, by which the relations between the transformed quantities and specific unknowns are given. Substituting the corresponding inverse transforms into the other boundary conditions provides systems of linear algebraic equations for determining the final analytical solutions. Comprehensive benchmark results for representative open cylindrical shells under different boundary conditions are shown. The results are well verified by other solution methods. With the present solutions, the effects of geometric parameters and boundary conditions on the vibration behaviors are quantitatively revealed by a parametric study. Due to its rigorous and straightforward solution procedure, this study presents a solid easy-to-implement way to explore new analytical solutions of similar problems.

An extensive benchmark study on biomedical text generation and mining with ChatGPT.

In recent years, the development of natural language process (NLP) technologies and deep learning hardware has led to significant improvement in large language models (LLMs). The ChatGPT, the state-of-the-art LLM built on GPT-3.5 and GPT-4, shows excellent capabilities in general language understanding and reasoning. Researchers also tested the GPTs on a variety of NLP-related tasks and benchmarks and got excellent results. With exciting performance on daily chat, researchers began to explore the capacity of ChatGPT on expertise that requires professional education for human and we are interested in the biomedical domain. To evaluate the performance of ChatGPT on biomedical-related tasks, this article presents a comprehensive benchmark study on the use of ChatGPT for biomedical corpus, including article abstracts, clinical trials description, biomedical questions, and so on. Typical NLP tasks like named entity recognization, relation extraction, sentence similarity, question and answering, and document classification are included. Overall, ChatGPT got a BLURB score of 58.50 while the state-of-the-art model had a score of 84.30. Through a series of experiments, we demonstrated the effectiveness and versatility of ChatGPT in biomedical text understanding, reasoning and generation, and the limitation of ChatGPT build on GPT-3.5. All the datasets are available from BLURB benchmark https://microsoft.github.io/BLURB/index.html. The prompts are described in the article.