Systematic Benchmarks Research Articles

Rapid, Accurate, Ranking of Protein-Ligand Binding Affinities with VM2, the Second-Generation Mining Minima Method.

The structure-based technologies most widely used to rank the affinities of candidate small molecule drugs for proteins range from faster but less reliable docking methods to slower but more accurate explicit solvent free energy methods. In recent years, we have advanced another technology, which is called mining minima because it "mines" out the main contributions to the chemical potentials of the free and bound molecular species by identifying and characterizing their main local energy minima. The present study provides systematic benchmarks of the accuracy and computational speed of mining minima, as implemented in the VeraChem Mining Minima Generation 2 (VM2) code, across two well-regarded protein-ligand benchmark data sets, for which there are already benchmark data for docking, free energy, and other computational methods. A core result is that VM2's accuracy approaches that of explicit solvent free energy methods at a far lower computational cost. In finer-grained analyses, we also examine the influence of various run settings, such as the treatment of crystallographic water molecules, on the accuracy, and define the costs in time and dollars of representative runs on Amazon Web Services (AWS) compute instances with various CPU and GPU combinations. We also use the benchmark data to determine the importance of VM2's correction from generalized Born to finite-difference Poisson-Boltzmann results for each energy well and find that this correction affords a remarkably consistent improvement in accuracy at a modest computational cost. The present results establish VM2 as a distinctive technology for early-stage drug discovery, which provides a strong combination of efficiency and predictivity.

Modeling Multi-Step Organic Reactions: Can Density Functional Theory Deliver Misleading Chemistry?

Many organic reactions are characterized by a complex mechanism with a variety of transition states and intermediates of different chemical natures. Their correct and accurate theoretical characterization critically depends on the accuracy of the computational method used. In this work, we study a complex ambimodal cycloaddition with five transition states, two intermediates, and three products, and we ask whether density functional theory (DFT) can provide a correct description of this type of complex and multifaceted reaction. Our work fills a gap in that most systematic benchmarks of DFT for chemical reactions have considered much simpler reactions. Our results show that many density functionals not only lead to seriously large errors but also differ from one another in predicting whether the reaction is ambimodal. Only a few of the available functionals provide a balanced description of the complex and multifaceted reactions. The parameters varied in the tested functionals are the ingredients, the treatment of medium-range and nonlocal correlation energy, and the inclusion of Hartree-Fock exchange. These results show a clear need for more benchmarks on the mechanisms of large molecules in complex reactions.

Spatial-linked alignment tool (SLAT) for aligning heterogenous slices

Spatially resolved omics technologies reveal the spatial organization of cells in various biological systems. Here we propose SLAT (Spatially-Linked Alignment Tool), a graph-based algorithm for efficient and effective alignment of spatial slices. Adopting a graph adversarial matching strategy, SLAT is the first algorithm capable of aligning heterogenous spatial data across distinct technologies and modalities. Systematic benchmarks demonstrate SLAT’s superior precision, robustness, and speed over existing state-of-the-arts. Applications to multiple real-world datasets further show SLAT’s utility in enhancing cell-typing resolution, integrating multiple modalities for regulatory inference, and mapping fine-scale spatial-temporal changes during development. The full SLAT package is available at https://github.com/gao-lab/SLAT.

Evaluation of computational phage detection tools for metagenomic datasets.

As new computational tools for detecting phage in metagenomes are being rapidly developed, a critical need has emerged to develop systematic benchmarks. In this study, we surveyed 19 metagenomic phage detection tools, 9 of which could be installed and run at scale. Those 9 tools were assessed on several benchmark challenges. Fragmented reference genomes are used to assess the effects of fragment length, low viral content, phage taxonomy, robustness to eukaryotic contamination, and computational resource usage. Simulated metagenomes are used to assess the effects of sequencing and assembly quality on the tool performances. Finally, real human gut metagenomes and viromes are used to assess the differences and similarities in the phage communities predicted by the tools. We find that the various tools yield strikingly different results. Generally, tools that use a homology approach (VirSorter, MARVEL, viralVerify, VIBRANT, and VirSorter2) demonstrate low false positive rates and robustness to eukaryotic contamination. Conversely, tools that use a sequence composition approach (VirFinder, DeepVirFinder, Seeker), and MetaPhinder, have higher sensitivity, including to phages with less representation in reference databases. These differences led to widely differing predicted phage communities in human gut metagenomes, with nearly 80% of contigs being marked as phage by at least one tool and a maximum overlap of 38.8% between any two tools. While the results were more consistent among the tools on viromes, the differences in results were still significant, with a maximum overlap of 60.65%. Discussion: Importantly, the benchmark datasets developed in this study are publicly available and reusable to enable the future comparability of new tools developed.

Quantum simulation with just-in-time compilation

Quantum technologies are moving towards the development of novel hardware devices based on quantum bits (qubits). In parallel to the development of quantum devices, efficient simulation tools are needed in order to design and benchmark quantum algorithms and applications before deployment on quantum hardware. In this context, we present a first attempt to perform circuit-based quantum simulation using the just-in-time (JIT) compilation technique on multiple hardware architectures and configurations based on single-node central processing units (CPUs) and graphics processing units (GPUs). One of the major challenges in scientific code development is to balance the level of complexity between algorithms and programming techniques without losing performance or degrading code readability. In this context, we have developed qibojit: a new module for the Qibo quantum computing framework, which uses a just-in-time compilation approach through Python. We perform systematic performance benchmarks between our JIT approach and a subset of relevant publicly available libraries for quantum computing. We show that our novel approach simplifies the complex aspects of the implementation without deteriorating performance.

Neutron-deuteron scattering cross sections with chiral NN interactions using wave-packet continuum discretization

In this work we present a framework that allows to solve the Faddeev equations for three-nucleon scattering using the wave-packet continuum-discretization method. We perform systematic benchmarks using results in the literature and study in detail the convergence of this method with respect to the number of wave packets. We compute several different elastic neutron-deuteron scattering cross-section observables for a variety of energies using chiral nucleon-nucleon interactions. For the interaction $\text{N2LO}_{\text{opt}}$ we find good agreement with data for nucleon scattering-energies $E_\text{Lab} \leq 70$ MeV and a slightly larger maximum of the neutron analyzing power $A_y(n)$ at $E_\text{Lab} = 10$ MeV and 21 MeV compared with other interactions. This work represents a first step towards a systematic inclusion of three-nucleon scattering observables in the construction of next-generation nuclear interactions.

Towards scalable online machine learning collaborations with OpenML

Is massively collaborative machine learning possible? Can we share and organize our collective knowledge of machine learning to solve ever more challenging problems? In a way, yes: as a community, we are already very successful at developing high-quality open-source machine learning libraries, thanks to frictionless collaboration platforms for software development. However, code is only one aspect. The answer is much less clear when we also consider the data that goes into these algorithms and the exact models that are produced. A tremendous amount of work and experience goes into the collection, cleaning, and preprocessing of data and the design, evaluation, and finetuning of models, yet very little of this is shared and organized in a way so that others can easily build on it. Suppose one had a global platform for sharing machine learning datasets, models, and reproducible experiments in a frictionless way so that anybody could chip in at any time to share a good model, add or improve data, or suggest an idea. OpenML is an open-source initiative to create such a platform. It allows anyone to share datasets, machine learning pipelines, and full experiments, organizes all of it online with rich metadata, and enables anyone to reuse and build on them in novel and unexpected ways. All data is open and accessible through APIs, and it is readily integrated into popular machine learning tools to allow easy sharing of models and experiments. This openness also allows a budding ecosystem of automated processes to scale up machine learning further, such as discovering similar datasets, creating systematic benchmarks, or learning from all collected results how to build the best machine learning models and even automatically doing so for any new dataset. We welcome all of you to become a part of it.

Shifting towards Proactive OHS Risk Management in Romanian Organizations: Systematic Benchmarks

As they are anticipative, proactive, and treating assessment as a quasi-continuous phase of risk management, the next step in keeping workers safe seems to be the proper implementation of the guidelines on risk assessments included in ISO 45001. They are indeed inspiring for organizations worldwide to consider risk management as really important, good for the business, and imperative for worker health and safety. What should the Romanian organizations know and do in this spirit? The main objective of this paper is to answer systematically and in a structured way the question of whether there is a set of principles, means to follow, recommendable approaches that, taking into account the general practice established at national level at this time of evolution, allow even to facilitate the transition from reactive to proactive management of occupational hazards, with all the advantages and - at the same time - challenges that this change implies? The main possible answers to this question are investigated, resulting in a set of conclusions with practical applicability, which can guide the process of streamlining the results of accidents at work and occupational diseases prevention work at the national level, within a coherent and unitary vision.

Stochastic model predictive control for central HVAC plants

We present a stochastic model predictive control (MPC) framework for central heating, ventilation, and air conditioning (HVAC) plants. The framework uses real data to forecast and quantify uncertainty of disturbances affecting the system over multiple timescales (electrical loads, heating/cooling loads, and energy prices). We conduct detailed closed-loop simulations and systematic benchmarks for the central HVAC plant of a typical university campus. Results demonstrate that deterministic MPC fails to properly capture disturbances and that this translates into economic penalties associated with peak demand charges and constraint violations in thermal storage capacity (overflow and/or depletion). Our results also demonstrate that stochastic MPC provides a more systematic approach to mitigate uncertainties and that this ultimately leads to cost savings of up to 7.5% and the mitigation of storage constraint violations. Benchmark results also indicate that these savings are close to ideal savings (9.6%) obtained under MPC with perfect information.

A Quadratic Pair Atomic Resolution of the Identity Based SOS-AO-MP2 Algorithm Using Slater Type Orbitals

We report a production level implementation of pair atomic resolution of the identity (PARI) based second-order Møller–Plesset perturbation theory (MP2) in the Slater type orbital (STO) based Amsterdam Density Functional (ADF) code. As demonstrated by systematic benchmarks, dimerization and isomerization energies obtained with our code using STO basis sets of triple-ζ-quality show mean absolute deviations from Gaussian type orbital, canonical, basis set limit extrapolated, global density fitting (DF)-MP2 results of less than 1 kcal/mol. Furthermore, we introduce a quadratic scaling atomic orbital based spin-opposite-scaled (SOS)-MP2 approach with a very small prefactor. Due to a worst-case scaling of , our implementation is very fast already for small systems and shows an exceptionally early crossover to canonical SOS-PARI-MP2. We report computational wall time results for linear as well as for realistic three-dimensional molecules and show that triple-ζ quality calculations on molecules of several hundreds of atoms are only a matter of a few hours on a single compute node, the bottleneck of the computations being the SCF rather than the post-SCF energy correction.

Combining accurate tumor genome simulation with crowdsourcing to benchmark somatic structural variant detection

BackgroundThe phenotypes of cancer cells are driven in part by somatic structural variants. Structural variants can initiate tumors, enhance their aggressiveness, and provide unique therapeutic opportunities. Whole-genome sequencing of tumors can allow exhaustive identification of the specific structural variants present in an individual cancer, facilitating both clinical diagnostics and the discovery of novel mutagenic mechanisms. A plethora of somatic structural variant detection algorithms have been created to enable these discoveries; however, there are no systematic benchmarks of them. Rigorous performance evaluation of somatic structural variant detection methods has been challenged by the lack of gold standards, extensive resource requirements, and difficulties arising from the need to share personal genomic information.ResultsTo facilitate structural variant detection algorithm evaluations, we create a robust simulation framework for somatic structural variants by extending the BAMSurgeon algorithm. We then organize and enable a crowdsourced benchmarking within the ICGC-TCGA DREAM Somatic Mutation Calling Challenge (SMC-DNA). We report here the results of structural variant benchmarking on three different tumors, comprising 204 submissions from 15 teams. In addition to ranking methods, we identify characteristic error profiles of individual algorithms and general trends across them. Surprisingly, we find that ensembles of analysis pipelines do not always outperform the best individual method, indicating a need for new ways to aggregate somatic structural variant detection approaches.ConclusionsThe synthetic tumors and somatic structural variant detection leaderboards remain available as a community benchmarking resource, and BAMSurgeon is available at https://github.com/adamewing/bamsurgeon.

On the performance of DFT/MRCI Hamiltonians for electronic excitations in transition metal complexes: The role of the damping function.

The combination of density functional theory and multireference configuration interaction (DFT/MRCI) is a well-established semi-empirical method suitable for computing spectral properties of large molecular systems. To this day, three different Hamiltonians and various parameter set combinations exist. These DFT/MRCI variants are well tried and tested when it comes to electronic excitations of organic molecules. For transition metal complexes, systematic benchmarks against experimental data are missing, however. Here we present an assessment of the DFT/MRCI variants and of time-dependent, linear-response density functional theory (TDDFT) for a diverse set of ligand-centered, metal-to-ligand charge transfer, metal-centered, and ligand-to-metal charge transfer (LMCT) excitations on 21 3d and 4d complexes comprising 10 small inorganic and 11 larger metalorganic compounds with closed-shell ground states. In the course of this assessment, we realized that the excitation energies of transition metal complexes can be very sensitive with respect to the details of the damping function that scales off-diagonal matrix elements. This scaling is required in DFT/MRCI to avoid double counting of dynamic electron correlation. These insights lead to a new Hamiltonian, denoted R2018, with improved performance on transition metal compounds, while the results for organic molecules are nearly unaffected by the modified damping function. Two parameter sets were optimized for this Hamiltonian: One set is to be used in conjunction with the standard configuration selection threshold of 1.0 E h and a second set is for use with a selection threshold of 0.8 E h which leads to shorter wave function expansions. The R2018 Hamiltonian in standard parameterization achieves root-mean-square errors (RMSEs) of merely 0.15 eV for the metalorganic complexes, followed by 0.20 eV for the original DFT/MRCI ansatz, and 0.25 eV for the redesigned DFT/MRCI approach. In comparison, TDDFT gives a much larger RMSE of 0.46 eV for metalorganic complexes. None of the DFT/MRCI variants yields convincing results for small oxides and fluorides which exhibit LMCT transitions. Here, TDDFT performs better. If the oxides and fluorides are excluded from the inorganic test set, satisfactory agreement can be achieved, with RMSE values between 0.26 eV and 0.30 eV for DFT/MRCI and 0.34 eV for TDDFT. The performance of the original and the new DFT/MRCI Hamiltonians deteriorates only slightly, when a tighter selection threshold is chosen, thus enabling the computation of reliable spectral properties even for large metalorganic complexes.

Personal Cloud Storage Benchmarks and Comparison

The large amount of space offered by personal cloud storage services (e.g., Dropbox and OneDrive), together with the possibility of synchronizing devices seamlessly, keep attracting customers to the cloud. Despite the high public interest, little information about system design and actual implications on performance is available when selecting a cloud storage service. Systematic benchmarks to assist in comparing services and understanding the effects of design choices are still lacking. This paper proposes a methodology to understand and benchmark personal cloud storage services. Our methodology unveils their architecture and capabilities. Moreover, by means of repeatable and customizable tests, it allows the measurement of performance metrics under different workloads. The effectiveness of the methodology is shown in a case study in which 11 services are compared under the same conditions. Our case study reveals interesting differences in design choices. Their implications are assessed in a series of benchmarks. Results show no clear winner, with all services having potential for improving performance. In some scenarios, the synchronization of the same files can take 20 times longer. In other cases, we observe a wastage of twice as much network capacity, questioning the design of some services. Our methodology and results are thus useful both as benchmarks and as guidelines for system design.

Complexity analysis of simulations with analytic bond-order potentials

The modeling of materials at the atomistic level with interatomic potentials requires a reliable description of different bonding situations and relevant system properties. For this purpose, analytic bond-order potentials (BOPs) provide a systematic and robust approximation to density functional theory (DFT) and tight binding (TB) calculations at reasonable computational cost. This paper presents a formal analysis of the computational complexity of analytic BOP simulations, based on a detailed assessment of the most computationally intensive parts. Different implementation algorithms are presented alongside with optimizations for efficient numerical processing. The theoretical complexity study is complemented by systematic benchmarks of the scalability of the algorithms with increasing system size and accuracy level of the BOP approximation. Both approaches demonstrate that the computation of atomic forces in analytic BOPs can be performed with a similar scaling as the computation of atomic energies.

A systematic benchmark of the ab initio Bethe-Salpeter equation approach for low-lying optical excitations of small organic molecules.

The predictive power of the ab initio Bethe-Salpeter equation (BSE) approach, rigorously based on many-body Green's function theory but incorporating information from density functional theory, has already been demonstrated for the optical gaps and spectra of solid-state systems. Interest in photoactive hybrid organic/inorganic systems has recently increased and so has the use of the BSE for computing neutral excitations of organic molecules. However, no systematic benchmarks of the BSE for neutral electronic excitations of organic molecules exist. Here, we study the performance of the BSE for the 28 small molecules in Thiel's widely used time-dependent density functional theory benchmark set [Schreiber et al., J. Chem. Phys. 128, 134110 (2008)]. We observe that the BSE produces results that depend critically on the mean-field starting point employed in the perturbative approach. We find that this starting point dependence is mainly introduced through the quasiparticle energies obtained at the intermediate GW step and that with a judicious choice of starting mean-field, singlet excitation energies obtained from BSE are in excellent quantitative agreement with higher-level wavefunction methods. The quality of the triplet excitations is slightly less satisfactory.

Effects of vibrational averaging on coupled cluster calculations of spin–spin coupling constants for hydrocarbons

We present vibrationally corrected nuclear spin–spin coupling constants for four hydrocarbons with different types of carbon–carbon bonds calculated with coupled cluster (CC) theory. First, we perform a systematic basis set investigation on acetylene for all of the four contributions (Fermi-contact, spin-dipole, para- and diamagnetic spin–orbit) to the spin–spin coupling constants and subsequently choose basis sets of sufficient flexibility to describe converged electronic properties. Then, in order to describe the effects of vibrational motion for the studied molecules we perform a Taylor expansion in the normal coordinates up to second order – a method that is well known for both its quality and efficiency – and rigorously estimate the resulting contribution for all types of spin–spin coupling constants. Combined, this allows us to obtain highly accurate benchmark estimates of the spin–spin coupling constants for acetylene, ethylene, ethane, and cyclopropane. This work provides one of the first systematic benchmarks of zero-point vibrational contributions to spin–spin coupling constants in poly-atomic molecules using the reliable CC theory and it is thus an important reference for further research within in-silico spin–spin coupling constant determination. We note that earlier computational estimates of zero-point vibrational effects agree well with those presented here (for acetylene, ethylene, and cyclopropane) while vibrational corrections for ethane are reported for the first time.