Benchmark Study Research Articles

Proteome-Scale Tissue Mapping Using Mass Spectrometry Based on Label-Free and Multiplexed Workflows

Multiplexed bimolecular profiling of tissue microenvironment, or spatial omics, can provide deep insight into cellular compositions and interactions in healthy and diseased tissues. Proteome-scale tissue mapping, which aims to unbiasedly visualize all the proteins in a whole tissue section or region of interest, has attracted significant interest because it holds great potential to directly reveal diagnostic biomarkers and therapeutic targets. While many approaches are available, however, proteome mapping still exhibits significant technical challenges in both protein coverage and analytical throughput. Since many of these existing challenges are associated with mass spectrometry-based protein identification and quantification, we performed a detailed benchmarking study of three protein quantification methods for spatial proteome mapping, including label-free, TMT-MS2, and TMT-MS3. Our study indicates label-free method provided the deepest coverages of ∼3500 proteins at a spatial resolution of 50μm and the highest quantification dynamic range, while TMT-MS2 method holds great benefit in mapping throughput at >125 pixels per day. The evaluation also indicates both label-free and TMT-MS2 provides robust protein quantifications in identifying differentially abundant proteins and spatially covariable clusters. In the study of pancreatic islet microenvironment, we demonstrated deep proteome mapping not only enables the identification of protein markers specific to different cell types, but more importantly, it also reveals unknown or hidden protein patterns by spatial coexpression analysis.

Calculated hydration free energies become less accurate with increases in molecular weight

In order for computer-aided drug design to fulfil its long held promise of delivering new medicines faster and cheaper, extensive development and validation work must be done first. This pertains particularly to molecular dynamics force fields where one important aspect–the hydration free energy (HFE) of small molecules–is often insufficiently analyzed. While most benchmarking studies report excellent accuracies of calculated hydration free energies–usually within 2 kcal/mol of experimental values–we find that deeper analysis reveals significant shortcomings. Herein, we report a dependence of HFE prediction errors on ligand molecular weight–the higher the weight, the bigger the prediction error and the higher the probability the calculated result is erroneous by a large amount. We show that in the drug-like molecular weight region, HFE predictions can easily be off by 5 kcal/mol or more. This is likely to be highly problematic in a drug discovery and development setting. We make our HFE results and molecular descriptors freely and fully available in order to encourage deeper analysis of future molecular dynamics results and facilitate development of the next generation of force fields.

Transfer Learning Approach to Multitarget QSRR Modeling in RPLC.

QSRR is a valuable technique for the retention time predictions of small molecules. This aims to bridge the gap between molecular structure and chromatographic behavior, offering invaluable insights for analytical chemistry. Given the challenge of simultaneous target prediction with variable experimental conditions and the scarcity of comprehensive data sets for such predictive modelings in chromatography, this study introduces a transfer learning-based multitarget QSRR approach to enhance retention time prediction. Through a comparative study of four models, both with and without the transfer learning approach, the performance of both single and multitarget QSRR was evaluated based on Mean Squared Error (MSE) and R2 metrics. Individual models were also tested for their performance against benchmark studies in this field. The findings suggest that transfer learning based multitarget models exhibit potential for enhanced accuracy in predicting retention times of small molecules, presenting a promising avenue for QSRR modeling. These models will be highly beneficial for optimizing experimental conditions in method development by better retention time predictions in Reversed-Phase Liquid Chromatography (RPLC). The reliable and effective predictive capabilities of these models make them valuable tools for pharmaceutical research and development endeavors.

A benchmark study on the energy efficiency and environmental impacts of alternative fuels in gulet-type sailing yachts

Gulet-type yachts are one of the symbols of maritime culture with their unique hull forms and schooner or ketch-type riggings. In parallel with the targets aiming for decarbonization in the maritime industry, energy efficiency and emission control are on the agenda for these types of yachts. Driven by this motivation, a novel benchmark study for the gulet-type yachts is conducted to evaluate the energy efficiency and environmental impacts associated with the adoption of LNG or methanol as primary fuel alternatives to MDO. The benchmark study is constructed in two steps: Presenting the design and propulsion characteristics of existing gulets, followed by a detailed analysis of the energy efficiency and environmental impacts of using alternative fuels. Therefore, 57 gulets with round and transom sterns whose hull forms are analyzed, modeled and general characteristics are presented. After modeling, the resistance and effective power of hull forms are predicted using the Holtrop-Mennen method and CFD. The flow characteristics around the hulls are obtained and the results are validated using the previous experimental studies in the first step. Then, installed and computed engine capacities are compared, and the Annual Emission Ratio and Energy Efficiency Index (EEI) are calculated for environmental impact assessments. Finally, the applicability of using LNG or MeOH-fueled propulsion systems for gulet-type yachts is discussed considering energy efficiency and design. According to the results, transitioning to alternative fuels can increase the Energy Efficiency Index of gulet-type yachts by over 20%.

Enhancing the photovoltaic properties of phenothiazine-based chromophores via structural integration of selenophene analogues

A series of oligomer-type donor photovoltaic materials (MP2–MP9) were designed via the incorporation of selenophene unit (n = 1–8) in MP1 as π-spacer. In order to accomplish the electronic, photophysical, and photovoltaic (PV) behavior of MPR and MP1–MP9, quantum chemical calculations were performed through the density functional theory (DFT) approach. The M06/6-311G (d,p) level was selected for current investigation through a benchmark study between experimental and DFT values of λmax at various functionals of MPR compound. Descending in energy gap (3.058–2.476 eV) with wider absorption wavelength (652.998–513.392 nm) in dichloromethane along with greater charge transmission were probed with the addition of selenophene unit as compared to MPR. The studied chromophores were perceived to have Voc and FF ranging from 1.320–1.728 V and 0.903–0.923 %, respectively. These findings provide valuable insights into the design of oligomer-type donor materials with optimized electronic and photophysical properties, which could enhance the performance and efficiency of organic photovoltaic devices.

Benchmark Study of Core-Ionization Energies with the Generalized Active Space-Driven Similarity Renormalization Group.

X-ray photoelectron spectroscopy (XPS) is a powerful experimental technique for probing the electronic structure of molecules and materials; however, interpreting XPS data requires accurate computational methods to model core-ionized states. This work proposes and benchmarks a new approach based on the generalized active space-driven similarity renormalization group (GAS-DSRG) for calculating core-ionization energies and treating correlation effects at the perturbative and nonperturbative levels. We tested the GAS-DSRG across three data sets. First, the vertical core-ionization energies of small molecules containing first-row elements are evaluated. GAS-DSRG achieves mean absolute errors below 0.3 eV, which is comparable to high-level coupled cluster methods. Next, the accuracy of GAS-DSRG is evaluated for larger organic molecules using the CORE65 data set, with the DSRG-MRPT3 level yielding a mean absolute error of only 0.34 eV for 65 core-ionization transitions. Insights are provided into the treatment of static and dynamic correlation, the importance of high-order perturbation theory, and notable differences from density functional theory in the predicted energy ordering of core-ionized states for specific molecules. Finally, vibrationally resolved XPS spectra of diatomic molecules (CO, N2, and O2) are simulated, showing excellent agreement with experimental data.

Temporal graphs anomaly emergence detection: benchmarking for social media interactions

Temporal graphs have become an essential tool for analyzing complex dynamic systems with multiple agents. Detecting anomalies in temporal graphs is crucial for various applications, including identifying emerging trends, monitoring network security, understanding social dynamics, tracking disease outbreaks, and understanding financial dynamics. In this paper, we present a comprehensive benchmarking study that compares 12 data-driven methods for anomaly detection in temporal graphs. We conduct experiments on two temporal graphs extracted from Twitter and Facebook, aiming to identify anomalies in group interactions. Surprisingly, our study reveals an unclear pattern regarding the best method for such tasks, highlighting the complexity and challenges involved in anomaly emergence detection in large and dynamic systems. The results underscore the need for further research and innovative approaches to effectively detect emerging anomalies in dynamic systems represented as temporal graphs.

A comprehensive benchmarking study of adaptive immune receptor repertoire sequencing data aligners

A comprehensive benchmarking study of adaptive immune receptor repertoire sequencing data aligners

Characterization of inter-panel connections in CLT floors using finite element model updating

Large cross-laminated timber (CLT) floors are inevitably composed of multiple CLT panels assembled on the construction site. Due to the sheer lack of adequate numerical characterization of the inter-panel connections, CLT floors are normally modelled as monolithic structures or independent “piano keyboards”, yielding errors between simulated and the actual vibration response. This paper presents a benchmark study for a deterministic finite element (FE) model updating of so called “an equivalent elastic strip” representation of the inter-panel connections. Fitting modelling parameters of the strip, i.e. its width and material properties, featured FE modelling and modal testing of two full-scale CLT floors composed of one and two panels connected with a half-lapped joint. The floors were made of (almost) identical panels and placed on elastic mats along their shorter edges. First, the single-panel floor was used to fit material properties of the timber and the boundary conditions. Then, the FE model updating (FEMU) of two-panel floor yielded the actual values of the strip modelling parameters. Finally, these were successfully verified by direct comparison between numerically calculated and experimentally measured modal properties of a full-scale model of a simply supported three-panel floor. The results implied that the rotational stiffness of the inter-panel connection depends strongly on its vertical vibration displacement. Consequently, values of the rotational stiffness derived from rare static tests published so far cannot be used reliably in a much subtler vibration response analysis. Future studies should feature other types of the inter-panel connections in various floor layouts and a probabilistic FEMU.

Quality and safety in prehospital airway management – retrospective analysis of 18,000 cases from an air rescue database in Germany

BackgroundPrehospital airway management remains crucial with regard to the quality and safety of emergency medical service (EMS) systems worldwide. In 2007, the benchmark study by Timmermann et al. hit the German EMS community hard by revealing a significant rate of undetected oesophageal intubations leading to an often-fatal outcome. Since then, much attention has been given to guideline development and training. This study evaluated the incidence and special circumstances of tube misplacement as an adverse peri-intubation event from a Helicopter Emergency Medical Services perspective.MethodsThis was a retrospective analysis of a German helicopter-based EMS database from January 1, 2012, to December 31, 2020. All registered patients were included in the primary analysis. The results were analysed using SPSS 27.0.1.0.ResultsOut of 227,459 emergency medical responses overall, a total of 18,087 (8.0%) involved invasive airway management. In 8141 (45.0%) of these patients, airway management devices were used by ground-based EMS staff, with an intubation rate of 96.6% (n = 7861), and alternative airways were used in 3.2% (n = 285). Overall, the rate of endotracheal intubation success was 94.7%, while adverse events in the form of tube misplacement were present in 5.3%, with a 1.2% rate of undetected oesophageal intubation. Overall tube misplacement and undetected oesophageal intubation occurred more often after intubation was carried out by paramedics (10.4% and 3.6%, respectively). In view of special circumstances, those errors occurred more often in the presence of trauma or cardiopulmonary resuscitation, with rates of 5.6% and 6.4%, respectively. Difficult airways with a Cormack 4 status were present in 2.1% (n = 213) of HEMS patients, accompanied by three or more intubation attempts in 5.2% (n = 11).ConclusionsPrehospital airway management success has improved significantly in recent years. However, adverse peri-intubation events such as undetected oesophageal intubation remain a persistent threat to patient safety.Trial registrationThe study was registered in the German Register for Clinical Studies (number DRKS00028068).

Does combining numerous data types in multi-omics data improve or hinder performance in survival prediction? Insights from a large-scale benchmark study

BackgroundPredictive modeling based on multi-omics data, which incorporates several types of omics data for the same patients, has shown potential to outperform single-omics predictive modeling. Most research in this domain focuses on incorporating numerous data types, despite the complexity and cost of acquiring them. The prevailing assumption is that increasing the number of data types necessarily improves predictive performance. However, the integration of less informative or redundant data types could potentially hinder this performance. Therefore, identifying the most effective combinations of omics data types that enhance predictive performance is critical for cost-effective and accurate predictions.MethodsIn this study, we systematically evaluated the predictive performance of all 31 possible combinations including at least one of five genomic data types (mRNA, miRNA, methylation, DNAseq, and copy number variation) using 14 cancer datasets with right-censored survival outcomes, publicly available from the TCGA database. We employed various prediction methods and up-weighted clinical data in every model to leverage their predictive importance. Harrell’s C-index and the integrated Brier Score were used as performance measures. To assess the robustness of our findings, we performed a bootstrap analysis at the level of the included datasets. Statistical testing was conducted for key results, limiting the number of tests to ensure a low risk of false positives.ResultsContrary to expectations, we found that using only mRNA data or a combination of mRNA and miRNA data was sufficient for most cancer types. For some cancer types, the additional inclusion of methylation data led to improved prediction results. Far from enhancing performance, the introduction of more data types most often resulted in a decline in performance, which varied between the two performance measures.ConclusionsOur findings challenge the prevailing notion that combining multiple omics data types in multi-omics survival prediction improves predictive performance. Thus, the widespread approach in multi-omics prediction of incorporating as many data types as possible should be reconsidered to avoid suboptimal prediction results and unnecessary expenditure.

A Benchmark Study for Real-Time Optimal Control of Connected HEVs: Design, Integration, and Evaluation

A Benchmark Study for Real-Time Optimal Control of Connected HEVs: Design, Integration, and Evaluation

Short-term vital parameter forecasting in the intensive care unit: A benchmark study leveraging data from patients after cardiothoracic surgery.

Patients in an Intensive Care Unit (ICU) are closely and continuously monitored, and many machine learning (ML) solutions have been proposed to predict specific outcomes like death, bleeding, or organ failure. Forecasting of vital parameters is a more general approach to ML-based patient monitoring, but the literature on its feasibility and robust benchmarks of achievable accuracy are scarce. We implemented five univariate statistical models (the naïve model, the Theta method, exponential smoothing, the autoregressive integrated moving average model, and an autoregressive single-layer neural network), two univariate neural networks (N-BEATS and N-HiTS), and two multivariate neural networks designed for sequential data (a recurrent neural network with gated recurrent unit, GRU, and a Transformer network) to produce forecasts for six vital parameters recorded at five-minute intervals during intensive care monitoring. Vital parameters were the diastolic, systolic, and mean arterial blood pressure, central venous pressure, peripheral oxygen saturation (measured by non-invasive pulse oximetry) and heart rate, and forecasts were made for 5 through 120 minutes into the future. Patients used in this study recovered from cardiothoracic surgery in an ICU. The patient cohort used for model development (n = 22,348) and internal testing (n = 2,483) originated from a heart center in Germany, while a patient sub-set from the eICU collaborative research database, an American multicenter ICU cohort, was used for external testing (n = 7,477). The GRU was the predominant method in this study. Uni- and multivariate neural network models proved to be superior to univariate statistical models across vital parameters and forecast horizons, and their advantage steadily became more pronounced for increasing forecast horizons. With this study, we established an extensive set of benchmarks for forecast performance in the ICU. Our findings suggest that supplying physicians with short-term forecasts of vital parameters in the ICU is feasible, and that multivariate neural networks are most suited for the task due to their ability to learn patterns across thousands of patients.

Planning Benchmark Study for Stereotactic Body Radiation Therapy of Pancreas Carcinomas With Simultaneously Integrated Boost and Protection: Results of the DEGRO/DGMP Working Group on Stereotactic Radiation Therapy and Radiosurgery

The proximity or overlap of planning target volume (PTV) and organs-at-risk (OARs) poses a major challenge in stereotactic body radiation therapy (SBRT) of pancreatic cancer (PACA). This international treatment planning benchmark study investigates whether simultaneous integrated boost (SIB) and simultaneous integrated protection (SIP) concepts in PACA SBRT can lead to improved and harmonized plan quality. A multiparametric specification of desired target doses (gross target volume [GTV]D50%, GTVD99%, PTVD95%, and PTV0.5cc) with 2 prescription doses of GTVD50% = 5 × 9.2Gy (46 Gy) and GTVD50% = 8 × 8.25 Gy (66 Gy) and OAR limits were distributed with planning computed tomography and contours from 3 PACA patients. In phase 1, plans were ranked using a scoring system for comparison of trade-offs between GTV/PTV and OAR. In phase 2, replanning was performed for the most challenging case and prescription with dedicated SIB and SIP contours provided for optimization after group discussion. For all 3 cases and both phases combined, 292 plans were generated from 42 institutions in 5 countries using commonly available treatment planning systems. The GTVD50% prescription was performed by only 76% and 74% of planners within 2% for 5 and 8 fractions, respectively. The GTVD99% goal was mostly reached, while the balance between OAR and target dose showed initial SIB/SIP-like optimization strategies in about 50% of plans. For plan ranking, 149 and 217 score penalties were given for 5 and 8 fractions, pointing to improvement possibilities. For phase 2, the GTVD50% prescription was performed by 95% of planners within 2%, and GTVD99% as well as OAR doses were better harmonized with notable less score penalties. Fourteen of 19 planners improved their plan rank, 9 of them by at least 2 ranks. Dedicated SIB/SIP concepts in combination with multiparametric prescriptions and constraints can lead to overall harmonized and high treatment plan quality for PACA SBRT. Standardized SIB/SIP treatment planning in multicenter clinical trials appears feasible after group consensus and training.

KF-PEV: a causal Kalman filter-based particle event velocimetry

Event-based pixel sensors asynchronously report changes in log-intensity in microsecond-order resolution. Its exceptional speed, cost effectiveness, and sparse event stream make it an attractive imaging modality for particle tracking velocimetry. In this work, we propose a causal Kalman filter-based particle event velocimetry (KF-PEV). Using the Kalman filter model to track the events generated by the particles seeded in the flow medium, KF-PEV yields the linear least squares estimate of the particle track velocities corresponding to the flow vector field. KF-PEV processes events in a computationally efficient and streaming manner (i.e., causal and iteratively updating). Our simulation-based benchmarking study with synthetic particle event data confirms that the proposed KF-PEV outperforms the conventional frame-based particle image/tracking velocimetry as well as the state-of-the-art event-based particle velocimetry methods. In a real-world water tunnel event-based sensor data experiment performed on what we believe to be the widest field view ever reported, KF-PEV accurately predicted the expected flow field of the SD7003 wing, including details such as the lower velocity in the wake and the flow separation around the underside of an angled wing.

Can a Transparent Machine Learning Algorithm Predict Better than Its Black Box Counterparts? A Benchmarking Study Using 110 Data Sets.

We developed a novel machine learning (ML) algorithm with the goal of producing transparent models (i.e., understandable by humans) while also flexibly accounting for nonlinearity and interactions. Our method is based on ranked sparsity, and it allows for flexibility and user control in varying the shade of the opacity of black box machine learning methods. The main tenet of ranked sparsity is that an algorithm should be more skeptical of higher-order polynomials and interactions a priori compared to main effects, and hence, the inclusion of these more complex terms should require a higher level of evidence. In this work, we put our new ranked sparsity algorithm (as implemented in the open source R package, sparseR) to the test in a predictive model "bakeoff" (i.e., a benchmarking study of ML algorithms applied "out of the box", that is, with no special tuning). Algorithms were trained on a large set of simulated and real-world data sets from the Penn Machine Learning Benchmarks database, addressing both regression and binary classification problems. We evaluated the extent to which our human-centered algorithm can attain predictive accuracy that rivals popular black box approaches such as neural networks, random forests, and support vector machines, while also producing more interpretable models. Using out-of-bag error as a meta-outcome, we describe the properties of data sets in which human-centered approaches can perform as well as or better than black box approaches. We found that interpretable approaches predicted optimally or within 5% of the optimal method in most real-world data sets. We provide a more in-depth comparison of the performances of random forests to interpretable methods for several case studies, including exemplars in which algorithms performed similarly, and several cases when interpretable methods underperformed. This work provides a strong rationale for including human-centered transparent algorithms such as ours in predictive modeling applications.

Struggling with fish age, a comparison of otolith preparation techniques to unravel age and growth of boarfish, Capros aper (Linnaeus, 1758)

Fish age and growth data are crucial for understanding vital species’ traits and their populations. Boarfish is a pelagic species widely distributed in the Northeast Atlantic that is one of the most discarded non-commercial species on the Portuguese coast. Due to its potential valorisation as a new fishery in the Northeast Atlantic, boarfish has become object of interest in several published studies. However, since no age standard protocol has been implemented for boarfish, studies have used a variety of different otolith preparation techniques to estimate age, leading to contradicting and discrepant results. This research aims to consolidate biological insights into boarfish age and growth, lay the foundation for a standardized ageing protocol and serve as a benchmark study for the Portuguese west coast before commercial exploitation. Through the comparison of two otolith ageing methodologies, this study recommends using whole otoliths as the best method, revealing a maximum age of 15 years. The growth pattern estimated, indicated a biphasic growth pattern, with a faster initial growth rate that slows down at 2.4 years, the moment of change, when energy is allocated for sexual maturation. This finding was corroborated by a maturity ogive and analysis of fish size and otolith length relationship.

Fast empirical scenarios

We seek to extract a small number of representative scenarios from large panel data that are consistent with sample moments. Among two novel algorithms, the first identifies scenarios that have not been observed before, and comes with a scenario-based representation of covariance matrices. The second proposal selects important data points from states of the world that have already realized, and are consistent with higher-order sample moment information. Both algorithms are efficient to compute and lend themselves to consistent scenario-based modeling and multi-dimensional numerical integration that can be used for interpretable decision-making under uncertainty. Extensive numerical benchmarking studies and an application in portfolio optimization favor the proposed algorithms.

A New Evaluation Metric for Quantitative Accuracy of LC-MS/MS-Based Proteomics with Data-Independent Acquisition.

Data-independent acquisition (DIA) has improved the identification and quantitation coverage of peptides and proteins in liquid chromatography-tandem mass spectrometry-based proteomics. However, different DIA data-processing tools can produce very different identification and quantitation results for the same data set. Currently, benchmarking studies of DIA tools are predominantly focused on comparing the identification results, while the quantitative accuracy of DIA measurements is acknowledged to be important but insufficiently investigated, and the absence of suitable metrics for comparing quantitative accuracy is one of the reasons. A new metric is proposed for the evaluation of quantitative accuracy to avoid the influence of differences in false discovery rate control stringency. The part of the quantitation results with high reliability was acquired from each DIA tool first, and the quantitative accuracy was evaluated by comparing quantification error rates at the same number of accurate ratios. From the results of four benchmark data sets, the proposed metric was shown to be more sensitive to discriminating the quantitative performance of DIA tools. Moreover, the DIA tools with advantages in quantitative accuracy were consistently revealed by this metric. The proposed metric can also help researchers in optimizing algorithms of the same DIA tool and sample preprocessing methods to enhance quantitative accuracy.

Enhancing patient safety using failure mode and effect criticality analysis: a benchmarking study on selected hospitals

PurposePatient safety is a top priority globally. A robust healthcare system requires strategic collaboration between research and development. The author analysed over 300 cases from seven hospitals using the failure modes, effects, and criticality analysis (FMECA) tool to understand the underlying causes of medical errors.Design/methodology/approachThe author studied seven hospitals and 300 cases using FMECA to prioritise activities. The findings showed that high-priority events occurred less frequently but had the potential to cause the most harm. Team members evaluated independently to ensure unbiased evaluations. This approach is useful for setting priorities or assessing difficulties.FindingsPoor communication and lack of coordination among staff in a healthcare organisation caused misunderstandings, ineffective decision-making, delays in patient care, and medical errors. Implementation of effective communication and coordination protocols can help avoid these problems.Practical implicationsThe study recommends using FMECA to identify and prioritise failures and conducting in-depth analyses to understand their root causes. It also highlights the importance of interdisciplinary knowledge and soft skills for healthcare staff.Originality/valueThis study reveals the significance of FMECA in healthcare risk management and benchmarking. FMECA helps identify system failures, develop prevention strategies, and evaluate effectiveness against industry benchmarks. It offers healthcare professionals a valuable tool to enhance patient safety and improve healthcare quality.