Empirical Benchmarks Research Articles

Proactive–reactive microservice architecture global scaling

We develop a novel approach for run-time global adaptation of microservice applications, based on synthesis of architecture-level reconfigurations. More precisely, we devise an algorithm for proactive–reactive automatic scaling that reaches a target system’s Maximum Computational Load by performing optimal deployment orchestrations. We evaluate our approach by developing a platform for the modeling and simulation of microservice architectures, and we use such a platform to compare local/global and reactive/proactive scaling. Empirical benchmarks, obtained through our platform, show that proactive global scaling consistently outperforms the reactive approach, but the best performances can be obtained by our original approach for mixing proactivity and reactivity. In particular, our approach surpasses the state-of-the-art when both performance and resource consumption are considered.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Observations of Extremely Metal-poor O Stars: Weak Winds and Constraints for Evolution Models

Metal-poor massive stars drive the evolution of low-mass galaxies, both locally and at high redshift. However, quantifying the feedback they impart to their local surroundings remains uncertain because models of stellar evolution, mass loss, and ionizing spectra are unconstrained by observations below 20% solar metallicity (Z ⊙). We present new Keck Cosmic Web Imager optical spectroscopy of three O-type stars in the nearby dwarf galaxies Leo P, Sextans A, and WLM, which have gas-phase oxygen abundances of 3%–14% Z ⊙. To characterize their fundamental stellar properties and radiation-driven winds, we fit PoWR atmosphere models to the optical spectra simultaneously with Hubble Space Telescope far-ultraviolet (FUV) spectra and multiwavelength photometry. We find that all three stars have effective temperatures consistent with their spectral types and surface gravities typical of main-sequence dwarf stars. Yet, the combination of those inferred parameters and luminosity for the two lower-Z stars is not reproduced by stellar evolution models, even those that include rotation or binary interactions. The scenario of multiple-star systems is difficult to reconcile with all available data, suggesting that these observations pose a challenge to current evolution models. We highlight the importance of validating the relationship between stellar mass, temperature, and luminosity at very low Z for accurate estimates of ionizing photon production and spectral hardness. Finally, all three stars’ FUV wind profiles reveal low mass-loss rates and terminal wind velocities in tension with expectations from widely adopted radiation-driven wind models. These results provide empirical benchmarks for future development of mass-loss and evolution models for metal-poor stellar populations.

Assessing tourism sustainability in European Union countries: A multi-directional benefit of the doubt composite indicator

We propose a composite indicator for the assessment of tourism sustainability in European Union (EU) countries relatively to years 2019 and 2021. Our proposal is a revisitation of the Travel & Tourism Development Index (TTDI), which is based on a uniform weighting scheme, by means of the Multi-Directional Benefit of the Doubt (MD-BoD) model, an innovative non-compensatory method able to derive a country-specific preference structure from the data. Thanks to the MD-BoD model, each pillar and elementary indicator receives a score that reflects the potential improvement of the country relatively to an empirical benchmark, therefore it is possible to objectively identify weak and strong dimensions of tourism sustainability before and after the Covid-19 pandemic. As such, our composite indicator represents a valuable support for the design of policies aimed at increasing the resilience of the European tourism ecosystem.

Treasury Richness

ABSTRACTWe provide estimates of Treasury convenience premia across the entire term structure of Treasury bills, notes, and bonds over more than a quarter of a century and document a variety of key stylized facts about their time‐series and cross‐sectional patterns. These results raise concerns about the evolving nature of Treasury markets and suggest that investors may now place less weight on the traditional role of Treasury securities as liquid trading vehicles. These stylized facts provide empirical benchmarks that could help guide future theoretical and empirical work about the economics of safe assets in financial markets.

Growth on 2019 State Achievement Tests: Empirical Benchmarks and the Role of Scale Choice

Benchmarks for “years of learning”–expectations for annual academic growth in standard deviation units–are popular to contextualize the practical significance of a causal effect. Though they are applied in studies using a variety of outcome measures, these benchmarks must be built using vertically scaled test scores. Vertical scales must be established to measure growth across grades, and different scales can lead to different conclusions about how much students grow. This article introduces new benchmarks for annual growth based on 2019 grade 3–8 state achievement tests in math and ELA, which are more appropriate for use in contemporary studies using test scores as an outcome measure than existing benchmarks based on older tests aligned to prior generation standards. While high-level trends in growth are generally consistent between prior studies and the present study, differences do arise that would lead to substantively different conclusions about the practical significance of a causal effect. Based on exploratory analyses of growth differences across 2019 achievement scales, further guidance is provided on when and how to use this type of aggregated benchmark for annual growth.

Multiply by 37 (or Divide by 0.027): A Surprisingly Accurate Rule of Thumb for Converting Effect Sizes From Standard Deviations to Percentile Points

Educational researchers often report effect sizes in standard deviation units (SD), but SD effects are hard to interpret. Effects are easier to interpret in percentile points, but converting SDs to percentile points involves a calculation that is not transparent to educational stakeholders. We show that if the outcome variable is normally distributed, we can approximate the percentile-point effect simply by multiplying the SD effect by 37 (or, equivalently, dividing the SD effect by 0.027). For students in the middle three-fifths of a normal distribution, this rule of thumb is always accurate to within 1.6 percentile points for effect sizes of up to 0.8 SD. Two examples show that the rule can be just as accurate for empirical effects from real studies. Applying the rule to empirical benchmarks, we find that the least effective third of educational interventions raise scores by 0 to 2 percentile points; the middle third raise scores by 2 to 7 percentile points; and the most effective third raise scores by more than 7 percentile points.

Effect size guidelines for cross-lagged effects.

Cross-lagged models are by far the most commonly used method to test the prospective effect of one construct on another, yet there are no guidelines for interpreting the size of cross-lagged effects. This research aims to establish empirical benchmarks for cross-lagged effects, focusing on the cross-lagged panel model (CLPM) and the random intercept cross-lagged panel model (RI-CLPM). We drew a quasirepresentative sample of studies published in four subfields of psychology (i.e., developmental, social-personality, clinical, and industrial-organizational). The dataset included 1,028 effect sizes for the CLPM and 302 effect sizes for the RI-CLPM, based on data from 174 samples. For the CLPM, the 25th, 50th, and 75th percentiles of the distribution corresponded to cross-lagged effect sizes of .03, .07, and .12, respectively. For the RI-CLPM, the corresponding values were .02, .05, and .11. Effect sizes did not differ significantly between the CLPM and RI-CLPM. Moreover, effect sizes did not differ significantly across subfields and were not moderated by design characteristics. However, effect sizes were moderated by the concurrent correlation between the constructs and the stability of the predictor. Based on the findings, we propose to use .03 (small effect), .07 (medium effect), and .12 (large effect) as benchmark values when interpreting the size of cross-lagged effects, for both the CLPM and RI-CLPM. In addition to aiding in the interpretation of results, the present findings will help researchers plan studies by providing information needed to conduct power analyses and estimate minimally required sample sizes. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Self-Paced Unified Representation Learning for Hierarchical Multi-Label Classification

Hierarchical Multi-Label Classification (HMLC) is a well-established problem that aims at assigning data instances to multiple classes stored in a hierarchical structure. Despite its importance, existing approaches often face two key limitations: (i) They employ dense networks to solely explore the class hierarchy as hard criterion for maintaining taxonomic consistency among predicted classes, yet without leveraging rich semantic relationships between instances and classes; (ii) They struggle to generalize in settings with deep class levels, since the mini-batches uniformly sampled from different levels ignore the varying complexities of data and result in a non-smooth model adaptation to sparse data. To mitigate these issues, we present a Self-Paced Unified Representation (SPUR) learning framework, which focuses on the interplay between instance and classes to flexibly organize the training process of HMLC algorithms. Our framework consists of two lightweight encoders designed to capture the semantics of input features and the topological information of the class hierarchy. These encoders generate unified embeddings of instances and class hierarchy, which enable SPUR to exploit semantic dependencies between them and produce predictions in line with taxonomic constraints. Furthermore, we introduce a dynamic hardness measurement strategy that considers both class hierarchy and instance features to estimate the learning difficulty of each instance. This strategy is achieved by incorporating the propagation loss obtained at each hierarchical level, allowing for a more comprehensive assessment of learning complexity. Extensive experiments on several empirical benchmarks demonstrate the effectiveness and efficiency of SPUR compared to state-of-the-art methods, especially in scenarios with missing features.

Gemini versus ChatGPT: applications, performance, architecture, capabilities, and implementation

This research paper presents an in-depth comparative examination of Gemini and ChatGPT, two prominent conversational AI models, exploring their respective applications, performance metrics, architectural variances, and overall capabilities. As conversational AI becomes increasingly prevalent across industries, comprehending the nuances of these models becomes pivotal for effective deployment. The paper initiates by outlining the wide array of applications for both Gemini and ChatGPT, spanning industries such as customer service, construction, finance, education, healthcare, and entertainment. It analyzes how each model addresses specific use cases, emphasizing their flexibility and potential impact across different sectors. Following this, the study assesses the performance of Gemini and ChatGPT through both empirical benchmarks and real-world deployment scenarios. Key metrics, including response coherence, accuracy, latency, and scalability, are scrutinized to gauge the models' ability to generate contextually appropriate and coherent responses in conversational contexts. Moreover, the paper elucidates the architectural distinctions between Gemini and ChatGPT, covering variances in training methodologies, model architectures, and underlying technologies. Understanding these architectural nuances provides deeper insights into the computational mechanisms underpinning each model's performance. Lastly, the paper explores the capabilities of Gemini and ChatGPT in handling complex linguistic phenomena, deciphering user intents, and sustaining engaging dialogues over prolonged interactions. This discussion encompasses language generation, sentiment analysis, context retention, and ethical considerations, shedding light on the potential of these models to facilitate meaningful human-computer interactions. Through this thorough comparative analysis, the research contributes to the ongoing conversation surrounding conversational AI systems. It offers valuable insights into the strengths and limitations of Gemini and ChatGPT, empowering stakeholders to make informed decisions regarding their optimal utilization across diverse applications.

A multiscale inflammatory map: linking individual stress to societal dysfunction.

As populations worldwide show increasing levels of stress, understanding emerging links among stress, inflammation, cognition, and behavior is vital to human and planetary health. We hypothesize that inflammation is a multiscale driver connecting stressors that affect individuals to large-scale societal dysfunction and, ultimately, to planetary-scale environmental impacts. We propose a 'central inflammation map' hypothesis to explain how the brain regulates inflammation and how inflammation impairs cognition, emotion, and action. According to our hypothesis, these interdependent inflammatory and neural processes, and the inter-individual transmission of environmental, infectious, and behavioral stressors - amplified via high-throughput digital global communications - can culminate in a multiscale, runaway, feed-forward process that could detrimentally affect human decision-making and behavior at scale, ultimately impairing the ability to address these same stressors. This perspective could provide non-intuitive explanations for behaviors and relationships among cells, organisms, and communities of organisms, potentially including population-level responses to stressors as diverse as global climate change, conflicts, and the COVID-19 pandemic. To illustrate our hypothesis and elucidate its mechanistic underpinnings, we present a mathematical model applicable to the individual and societal levels to test the links among stress, inflammation, control, and healing, including the implications of transmission, intervention (e.g., via lifestyle modification or medication), and resilience. Future research is needed to validate the model's assumptions, expand the factors/variables employed, and validate it against empirical benchmarks. Our model illustrates the need for multilayered, multiscale stress mitigation interventions, including lifestyle measures, precision therapeutics, and human ecosystem design. Our analysis shows the need for a coordinated, interdisciplinary, international research effort to understand the multiscale nature of stress. Doing so would inform the creation of interventions that improve individuals' lives and communities' resilience to stress and mitigate its adverse effects on the world.

Serial processing of proximity groups and similarity groups.

Proximity and feature similarity are two important determinants of perceptual grouping in vision. When viewing visual scenes conveying both grouping options simultaneously, people most usually detect proximity groups faster than similarity groups. This article demonstrates that perceptual judgments of grouping orientation guided by either proximity or contrast similarity are indicative of a sequential organization of grouping operations in the visual pathway, which lends a temporal processing advantage to proximity grouping (Experiment 1). Invoking the double-factorial paradigm, latent cognitive architecture for perceptual grouping is also investigated in a task with redundant signals (Experiment 2). Reaction time data from this task is assessed in terms of the race model inequality, workload capacity analysis, and interaction contrasts of means and survivor functions. Again, empirical benchmarks indicate serial processing of proximity groups and similarity groups, with a self-terminating stopping rule for processing. A subset of participants exhibit atypical performance metrics, hinting at possible individual differences in configural visual processing.

Digital Twin Data Management: Framework and Performance Metrics of Cloud-Based ETL System

This study delves into the EA-SAS platform, a digital twin environment developed by our team, with a particular focus on the EA-SAS Cloud Scheduler, our bespoke program designed to optimize ETL (extract, transform, and load) scheduling and thereby enhance automation within industrial systems. We elucidate the architectural intricacies of the EA-SAS Cloud Scheduler, demonstrating its adeptness in efficiently managing computationally heavy tasks, a capability underpinned by our empirical benchmarks. The architecture of the scheduler incorporates Docker to create isolated task environments and leverages RabbitMQ for effective task distribution. Our analysis reveals the EA-SAS Cloud Scheduler’s prowess in maintaining minimal overhead times, even in scenarios characterized by high operational loads, underscoring its potential to markedly bolster operational efficiency in industrial settings. While acknowledging the limitations inherent in our current assessment, particularly in simulating real-world industrial complexities, the study also charts potential future research pathways. These include a thorough exploration of the EA-SAS Cloud Scheduler’s adaptability across diverse industrial scenarios and an examination of the integration challenges associated with its reliance on specific technological frameworks.

Neurocomputational Models of Interval Timing: Seeing the Forest for the Trees.

Extracting temporal regularities and relations from experience/observation is critical for organisms' adaptiveness (communication, foraging, predation, prediction) in their ecological niches. Therefore, it is not surprising that the internal clock that enables the perception of seconds-to-minutes-long intervals (interval timing) is evolutionarily well-preserved across many species of animals. This comparative claim is primarily supported by the fact that the timing behavior of many vertebrates exhibits common statistical signatures (e.g., on-average accuracy, scalar variability, positive skew). These ubiquitous statistical features of timing behaviors serve as empirical benchmarks for modelers in their efforts to unravel the processing dynamics of the internal clock (namely answering how internal clock "ticks"). In this chapter, we introduce prominent (neuro)computational approaches to modeling interval timing at a level that can be understood by general audience. These models include Treisman's pacemaker accumulator model, the information processing variant of scalar expectancy theory, the striatal beat frequency model, behavioral expectancy theory, the learning to time model, the time-adaptive opponent Poisson drift-diffusion model, time cell models, and neural trajectory models. Crucially, we discuss these models within an overarching conceptual framework that categorizes different models as threshold vs. clock-adaptive models and as dedicated clock/ramping vs. emergent time/population code models.

Schizophrenia Spectrum Disorders: An Empirical Benchmark Study of Real-world Diagnostic Accuracy and Reliability Among Leading International Psychiatrists.

Assigning a psychiatric diagnosis in real-world situations is often difficult, given that the clinical presentation does not usually conform to the list of condensed, simplified behavioral descriptors of mainstream operational taxonomies (MOT) (eg, ICD-11 and DSM-5). The goal of this study was to benchmark diagnostic accuracy and reliability on a central and severe spectrum of psychopathology (ie, the schizophrenia spectrum disorders [SSDs]), adopting a pragmatic approach as close as possible to real-world clinical settings. We examined the diagnostic performance of 30 international psychiatrists experts in SSD. The clinicians were asked to make their clinical best diagnostic estimate for two written clinical vignettes excerpted from real-world SSD cases. In the first vignette, 22 out of the 30 clinicians (73.5%) indicated a SSD as their main diagnostic hypothesis. In the second vignette, 12 clinicians (40%) chose SSD as their main diagnostic hypothesis. Only 10 of the 30 clinicians (33%) correctly identified both vignettes as cases of SSD. The level of interrater diagnostic agreement (Fleiss' Kappa) was lowbut statistically significant (KFleiss = 0.08, P = .01). The results suggest that, even in a sample of influential international psychiatrists, the diagnostic accuracy and reliability on SSD presentations is poor and substantially inferior to those obtained in reliability studies using structured or semi-structured interviews. The widespread adoption of MOT systems in the last decades may have inadvertently eroded the ability of clinicians to detect a typical pattern of psychiatric illnesses.

From cross-lagged effects to feedback effects: Further insights into the estimation and interpretation of bidirectional relations.

Bidirectional relations have long been of interest in psychology and other social behavioral sciences. In recent years, the widespread use of intensive longitudinal data has provided new opportunities to examine dynamic bidirectional relations between variables. However, most previous studies have focused on the effect of one variable on the other (i.e., cross-lagged effects) rather than the overall effect representing the dynamic interplay between two variables (i.e., feedback effects), which we believe may be due to a lack of relevant methodological guidance. To quantify bidirectional relations as a whole, this study attempted to provide guidance for the estimation and interpretation of feedback effects based on dynamic structural equation models. First, we illustrated the estimation procedure for the average and person-specific feedback effects. Then, to facilitate the interpretation of feedback effects, we established an empirical benchmark by quantitatively synthesizing the results of relevant empirical studies. Finally, we used a set of empirical data to demonstrate how feedback effects can help (a) test theories based on bidirectional relations and (b) reveal correlates of individual differences in bidirectional relations. We also discussed the broad application prospects of feedback effects from a dynamic systems perspective. This study provides guidance for applied researchers interested in further examining feedback effects in bidirectional relations, and the shift from focusing on cross-lagged effects only to a comprehensive consideration of feedback effects may provide new insights into the study of bidirectional relations.

How well do rudimentary plasticity rules predict adult visual object learning?

A core problem in visual object learning is using a finite number of images of a new object to accurately identify that object in future, novel images. One longstanding, conceptual hypothesis asserts that this core problem is solved by adult brains through two connected mechanisms: 1) the re-representation of incoming retinal images as points in a fixed, multidimensional neural space, and 2) the optimization of linear decision boundaries in that space, via simple plasticity rules applied to a single downstream layer. Though this scheme is biologically plausible, the extent to which it explains learning behavior in humans has been unclear-in part because of a historical lack of image-computable models of the putative neural space, and in part because of a lack of measurements of human learning behaviors in difficult, naturalistic settings. Here, we addressed these gaps by 1) drawing from contemporary, image-computable models of the primate ventral visual stream to create a large set of testable learning models (n = 2,408 models), and 2) using online psychophysics to measure human learning trajectories over a varied set of tasks involving novel 3D objects (n = 371,000 trials), which we then used to develop (and publicly release) empirical benchmarks for comparing learning models to humans. We evaluated each learning model on these benchmarks, and found those based on deep, high-level representations from neural networks were surprisingly aligned with human behavior. While no tested model explained the entirety of replicable human behavior, these results establish that rudimentary plasticity rules, when combined with appropriate visual representations, have high explanatory power in predicting human behavior with respect to this core object learning problem.

Empirical Benchmarks for Planning and Interpreting Causal Effects of Community College Interventions

Randomized controlled trials (RCTs) are an increasingly common research design for evaluating the effectiveness of community college (CC) interventions. However, when planning an RCT evaluation of a CC intervention, there is limited empirical information about what sized effects an intervention might reasonably achieve, which can lead to under- or over-powered studies. Relatedly, when interpreting results from an evaluation of a CC intervention, there is limited empirical information to contextualize the magnitude of an effect estimate relative to what sized effects have been observed in past evaluations. We provide empirical benchmarks to help with the planning and interpretation of community college evaluations. To do so, we present findings across well-executed RCTs of 39 CC interventions that are part of a unique dataset known as The Higher Education Randomized Controlled Trials (THE-RCT). The analyses include 21,163–65,604 students (depending on outcome and semester) enrolled in 44 institutions. Outcomes include enrollment, credits earned, and credential attainment. Effect size distributions are presented by outcome and semester. For example, across the interventions examined, the mean effect on cumulative credits earned after three semesters is 1.14 credits. Effects around 0.16 credits are at the 25th percentile of the distribution. Effects around 1.69 credits are at the 75th percentile of the distribution. This work begins to provide empirical benchmarks for planning and interpreting effects of CC evaluations. A public database with effect sizes is available to researchers (https://www.mdrc.org/the-rct-empirical-benchmarks).

The Case for Administrative Benchmarks (and Some Challenges)

Summary Most U.S. Medicare beneficiaries are in population-based programs where spending targets, known as benchmarks, are used to determine payments. These include both the Medicare Advantage (MA) program, which relies on private health plans to manage care, and Accountable Care Organization (ACO) models, where providers are rewarded if they efficiently deliver care. When well designed, these models can incentivize reductions in spending and improve patient outcomes. A crucial aspect of these models, however, is how the benchmarks are determined. Policy makers have multiple approaches at their disposal, and each comes with unique strengths and weaknesses. The most common benchmark setting approach is empirical in that benchmarks are tied to actual spending or near-term forecasts based on lagged spending in fee-for-service traditional Medicare (TM). Empirical benchmarks reduce the risk that plans and providers face from fluctuations in costs that are beyond their control. That risk is absorbed by Medicare, so program spending will rise or fall with TM spending. Empirical benchmark systems have several significant weaknesses — including a ratchet effect and a collective success problem — that make it difficult for plans and providers to share in savings over the long run. This limits the long-term success of these population-based models. In contrast, administrative benchmarks are set by taking a base rate (such as historical spending) and growing it by an administrative factor that can reflect policy goals, anticipated volume, and intensity growth, or a socially agreed-upon rate, such as gross domestic product growth. The key feature is that, without policy action, the benchmark trajectory is not influenced by realized spending trends. An example of the administrative benchmark approach is the Accountable Care Prospective Trend component of the benchmarks for ACOs in the Medicare Shared Savings Program, which uses an initial benchmark, based on baseline spending, that grows based on spending growth that was forecast when the program was launched rather than realized spending growth. This administrative benchmark allows benchmark growth to diverge from, rather than track with, realized spending growth, which avoids shortcomings of empirical benchmarks and allows for greater incentives to deliver care efficiently, providing predictability over time. The key to success for administrative benchmarks is that the administrative growth rate needs to be set low enough so that Medicare saves money relative to what would happen without ACOs or MA, but high enough to still encourage participation.

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation). In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets). Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partial-dependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set ( N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials ( https://osf.io/9273g/ ).

Modelling constituent order despite symmetric associations in memory

Mathematical models of association memory (study AB, given A, recall B) either predict that knowledge for constituent order of a word pair (AB vs. BA) is perfectly unrelated, or completely dependent on knowledge of the pairing itself. Data contradict both predictions; when a pair is remembered, constituent-order is above chance, but still fairly low. Convolution-based models are inherently symmetric and can explain associative symmetry, but cannot discriminate AB from BA. We evaluated four extensions of convolution, where order is incorporated as item features, partial permutations of features, item-position associations, or by adding item and position vectors. All approaches could discriminate order within behaviourally observed ranges, without compromising associative symmetry. Only the permutation model could disambiguate AB from BC in double-function lists, as humans can do. It is possible that each of our proposed mechanisms might apply to a different, particular task setting. However, the partial permutation model can thus far explain the broadest set of empirical benchmarks.