Benchmark For Evaluation Research Articles

Me-LLaMA: Foundation Large Language Models for Medical Applications.

Recent advancements in large language models (LLMs) such as ChatGPT and LLaMA have hinted at their potential to revolutionize medical applications, yet their application in clinical settings often reveals limitations due to a lack of specialized training on medical-specific data. In response to this challenge, this study introduces Me-LLaMA, a novel medical LLM family that includes foundation models - Me-LLaMA 13/70B, along with their chat-enhanced versions - Me-LLaMA 13/70B-chat, developed through continual pre-training and instruction tuning of LLaMA2 using large medical datasets. Our methodology leverages a comprehensive domain-specific data suite, including a large-scale, continual pre-training dataset with 129B tokens, an instruction tuning dataset with 214k samples, and a new medical evaluation benchmark (MIBE) across six critical medical tasks with 12 datasets. Our extensive evaluation using the MIBE shows that Me-LLaMA models achieve overall better performance than existing open-source medical LLMs in zero-shot, few-shot and supervised learning abilities. With task-specific instruction tuning, Me-LLaMA models outperform ChatGPT on 7 out of 8 datasets and GPT-4 on 5 out of 8 datasets. In addition, we investigated the catastrophic forgetting problem, and our results show that Me-LLaMA models outperform other open-source medical LLMs in mitigating this issue. Me-LLaMA is one of the largest open-source medical foundation LLMs that use both biomedical and clinical data. It exhibits superior performance across both general and medical tasks compared to other open-source medical LLMs, rendering it an attractive choice for medical AI applications. We release our models, datasets, and evaluation scripts at: https://github.com/BIDS-Xu-Lab/Me-LLaMA.

Introducing User Feedback-Based Counterfactual Explanations (UFCE)

Machine learning models are widely used in real-world applications. However, their complexity makes it often challenging to interpret the rationale behind their decisions. Counterfactual explanations (CEs) have emerged as a viable solution for generating comprehensible explanations in eXplainable Artificial Intelligence (XAI). CE provides actionable information to users on how to achieve the desired outcome with minimal modifications to the input. However, current CE algorithms usually operate within the entire feature space when optimising changes to turn over an undesired outcome, overlooking the identification of key contributors to the outcome and disregarding the practicality of the suggested changes. In this study, we introduce a novel methodology, that is named as user feedback-based counterfactual explanation (UFCE), which addresses these limitations and aims to bolster confidence in the provided explanations. UFCE allows for the inclusion of user constraints to determine the smallest modifications in the subset of actionable features while considering feature dependence, and evaluates the practicality of suggested changes using benchmark evaluation metrics. We conducted three experiments with five datasets, demonstrating that UFCE outperforms two well-known CE methods in terms of proximity, sparsity, and feasibility. Reported results indicate that user constraints influence the generation of feasible CEs.

Mapping an intelligent algorithm for predicting female adolescents' cervical vertebrae maturation stage with high recall and accuracy.

The present study was designed to define a novel algorithm capable of predicting female adolescents' cervical vertebrae maturation stage with high recall and accuracy. A total of 560 female cephalograms were collected, and cephalograms with unclear vertebral shapes and deformed scales were removed. 480 films from female adolescents (mean age: 11.5 years; age range: 6-19 years) were used for the model development phase, and 80 subjects were randomly and stratified allocated to the validation cohort to further assess the model's performance. Derived significant predictive parameters from 15 anatomic points and 25 quantitative parameters of the second to fourth cervical vertebrae (C2-C4) to establish the ordinary logistic regression model. Evaluation metrics including precision, recall, and F1 score are employed to assess the efficacy of the models in each identified cervical vertebrae maturation stage (iCS). In cases of confusion and mispredictions, the model underwent modification to improve consistency. Four significant parameters, including chronological age, the ratio of D3 to AH3 (D3:AH3), anterosuperior angle of C4 (@4), and distance between C3lp and C4up (C3lp-C4up) were administered into the ordinary regression model. The primary predicting model that implements the novel algorithm was built and the performance evaluation with all stages of 93.96% for accuracy, 93.98% for precision, 93.98% for recall, and 93.95% for F1-score were obtained. Despite the hybrid logistic-based model achieving high accuracy, the unsatisfactory performance of stage estimation was noticed for iCS3 in the primary cohort (89.17%) and validation cohort (85.00%). Through bivariate logistic regression analysis, the posterior height of C4 (PH4) was further selected in the iCS3 to establish a corrected model, thus the evaluation metrics were upgraded to 95.83% and 90.00%, respectively. An unbiased and objective assessment of the cervical vertebrae maturation (CVM) method can function as a decision-support tool, assisting in the evaluation of the optimal timing for treatment in growing adults. Our novel proposed logistic model yielded individual formulas for each specific CVM stage and attained exceptional performance, indicating the capability to function as a benchmark for maturity evaluation in clinical craniofacial orthopedics for Chinese female adolescents.

End-to-end multi-perspective multimodal posts relevance score reasoning prediction

Driven by the surge in students sharing insights on online platforms, evaluating discussion forum posts has emerged as a significant research focus. This assessment is crucial for both students and educators, as it enables teachers to efficiently gauge student comprehension and helps students tailor their learning approaches to suit their individual needs, thereby fostering personalized growth. However, existing research on post-quality assessment faces challenges: 1) They overlook multimodal information interaction between post and topic, leading to inaccurate evaluations. 2) They are usually classification tasks that fail to provide feedback on the minor and relative differences between posts on the same topic. 3) They are usually evaluated using a single perspective, which is insufficient for capturing the complexity of the relationship between post and topic. Based on the above challenges, we propose a new assessment task, Multimodal Topic-Post Relevance Score Prediction (MTRSP), which analyzes whether a student's post comprehensively answers the question of the discussion topic by combining text and images to predict topic-post relevance scores, i.e., the degree of correctness of the answer. We develop an end-to-end Multi-perspective Topic-Post Relevance Score Reasoning (MTRSR) Method to solve the MTRSP Task, which leverages images and text from both the post and topic to infer topic-post relevance scores based on semantic similarity and logical coherence. Specifically, the topic-post content relevance reasoning module uses multimodal fusion to learn the semantic similarity of posts and topics. The logical coherence inference module examines the logical connections between posts and topics. Finally, we use three newly collected multimodal topic-post datasets and the public dataset Lazada-Home as an evaluation benchmark for the MTRSP task. Experimental results show that our MTRSR method can bring up to 9.02% in the NDCG@3 (Normalized Discounted Cumulative Gain) metric compared to the best-performing text-only model. The source code and dataset will be made public.

Validity and Reliability of Physical Sports and Health Education's Learning Assessment of Self Defense Material for Class VII Junior High School Stud

Instruments adhering to appropriate standards are crucial in gauging the conclusive outcomes of learning activities. This ensures that student learning achievements are accurately identified, serving as evaluation benchmarks for subsequent educational programs. The present research endeavors to develop an assessment instrument for Physical Education and Health (PJOK) learning, specifically focusing on self-defense material designed for seventh-grade junior high school students. This instrument is envisaged as a valuable tool to assist educators in conducting assessments, leading to the optimization of learning skills evaluation outcomes. The research methodology employs the Research and Development approach, referencing the ADDIE development model. The analysis stage involves observation and interviews, instrument design, product development, as well as Focus Group Discussions (FGD) with learning evaluation experts. Additionally, instrument product trials are conducted to examine validity and reliability. The research involves 66 seventh-grade junior high school students as participants. The validity test was carried out using the Reproducibility Coefficient and Scalability Coefficient according to the Guttman scale test, while the reliability test used Kuder Richardson (K-R) 20. The validity test results showed that all variables had values ​​> 0.90 for the reproducibility coefficient and > 0.60 for the scalability coefficient so it is said to be valid. The reliability test results showed a value of > 0.60 for K-R 20 so it is considered reliable. In conclusion, the PJOK learning assessment instrument for self-defense material emerges as a reliable and valid tool. Educators can confidently integrate this instrument into their assessments, leveraging its proven to be valid and reliable.

Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT

BackgroundPretraining large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing. With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from free text has improved significantly in both the general and medical domains. However, it is difficult to train specific BERT models to perform well in domains for which few databases of a high quality and large size are publicly available. ObjectiveWe hypothesized that this problem could be addressed by oversampling a domain-specific corpus and using it for pretraining with a larger corpus in a balanced manner. In the present study, we verified our hypothesis by developing pretraining models using our method and evaluating their performance. MethodsOur proposed method was based on the simultaneous pretraining of models with knowledge from distinct domains after oversampling. We conducted three experiments in which we generated (1) English biomedical BERT from a small biomedical corpus, (2) Japanese medical BERT from a small medical corpus, and (3) enhanced biomedical BERT pretrained with complete PubMed abstracts in a balanced manner. We then compared their performance with those of conventional models. ResultsOur English BERT pretrained using both general and small medical domain corpora performed sufficiently well for practical use on the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our proposed method was more effective than the conventional methods for each biomedical corpus of the same corpus size in the general domain. Our Japanese medical BERT outperformed the other BERT models built using a conventional method for almost all the medical tasks. The model demonstrated the same trend as that of the first experiment in English. Further, our enhanced biomedical BERT model, which was not pretrained on clinical notes, achieved superior clinical and biomedical scores on the BLUE benchmark with an increase of 0.3 points in the clinical score and 0.5 points in the biomedical score. These scores were above those of the models trained without our proposed method. ConclusionsWell-balanced pretraining using oversampling instances derived from a corpus appropriate for the target task allowed us to construct a high-performance BERT model.

A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models

Due to the complexity of the biomedical domain, the ability to capture semantically meaningful representations of terms in context is a long-standing challenge. Despite important progress in the past years, no evaluation benchmark has been developed to evaluate how well language models represent biomedical concepts according to their corresponding context. Inspired by the Word-in-Context (WiC) benchmark, in which word sense disambiguation is reformulated as a binary classification task, we propose a novel dataset, BioWiC, to evaluate the ability of language models to encode biomedical terms in context. BioWiC comprises 20’156 instances, covering over 7’400 unique biomedical terms, making it the largest WiC dataset in the biomedical domain. We evaluate BioWiC both intrinsically and extrinsically and show that it could be used as a reliable benchmark for evaluating context-dependent embeddings in biomedical corpora. In addition, we conduct several experiments using a variety of discriminative and generative large language models to establish robust baselines that can serve as a foundation for future research.

A concise but high-performing network for image guided depth completion in autonomous driving

Depth completion is a crucial task in autonomous driving, aiming to convert a sparse depth map into a dense depth prediction. The sparse depth map serves as a partial reference for the actual depth, and the fusion of RGB images is frequently employed to augment the completion process owing to its inherent richness in semantic information. Image-guided depth completion confronts three principal challenges: (1) the effective fusion of the two modalities; (2) the enhancement of depth information recovery; and (3) the realization of real-time predictive capabilities requisite for practical autonomous driving scenarios. In response to these challenges, we propose a concise but high-performing network, named CHNet, to achieve high-performance depth completion with an elegant and straightforward architecture. Firstly, we use a fast guidance module to fuse the two sensor features, harnessing abundant auxiliary information derived from the color space. Unlike the prevalent complex guidance modules, our approach adopts an intuitive and cost-effective strategy. In addition, we find and analyze the optimization inconsistency problem for observed and unobserved positions. To mitigate this challenge, we introduce a decoupled depth prediction head, tailored to better discern and predict depth values for both valid and invalid positions, incurring minimal additional inference time. Capitalizing on the dual-encoder and single-decoder architecture, the simplicity of CHNet facilitates an optimal balance between accuracy and computational efficiency. In benchmark evaluations on the KITTI depth completion dataset, CHNet demonstrates competitive performance metrics and inference speeds relative to contemporary state-of-the-art methodologies. To assess the generalizability of our approach, we extend our evaluations to the indoor NYUv2 dataset, where CHNet continues to yield impressive outcomes. The code of this work will be available at https://github.com/lmomoy/CHNet.

A Review of Machine Learning Approaches for Rumour Detection: Techniques, Challenges, and Future Directions (Machine Learning for Rumour Detection)

Rumours and misinformation propagate rapidly across online social networks, posing significant challenges to maintaining the integrity of information dissemination. In recent years, machine learning (ML) techniques have emerged as promising tools for automating the detection and mitigation of rumours. This review paper provides a comprehensive examination of the advancements in rumour detection using ML approaches. The paper begins by outlining the landscape of rumour dissemination in online social networks, highlighting the characteristics and challenges associated with rumour detection. Subsequently, it systematically categorizes and analyzes various ML methods employed for rumour detection, including supervised, unsupervised, and semi-supervised learning approaches. Furthermore, the review delves into the diverse features and representations utilized in ML models for rumour detection, such as textual content, user engagement patterns, network structures, and temporal dynamics. It discusses the strengths and limitations of different feature sets and their impact on the effectiveness of rumour detection systems. Moreover, the paper explores the intricacies of dataset construction and evaluation methodologies for training and testing rumour detection models. It examines commonly used benchmark datasets and evaluation metrics, emphasizing the importance of robust evaluation frameworks for assessing the performance of ML-based rumour detection systems accurately. Additionally, the review identifies key challenges and open research questions in the field of rumour detection using ML, including handling evolving rumour patterns, addressing adversarial attacks, and enhancing the interpretability and explain ability of ML models. It also discusses potential directions for future research aimed at advancing the state-of-the-art in rumour detection and mitigation.

Real-time Pose Estimation for Mobile Devices: A Review

Human pose estimation (HPE) is a critical computer vision task that underpins various applications such as motion analysis and human-computer interaction. Deep learning has significantly improved HPE accuracy, but challenges, specifically in limited training data and occlusions remain. This work focuses on real-time HPE on mobile devices. We compare three prominent frameworks (BlazePose, OpenPose, AlphaPose) by analyzing their input data requirements, underlying inference procedures, and performance on popular datasets. We also explore commonly used HPE benchmark datasets and evaluation metrics. This comparative analysis provides a clear understanding of the current state-of-the-art for real-time mobile HPE, benefiting researchers and developers working on mobile applications that leverage HPE functionalities.

A benchmark for evaluation of structure-based online tools for antibody-antigen binding affinity

The prediction of binding affinity changes caused by missense mutations can elucidate antigen-antibody interactions. A few accessible structure-based online computational tools have been proposed. However, selecting suitable software for particular research is challenging, especially research on the SARS-CoV-2 spike protein with antibodies. Therefore, benchmarking of the mutation-diverse SARS-CoV-2 datasets is critical. Here, we collected the datasets including 1216 variants about the changes in binding affinity of antigens from 22 complexes for SARS-CoV-2 S proteins and 22 monoclonal antibodies as well as applied them to evaluate the performance of seven binding affinity prediction tools. The tested tools' Pearson correlations between predicted and measured changes in binding affinity were between −0.158 and 0.657, while accuracy in classification tasks on predicting increasing or decreasing affinity ranged from 0.444 to 0.834. These tools performed relatively better on predicting single mutations, especially at epitope sites, whereas poor performance on extremely decreasing affinity. The tested tools were relatively insensitive to the experimental techniques used to obtain structures of complexes. In summary, we constructed a list of datasets and evaluated a range of structure-based online prediction tools that will explicate relevant processes of antigen-antibody interactions and enhance the computational design of therapeutic monoclonal antibodies.

Bridging Requirements, Planning, and Evaluation: A Review of Social Robot Navigation.

Navigation lies at the core of social robotics, enabling robots to navigate and interact seamlessly in human environments. The primary focus of human-aware robot navigation is minimizing discomfort among surrounding humans. Our review explores user studies, examining factors that cause human discomfort, to perform the grounding of social robot navigation requirements and to form a taxonomy of elementary necessities that should be implemented by comprehensive algorithms. This survey also discusses human-aware navigation from an algorithmic perspective, reviewing the perception and motion planning methods integral to social navigation. Additionally, the review investigates different types of studies and tools facilitating the evaluation of social robot navigation approaches, namely datasets, simulators, and benchmarks. Our survey also identifies the main challenges of human-aware navigation, highlighting the essential future work perspectives. This work stands out from other review papers, as it not only investigates the variety of methods for implementing human awareness in robot control systems but also classifies the approaches according to the grounded requirements regarded in their objectives.

Navigating the Power of Artificial Intelligence in Risk Management: A Comparative Analysis

This study presents a responsive analysis of the role of artificial intelligence (AI) in risk management, contrasting traditional approaches with those augmented by AI and highlighting the challenges and opportunities that emerge. AI, intense learning methodologies such as convolutional neural networks (CNNs), have been identified as pivotal in extracting meaningful insights from image data, a form of analysis that holds significant potential in identifying and managing risks across various industries. The research methodology involves a strategic selection and processing of images for analysis and introduces three case studies that serve as benchmarks for evaluation. These case studies showcase the application of AI, in place of image processing capabilities, to identify hazards, evaluate risks, and suggest control measures. The comparative evaluation focuses on the accuracy, relevance, and practicality of the AI-generated findings alongside the system’s response time and comprehensive understanding of the context. Results reveal that AI can significantly enhance risk assessment processes, offering rapid and detailed insights. However, the study also recognises the intrinsic limitations of AI in contextual interpretation, advocating for a synergy between technological and domain-specific expertise. The conclusion underscores the transformative potential of AI in risk management, supporting continued research to further integrate AI effectively into risk assessment frameworks.

ESG-Ratings: Nonparametric Methods of Construction

Many of the largest Russian companies are evaluated by international financial institutions or rating agencies in terms of their influence on ESG factors that take into account environmental issues, interaction with society and corporate governance. Such ratings can have various names, most often referred to as ESG ratings. The inherent subjectivity of the assessments, along with the lack of generally recognized standards and transparency of the methodology, cause concern both from the assessed companies and from investors and regulators. ESG ratings of Russian rating agencies are at an early stage of their development, which is reflected in a small number of evaluated companies. The purpose of the study is to study the main methodological problems in the compilation of ESG ratings identified by the academic and business community with a focus on studying the issue of choosing weights when constructing a summary indicator. The information base of the study is the data published by the rating agency RAEX, which is positioned as the largest agency in the field of non-credit ratings. A comparative analysis of the rating agency’s weight selection methods with nonparametric methods, such as methods of shell analysis, determination of preferences based on similarity with the ideal solution and calculation of the geometric mean is carried out. Based on the results of the study of the initial data of the rating agency, it can be concluded that most companies have low ratings for the environmental component and high ratings for the corporate governance component, while none of the companies has a benchmark rating. The main methodological problem in the selection of weights is the subjective nature of the weights used by the rating agency. Correlation analysis of the studied non-parametric methods showed a high correlation with each other and with the initial ratings of the rating agency, however, at the level of some individual companies, the ratings may differ depending on the chosen method.

An overview on deep clustering

In recent years, with the great success of deep learning and especially deep unsupervised learning, many deep architectural clustering methods, collectively known as deep clustering, have emerged. Deep clustering shows the potential to outperform traditional methods, especially in handling complex high-dimensional data, taking full advantage of deep learning. To achieve a comprehensive overview of the field of deep clustering, this review systematically explores deep clustering methods and their various applications. First, the basic network architecture of deep clustering is described in detail, including the common network frameworks, and loss functions. Subsequently, deep clustering is divided into several categories based on the network architecture, and benchmark datasets and evaluation metrics in the field are introduced. Next, the real-world applications of deep clustering are explored in depth, providing successful cases in the fields of bioinformatics, medicine, anomaly detection, and image processing, highlighting the broad applicability of deep clustering in solving real-world challenges. Finally, the paper summarizes its contributions and explores potential directions for future research in deep clustering.

The computational and energy cost of simulation and storage for climate science: lessons from CMIP6

Abstract. The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts aimed at better understanding the past, present, and future of climate changes in a multi-model context. A total of 21 model intercomparison projects (MIPs) were endorsed in its sixth phase (CMIP6), which included 190 different experiments that were used to simulate 40 000 years and produced around 40 PB of data in total. This paper presents the main findings obtained from the CPMIP (the Computational Performance Model Intercomparison Project), a collection of a common set of metrics, specifically designed for assessing climate model performance. These metrics were exclusively collected from the production runs of experiments used in CMIP6 and primarily from institutions within the IS-ENES3 consortium. The document presents the full set of CPMIP metrics per institution and experiment, including a detailed analysis and discussion of each of the measurements. During the analysis, we found a positive correlation between the core hours needed, the complexity of the models, and the resolution used. Likewise, we show that between 5 %–15 % of the execution cost is spent in the coupling between independent components, and it only gets worse by increasing the number of resources. From the data, it is clear that queue times have a great impact on the actual speed achieved and have a huge variability across different institutions, ranging from none to up to 78 % execution overhead. Furthermore, our evaluation shows that the estimated carbon footprint of running such big simulations within the IS-ENES3 consortium is 1692 t of CO2 equivalent. As a result of the collection, we contribute to the creation of a comprehensive database for future community reference, establishing a benchmark for evaluation and facilitating the multi-model, multi-platform comparisons crucial for understanding climate modelling performance. Given the diverse range of applications, configurations, and hardware utilised, further work is required for the standardisation and formulation of general rules. The paper concludes with recommendations for future exercises aimed at addressing the encountered challenges which will facilitate more collections of a similar nature.

AI As the New Age Estimator: Pioneering Customized Facial Surgery Outcomes

Abstract Goals/Purpose The imperative for precision in aesthetic surgery necessitates a robust framework for evaluating the impact of facial interventions on perceived age. Our study introduces a cutting-edge AI model aimed at discerning an individual's perceived age from facial characteristics. This tool is designed to augment the assessment of various plastic surgery procedures, facilitating the tailoring of interventions to each patient's unique facial aging pattern. Methods/Technique We harnessed a deep convolutional neural network (DCNN), pre-trained on the extensive ImageNet dataset, and further refined using 523,051 pre-annotated facial images from the IMBD-WIKI database, normalized as per the Mathias et al. face detection paradigm. Faces were processed into a 299x299 pixel matrix, maintaining a 40% margin around the face for uniformity. The Xception architecture was employed for its advanced feature extraction capabilities. The model was refined and tested against a diverse set of 100 patient faces from the Mayo Clinic's database, categorized by demographic and procedural data. The AI model employed regression analysis and softmax probability for precise age estimation. Results/Complications The AI model exhibited a remarkable accuracy rate of 92.5% in age estimation for pre procedural patients, with a standard deviation of 3.2 years. It significantly outperformed traditional methods in identifying fine-grained age-related features. The AI model discerned an average perceived age reduction of 3.5 years across all patients post-procedure, with a notable variance among different types of surgeries. Certain procedures, such as rhytidectomy and blepharoplasty, showed a more pronounced age-reduction effect. Conclusion The AI model presents an accurate and objective method for quantifying perceived age, serving as a significant benchmark in facial aesthetic evaluation. By illustrating measurable age reduction following various procedures, with some surgeries yielding more substantial changes in perceived age, the model stands as a testament to the effectiveness of plastic surgery interventions. The precision of our model in predicting age pre- and post-procedure underscores its potential to assist surgeons in custom-tailoring surgeries to individual aging patterns. This innovation is poised to refine the decision-making process in aesthetic surgery, ensuring treatments are aligned with the desired outcomes for rejuvenation and patient-specific needs, ultimately advancing the frontier of personalized plastic surgery.

A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges

Graph Neural Networks (GNNs) perform well in community detection and molecule classification. Counterfactual Explanations (CE) provide counter-examples to overcome the transparency limitations of black-box models. Due to the growing attention in graph learning, we focus on the concepts of CE for GNNs. We analysed the SoA to provide a taxonomy, a uniform notation, and the benchmarking datasets and evaluation metrics. We discuss fourteen methods, their evaluation protocols, twenty-two datasets, and nineteen metrics. We integrated the majority of methods into the GRETEL library to conduct an empirical evaluation to understand their strengths and pitfalls. We highlight open challenges and future work.

Multi-model approach in a variable spatial framework for streamflow simulation

Abstract. Accounting for the variability of hydrological processes and climate conditions between catchments and within catchments remains a challenge in rainfall–runoff modelling. Among the many approaches developed over the past decades, multi-model approaches provide a way to consider the uncertainty linked to the choice of model structure and its parameter estimates. Semi-distributed approaches make it possible to account explicitly for spatial variability while maintaining a limited level of complexity. However, these two approaches have rarely been used together. Such a combination would allow us to take advantage of both methods. The aim of this work is to answer the following question: what is the possible contribution of a multi-model approach within a variable spatial framework compared to lumped single models for streamflow simulation? To this end, a set of 121 catchments with limited anthropogenic influence in France was assembled, with precipitation, potential evapotranspiration, and streamflow data at the hourly time step over the period 1998–2018. The semi-distribution set-up was kept simple by considering a single downstream catchment defined by an outlet and one or more upstream sub-catchments. The multi-model approach was implemented with 13 rainfall–runoff model structures, three objective functions, and two spatial frameworks, for a total of 78 distinct modelling options. A simple averaging method was used to combine the various simulated streamflow at the outlet of the catchments and sub-catchments. The lumped model with the highest efficiency score over the whole catchment set was taken as the benchmark for model evaluation. Overall, the semi-distributed multi-model approach yields better performance than the different lumped models considered individually. The gain is mainly brought about by the multi-model set-up, with the spatial framework providing a benefit on a more occasional basis. These results, based on a large catchment set, evince the benefits of using a multi-model approach in a variable spatial framework to simulate streamflow.

Experimental Covariance Determination for Critical Integral Experiments

Integral benchmarks for criticality safety and nuclear data validation require expensive uncertainty quantification studies. In general, uncertainty quantification techniques ignore correlations between experiments and shared components. Experiments, such as the Thermal/Epithermal eXperiments (TEX) campaigns, consist of many shared components, such as the Jemima highly enriched uranium (HEU) fuel plates, which create a strong correlation in their uncertainties. While these correlations are known to exist, they are often not estimated because of the complexity of such calculations. This paper describes an intuitive method of determining the covariance for each of the experimental components, providing a correlation for each family of components across the multiple cases examined within a benchmark. A proof-of-principle study using the TEX-HEU experimental campaign was performed and verified that the covariance and correlation matrices can be calculated with information commonly found in the International Criticality Safety Benchmark Evaluation Project benchmarks. This study showed that the introduction of model and experimental covariances reduces the χ2 per degree of freedom from 2.203 to 1.179, indicating that the omission causes overly pessimistic bias quantifications. This technique can be seamlessly integrated to current benchmark evaluations as well as reevaluations of legacy benchmarks.