Framework For Model Evaluation Research Articles

Low-carbon development path based on carbon emission accounting and carbon emission performance evaluation: a case study of Chinese coal production enterprises.

Carbon emission accounting is the basic premise of effective carbon emission reduction and management. This study aimed to establish the carbon emission model and performance evaluation framework of coal mine production enterprises and clarify the low-carbon development path of enterprises. In this study, we took a typical coal production enterprise (K enterprise) in the Shanxi province of China as the research object. We also estimated the carbon emissions of the enterprise mainly according to the Chinese Carbon Emission Accounting Standard (GB/T 32151.11-2018). The triangular model was used to construct the carbon performance evaluation framework. On this basis, we suggested the enterprise's low-carbon development path. The results showed that (1) the carbon emission of K enterprise in 2021 was 36,875.38 tCO2eq; the carbon emission intensity of each ton of coal produced was 0.089 tCO2eq. The critical carbon emissions were electricity consumption and methane fugitive emissions during production. (2) The evaluation indicators for carbon emission performance revealed an imbalance in K enterprise's economic, energy, and environmental development in 2021. The work on energy saving and consumption reduction was relatively weak. (3) Countermeasures for low-carbon development, including a carbon emission ledger, were proposed based on carbon emission accounting and performance evaluation results. This study can help typical underground coal production enterprises in Shanxi province obtain more accurate carbon emission data, providing practical guidance and reference for the same underground coal production enterprises to improve the carbon emission control effect.

The Icelandic Prevention Model Evaluation Framework and Implementation Integrity and Consistency Assessment

The Icelandic Prevention Model (IPM) follows a systematic but flexible process of community capacity building, data collection, analysis, dissemination, and community-engaged decision-making to guide the data-informed selection, prioritization, and implementation of intervention strategies in preventing adolescent substance use. This paper describes two new evaluation tools intended to assess the: 1) integrity of IPM implementation, and 2) unique aspects of IPM implementation in different community contexts. These evaluation tools include a: 1) five-phase IPM Evaluation Framework for Assessing Value Across Communities, Cultures, and Outcomes (IPM-EF); and 2) 10-Step IPM Implementation Integrity and Consistency Assessment (IPM-IICA) that utilizes both quantitative (scored) and qualitative (narrative) data elements to characterize implementation integrity and consistency at both community coalition and school community levels. The IPM-EF includes five phases. Phase 1: Describe the Intervention Context; Phase 2a: Document the Extent to Which the 10 Steps of the IPM were Implemented (using the IPM-IICA scored); Phase 2b: Document the Unique Community-Specific Methods Used within the 10 Steps of the IPM to Tailor Local Intervention Delivery (using the IPM-IICA narrative); Phase 3: Measure Changes in Community Risk and Protective Factors; Phase 4: Measure the Outcomes Associated with the IPM; and Phase 5: Investigate Multiple Full Cycles Over Time.

Abstract P198: Developing a Practical, Evidence-Based Framework for Evaluating Community Nutrition Security Interventions

Introduction: Addressing nutrition insecurity at a population level is complex and requires changing policies, systems, and environments. The American Heart Association (AHA) has identified 17 research informed systemic interventions that have the potential to increase nutrition security over time. These interventions can be leveraged individually or together based on community needs and assets. Evaluating these interventions is essential for understanding impact, improving implementation, and allocating resources to effective strategies. This project sought to establish a pragmatic yet rigorous framework for evaluating the health impact of these strategies in communities across the United States. Hypothesis: We assessed the hypothesis that we could evaluate community nutrition security interventions by using program records, secondary data sources, and judicious use of primary data collection. Methods: Researchers reviewed details from the AHA's community impact database detailing 188 research informed nutrition security projects in communities across 29 states and the District of Columbia. Impacts were assessed for reach of intervention and strength of evidence to determine relative population dose of interventions. Seven interventions were determined to be ‘high dose,’ and researchers identified community and proximal outcomes for three high-dose interventions where the AHA is typically the catalyst that is leading and influencing work with community partners. Researchers reviewed 11 national databases to identify existing community-level data aggregated at the county, city, zip code, or census track that may be associated with the three high dose impact opportunities. Researchers then reviewed the evidence-based literature associated with proximal outcomes - organizational, environmental, policy, or individual/household changes - that have been documented in research studies in association with the three high dose interventions. Using the insight from both reviews, researchers were able to develop a model evaluation framework for assessing the impact of community level interventions over time. Results: Researchers were able to identify 17 proximal outcomes and two community level outcomes with evidence associated with the three high dose interventions. Conclusion: Demonstrating the impact of community level nutrition security interventions is challenging given the complexity of the environments in which they are implemented and the limitations of evaluation resources. This approach prioritizes high dose evidence-based interventions and then leverages existing community level datasets and organizational measures to assess impact. It provides a framework for other organizations to prioritize specific systemic interventions to evaluate and further informs the evidence base for future interventions.

Getting the leaves right matters for estimating temperature extremes

Abstract. Atmospheric reanalyses combine observations and models through data assimilation techniques to provide spatio-temporally continuous fields of key surface variables. They can do so for extended historical periods whilst ensuring a coherent representation of the main Earth system cycles. ERA5 and its enhanced land surface component, ERA5-Land, are widely used in Earth system science and form the flagship products of the Copernicus Climate Change Service (C3S) of the European Commission. Such land surface modelling frameworks generally rely on a state variable called leaf area index (LAI), representing the number of leaves in a grid cell at a given time, to quantify the fluxes of carbon, water and energy between the vegetation and the atmosphere. However, the LAI within the modelling framework behind ERA5 and ERA5-Land is prescribed as a climatological seasonal cycle, neglecting any interannual variability and the potential consequences that this uncoupling between vegetation and atmosphere may have on the surface energy balance and the climate. To evaluate the impact of this mismatch in LAI, we analyse the corresponding effect it has on land surface temperature (LST) by comparing what is simulated to satellite observations. We characterise a hysteretic behaviour between LST biases and LAI biases that evolves differently along the year depending on the background climate. We further analyse the repercussions for the reconstructed climate during more extreme conditions in terms of LAI deviations, with a specific focus on the 2003, 2010 and 2018 heat waves in Europe for which LST mismatches are exacerbated. We anticipate that our results will assist users of ERA5 and ERA5-Land data in understanding where and when the larger discrepancies can be expected, but also guide developers towards improving the modelling framework. Finally, this study could provide a blueprint for a wider benchmarking framework for land surface model evaluation that exploits the capacity of LST to integrate the effects of both radiative and non-radiative processes affecting the surface energy.

BISON-FIPD integration enhanced low-burnup SFR metallic fuel swelling model evaluation framework

Experiments indicated that metallic fuel in sodium-cooled fast reactors (SFRs) rapidly swells radially and axially at low burnup. Despite that, prior studies have been focused on describing high burnup axial fuel elongation. With recent conventional and non-conventional metallic fuel concepts being considered for license applications, understanding multidimensional fuel swelling at a wide range of burnup levels is important to fuel analysis and qualification activities. Here, we report the development and demonstration efforts of a low-burnup SFR metallic fuel swelling model evaluation framework using the BISON advanced fuel performance code. The framework leverages the Integral Fast Reactor (IFR) program X423 experiment data through the ongoing integration project to enable standardized and automated use of legacy metallic fuel irradiation data maintained in the SFR fuel irradiation and physics database (FIPD) for BISON metallic fuel model verification and validation. The performance of the framework was demonstrated using the two representative metallic fuel swelling model sets implemented in BISON, with a series of insights about future advanced swelling model development.

Theoretical and Empirical Models of Organizational Learning Processes in Knowledge Management

Introduction. This study undertakes a comprehensive exploration of existing instructional organizational models, spanning various disciplines within contemporary educational theory and knowledge management practice. The core objective is to propose an all-encompassing model tailored specifically to the preparation of future educational managers. This model places a significant emphasis on integrated educational strategies, further enriched by the integration of organizational learning processes in the context of knowledge management.
 Aim and tasks. This study critically examines established models of organizational learning processes with the goal of developing a tailored model for training future educational managers. The goal is to equip aspiring educational managers with integrated didactic skills based on analyses of existing educational models that include concept analysis, model evaluation, and theoretical framework establishment.
 Result. Organizational learning principles drive data-driven refinement, collaborative cross-disciplinary strategies, and leadership development. Sharing best practices enhances strength, whereas iterative feedback processes mitigate its limitations. This dynamic framework encourages adaptable education, fostering continuous improvement in teaching methods, curricula, and managerial training for a sustained educational evolution. Leveraging insights from existing models, the primary aim is to establish an instructional framework that seamlessly integrates a diverse range of content. Notably, the suggested model for training educational managers integrates teaching methodologies, character development, and methodological support for cultivating cultural learning skills, all underpinned by organizational learning processes within the domain of knowledge management. Furthermore, this integrated model incorporates progressive learning objectives that progressively increase in complexity and span the methodologies and resources employed to ensure effective learning outcomes based on comprehensive feature assessment techniques that gauge understanding and competencies.
 Conclusions. This study navigates the landscape of models, culminating in the proposal of an integrated framework that caters to comprehensive aspiring training. This model facilitates the harmonious amalgamation of various subjects, and proficiencies introduce organizational learning processes within the domain of knowledge management. By fostering a multidisciplinary and holistic approach, this model equips future educators with the multifaceted demands of modern primary education while adequately managing knowledge within their organizational contexts.

A Blockchain-Based Trustworthy Model Evaluation Framework for Deep Learning and Its Application in Moving Object Segmentation.

Model evaluation is critical in deep learning. However, the traditional model evaluation approach is susceptible to issues of untrustworthiness, including insecure data and model sharing, insecure model training, incorrect model evaluation, centralized model evaluation, and evaluation results that can be tampered easily. To minimize these untrustworthiness issues, this paper proposes a blockchain-based model evaluation framework. The framework consists of an access control layer, a storage layer, a model training layer, and a model evaluation layer. The access control layer facilitates secure resource sharing. To achieve fine-grained and flexible access control, an attribute-based access control model combining the idea of a role-based access control model is adopted. A smart contract is designed to manage the access control policies stored in the blockchain ledger. The storage layer ensures efficient and secure storage of resources. Resource files are stored in the IPFS, with the encrypted results of their index addresses recorded in the blockchain ledger. Another smart contract is designed to achieve decentralized and efficient management of resource records. The model training layer performs training on users' servers, and, to ensure security, the training data must have records in the blockchain. The model evaluation layer utilizes the recorded data to evaluate the recorded models. A method in the smart contract of the storage layer is designed to enable evaluation, with scores automatically uploaded as a resource attribute. The proposed framework is applied to deep learning-based motion object segmentation, demonstrating its key functionalities. Furthermore, we validated the storage strategy adopted by the framework, and the trustworthiness of the framework is also analyzed.

Identifying SARS-CoV-2 infected cells with scVDN.

Single-cell RNA sequencing (scRNA-seq) is a powerful tool for understanding cellular heterogeneity and identifying cell types in virus-related research. However, direct identification of SARS-CoV-2-infected cells at the single-cell level remains challenging, hindering the understanding of viral pathogenesis and the development of effective treatments. In this study, we propose a deep learning framework, the single-cell virus detection network (scVDN), to predict the infection status of single cells. The scVDN is trained on scRNA-seq data from multiple nasal swab samples obtained from several contributors with varying cell types. To objectively evaluate scVDN's performance, we establish a model evaluation framework suitable for real experimental data. Our results demonstrate that scVDN outperforms four state-of-the-art machine learning models in identifying SARS-CoV-2-infected cells, even with extremely imbalanced labels in real data. Specifically, scVDN achieves a perfect AUC score of 1 in four cell types. Our findings have important implications for advancing virus research and improving public health by enabling the identification of virus-infected cells at the single-cell level, which is critical for diagnosing and treating viral infections. The scVDN framework can be applied to other single-cell virus-related studies, and we make all source code and datasets publicly available on GitHub at https://github.com/studentiz/scvdn.

Evaluation of Academic Atmosphere Formation in Islamic Junior High School Through Digital Learning System

This article aims to evaluate the formation of an academic atmosphere for students in MTsN 1 Kediri City through a digital learning system. This article is written based on field research with a qualitative approach of evaluative research type, using the CIPP model evaluation framework (context, input, process, and product). Data were collected through personal interviews, observation, and documentation. Data is analyzed interactively, including the stages of data reduction, data presentation, and conclusions. Triangulation of credibility, transferability, dependability, and confirmability is carried out to ensure the validity of data obtained from informants. This article concludes: 1) Context: madrasah has a good understanding of the digital learning system. The legality of the program is in accordance with the circular issued by the government. The environment strongly supports digital learning. The purpose of the program is to train students' in critical and innovative thinking. 2) Input: Human resources in the madrasah collaborate well in the implementation of digital learning. Supporting infrastructure facilities have been optimally met. The source of funds comes from the central government and there is internet quota assistance to students. 3) Process: the implementation of learning is carried out completely online using Madrasah E-Learning and other digital applications such as WhatsApp, Google Form, Youtube, and so on. Learning monitoring is carried out every month by the Head of Madrasah and the IT team. Some of the obstacles experienced in the implementation of digital learning include the variety of teachers' ability to utilize IT, fluctuating internet signals, various gadget storage capacities, and students' lack of understanding regarding online learning. 4) Product: the academic atmosphere formed can be seen from a number of indicators. The aspect of the learning atmosphere in class gets a score of 80% (high). The aspect of relations between class citizens received a score of 86% (high). The aspect of teaching and learning activities scored 93% (very high). The discipline aspect of students in the classroom gets a score of 70% (high). The online learning aspect scored 73% (high).

UPRLIMET: UPstream Regional LiDAR Model for Extent of Trout in stream networks

Predicting the edges of species distributions is fundamental for species conservation, ecosystem services, and management decisions. In North America, the location of the upstream limit of fish in forested streams receives special attention, because fish-bearing portions of streams have more protections during forest management activities than fishless portions. We present a novel model development and evaluation framework, wherein we compare 26 models to predict upper distribution limits of trout in streams. The models used machine learning, logistic regression, and a sophisticated nested spatial cross-validation routine to evaluate predictive performance while accounting for spatial autocorrelation. The model resulting in the best predictive performance, termed UPstream Regional LiDAR Model for Extent of Trout (UPRLIMET), is a two-stage model that uses a logistic regression algorithm calibrated to observations of Coastal Cutthroat Trout (Oncorhynchus clarkii clarkii) occurrence and variables representing hydro-topographic characteristics of the landscape. We predict trout presence along reaches throughout a stream network, and include a stopping rule to identify a discrete upper limit point above which all stream reaches are classified as fishless. Although there is no simple explanation for the upper distribution limit identified in UPRLIMET, four factors, including upstream channel length above the point of uppermost fish, drainage area, slope, and elevation, had highest importance. Across our study region of western Oregon, we found that more of the fish-bearing network is on private lands than on state, US Bureau of Land Mangement (BLM), or USDA Forest Service (USFS) lands, highlighting the importance of using spatially consistent maps across a region and working across land ownerships. Our research underscores the value of using occurrence data to develop simple, but powerful, prediction tools to capture complex ecological processes that contribute to distribution limits of species.

Source Relationships and Model Structures Determine Information Flow Paths in Ecohydrologic Models

AbstractIn a complex ecohydrologic system, vegetation and soil variables combine to dictate heat fluxes, and these fluxes may vary depending on the extent to which drivers are linearly or nonlinearly interrelated. From a modeling and causality perspective, uncertainty, sensitivity, and performance measures all relate to how information from different sources “flows” through a model to produce a target, or output. We address how model structure, broadly defined as a mapping from inputs to an output, combines with source dependencies to produce a range of information flow pathways from sources to a target. We apply information decomposition, which partitions reductions in uncertainty into synergistic, redundant, and unique information types, to a range of model cases. Toy models show that model structure and source dependencies both restrict the types of interactions that can arise between sources and targets. Regressions based on weather data illustrate how different model structures vary in their sensitivity to source dependencies, thus affecting predictive and functional performance. Finally, we compare the Surface Flux Equilibrium theory, a land‐surface model, and neural networks in estimating the Bowen ratio and find that models trade off information types particularly when sources have the highest and lowest dependencies. Overall, this study extends an information theory‐based model evaluation framework to incorporate the influence of source dependency on information pathways. This could be applied to explore behavioral ranges for both machine learning and process‐based models, and guide model development by highlighting model deficiencies based on information flow pathways that would not be apparent based on existing measures.

Posterior Summaries of Grocery Retail Topic Models: Evaluation, Interpretability and Credibility

AbstractUnderstanding the shopping motivations behind market baskets has significant commercial value for the grocery retail industry. The analysis of shopping transactions demands techniques that can cope with the volume and dimensionality of grocery transactional data while delivering interpretable outcomes. Latent Dirichlet allocation (LDA) allows processing grocery transactions and the discovering of customer behaviours. Interpretations of topic models typically exploit individual samples overlooking the uncertainty of single topics. Moreover, training LDA multiple times show topics with large uncertainty, that is, topics (dis)appear in some but not all posterior samples, concurring with various authors in the field. In response, we introduce a clustering methodology that post-processes posterior LDA draws to summarise topic distributions represented as recurrent topics. Our approach identifies clusters of topics that belong to different samples and provides associated measures of uncertainty for each group. Our proposed methodology allows the identification of an unconstrained number of customer behaviours presented as recurrent topics. We also establish a more holistic framework for model evaluation, which assesses topic models based not only on their predictive likelihood but also on quality aspects such as coherence and distinctiveness of single topics and credibility of a set of topics. Using the outcomes of a tailored survey, we set thresholds that aid in interpreting quality aspects in grocery retail data. We demonstrate that selecting recurrent topics not only improves predictive likelihood but also outperforms interpretability and credibility. We illustrate our methods with an example from a large British supermarket chain.

Toward a Next Generation Particle Precipitation Model: Mesoscale Prediction Through Machine Learning (a Case Study and Framework for Progress)

AbstractWe advance the modeling capability of electron particle precipitation from the magnetosphere to the ionosphere through a new database and use of machine learning (ML) tools to gain utility from those data. We have compiled, curated, analyzed, and made available a new and more capable database of particle precipitation data that includes 51 satellite years of Defense Meteorological Satellite Program (DMSP) observations temporally aligned with solar wind and geomagnetic activity data. The new total electron energy flux particle precipitation nowcast model, a neural network called PrecipNet, takes advantage of increased expressive power afforded by ML approaches to appropriately utilize diverse information from the solar wind and geomagnetic activity and, importantly, their time histories. With a more capable representation of the organizing parameters and the target electron energy flux observations, PrecipNet achieves a >50% reduction in errors from a current state‐of‐the‐art model oval variation, assessment, tracking, intensity, and online nowcasting (OVATION Prime), better captures the dynamic changes of the auroral flux, and provides evidence that it can capably reconstruct mesoscale phenomena. We create and apply a new framework for space weather model evaluation that culminates previous guidance from across the solar‐terrestrial research community. The research approach and results are representative of the “new frontier” of space weather research at the intersection of traditional and data science‐driven discovery and provides a foundation for future efforts.

Semantic and knowledge based support to business model evaluation to stimulate green behaviour of electric vehicles’ drivers and energy prosumers

This paper proposes a semantic framework for Business Model evaluation and its application to a real case study in the context of smart energy and sustainable mobility. It presents an ontology based representation of an original business model and examples of inferential rules for knowledge extraction and automatic population of the ontology. The real case study belongs to the GreenCharge European Project, that in these last years is proposing some original business models to promote sustainable e-mobility plans. An original OWL Ontology contains all relevant Business Model concepts referring to GreenCharge’s domain, including a semantic description of TestCards, survey results and inferential rules.

Reliability assessment of stochastic networks with ER connectivity and ER dependency

In stochastic networks, nodes usually function dependently and interact with other nodes through connectivity links or dependency links. In this paper, the model for stochastic networks considering sub-networks with connectivity and dependency links of Erdös-Rényi (ER) topology is proposed, which is defined as networks with arbitrary pair of nodes randomly connected/depended by a constant probability. The reliability evaluation framework for the proposed networks is developed, where both of the extended multi-valued decision diagram (MDD) method and Monte Carlo simulation (MCS) are involved. The MDD method is proposed to assess the reliability of deterministic stochastic networks with ER connectivity and dependency, where arbitrary time to failure distributions of nodes are allowed. Based on the reliability evaluation for a stochastic network with a deterministic structure, the MCS is employed to achieve the reliability analysis of corresponding stochastic networks. Numerical examples are presented to demonstrate the proposed stochastic network model and reliability evaluation framework, where the probability distributions for the reliability of stochastic networks are provided.

An Ensemble Learning Framework for Model Fitting and Evaluation in Inverse Linear Optimization

We develop a generalized inverse optimization framework for fitting the cost vector of a single linear optimization problem given multiple observed decisions. This setting is motivated by ensemble learning, where building consensus from base learners can yield better predictions. We unify several models in the inverse optimization literature under a single framework and derive assumption-free and exact solution methods for each one. We extend a goodness-of-fit metric previously introduced for the problem with a single observed decision to this new setting and demonstrate several important properties. Finally, we demonstrate our framework in a novel inverse optimization-driven procedure for automated radiation therapy treatment planning. Here, the inverse optimization model leverages an ensemble of dose predictions from different machine learning models to construct a consensus treatment plan that outperforms baseline methods. The consensus plan yields better trade-offs between the competing clinical criteria used for plan evaluation.

A model evaluation framework applied to the Forest Vegetation Simulator (FVS) in Colorado and Wyoming lodgepole pine forests

The Forest Vegetation Simulator (FVS) growth and yield model is widely used throughout the United States, but recent studies have reported unexpectedly large bias for some regional model variants. Here we propose a general framework for model evaluation, designed to highlight model strengths and weaknesses and inform calibration efforts. We apply the framework to evaluate the Lodgepole Pine (LP) model of the FVS Central Rockies Variant (FVS-CR), which has rarely been evaluated in the literature despite its widespread use in the western US. We started with a qualitative verification of the structure and logic of the FVS-CR LP model against a modified Bakuzis matrix, determining that it adequately reproduces known patterns of stand dynamics. We then compared stand-level growth simulations to a chronosequence developed from 554 Forest Inventory and Analysis (FIA) plots measured in Colorado and Wyoming lodgepole pine forests. This quantitative validation exercise revealed that the default model settings substantially over-predict basal area and total stand carbon after 50 years for both pure lodgepole pine stands and mixed lodgepole pine stands containing a minor proportion of Engelmann spruce, subalpine fir, and quaking aspen. Using equivalence testing to validate the large-tree diameter increment model against a separate dataset of 3,790 remeasured trees on 124 lodgepole pine FIA plots, we found that the default FVS–CR LP model adequately predicts lodgepole diameter growth, though we observed large variation in model errors. Equivalence tests also revealed systematic under-estimation of spruce–fir and over-estimation of aspen productivity. Finally, we conducted a sensitivity analysis to identify the most important model parameters and data inputs driving simulated stand structure and carbon accumulation in both the short- (50 years) and long-term (200 years). The model shows the greatest sensitivity to initial species composition; a small proportion of aspen, spruce, or fir seedlings led to long-term stand re-structuring and greatly increased carbon accumulation. Other sensitive parameters included site index, the fixed large-tree diameter increment adjustment factor, and the parameter controlling the point at which density-dependent self-thinning begins (maximum stand density index). This evaluation leads us to conclude that the FVS-CR LP model tends to under-estimate tree mortality (particularly in young stands) and the default site index represents relatively high productivity stands, such that re-calibration may often be necessary to capture realistic long-term stand behaviors. Our sensitivity analysis provides guidance for future efforts to re-calibrate the FVS-CR LP model and highlights the importance of collecting site index and seedling species composition data wherever possible to produce the most realistic simulations.

Towards successful business model management with analytic network process-based feasibility evaluation and portfolio management

Despite the great importance of designing innovative business models, a validated methodology that helps related process is still not available. To fill this gap, this research introduces business model management concepts that consider business models as individual entities that can be systematically developed and managed by utilizing science based methodology. This study mainly focuses on proposing business model evaluation and deployment framework by applying analytic network process (ANP), a multi multiple criteria decision-making method and portfolio matrix strategy. The proposed evaluation method is designed to consider both internal and external perspectives where internal perspective is measured through ANP method by employees, while external perspective is measured through customer survey by in-depth interview. The obtained values from ANP and customer survey are then used to construct the business model portfolio matrix for continuous business model management. To this end, this study identifies 12 criteria within three different categories employed in ANP analysis and business model portfolio matrix, respectively. The proposed framework for managing business models provides new insights to both practitioners and academics. It regards business models as individual entities and introduces a way to practically apply the business model concept. It also provides managers with guidelines to manage business model alternatives in terms of evaluation, selection, and continuous management. Finally, because this study is, to the best of our knowledge, one of the first attempts to employ the portfolio management concept in business model research, it can function as the foundation for future applied business model research.

Toward a Better Performance Evaluation Framework for Fake News Classification

The rising prevalence of fake news and its alarming downstream impact have motivated both the industry and academia to build a substantial number of fake news classification models, each with its unique architecture. Yet, the research community currently lacks a comprehensive model evaluation framework that can provide multifaceted comparisons between these models beyond the simple evaluation metrics such as accuracy or f1 scores. In our work, we examine a representative subset of classifiers using a very simple set of performance evaluation and error analysis steps. We demonstrate that model performance varies considerably based on i) dataset, ii) evaluation archetype, and iii) performance metrics. Additionally, classifiers also demonstrate a potential bias against small and conservative-leaning credible news sites. Finally, models' performance varies based on external events and article topics. In sum, our results highlight the need to move toward systematic benchmarking.

Modelling near-real-time order arrival demand in e-commerce context: a machine learning predictive methodology

PurposeAccurate prediction of order demand across omni-channel supply chains improves the management's decision-making ability at strategic, tactical and operational levels. The paper aims to develop a predictive methodology for forecasting near-real-time e-commerce order arrivals in distribution centres, allowing third-party logistics service providers to manage the hour-to-hour fast-changing arrival rates of e-commerce orders better.Design/methodology/approachThe paper proposes a novel machine learning predictive methodology through the integration of the time series data characteristics into the development of an adaptive neuro-fuzzy inference system. A four-stage implementation framework is developed for enabling practitioners to apply the proposed model.FindingsA structured model evaluation framework is constructed for cross-validation of model performance. With the aid of an illustrative case study, forecasting evaluation reveals a high level of accuracy of the proposed machine learning approach in forecasting the arrivals of real e-commerce orders in three different retailers at three-hour intervals.Research limitations/implicationsResults from the case study suggest that real-time prediction of individual retailer's e-order arrival is crucial in order to maximize the value of e-order arrival prediction for daily operational decision-making.Originality/valueEarlier researchers examined supply chain demand, forecasting problem in a broader scope, particularly in dealing with the bullwhip effect. Prediction of real-time, hourly based order arrivals has been lacking. The paper fills this research gap by presenting a novel data-driven predictive methodology.