Model Selection Process Research Articles

Design of an intelligent data analysis platform for pharmaceutical forecasts

This study considers the task to design a data analysis platform with predictive capabilities of neural networks. The object of research is intelligent decision-making systems built on deep learning methods. The proposed intelligent platform takes into account the specificity of working with data in the dynamic and uncertain environment of the pharmaceutical market. Its main purpose is the processing of various data formats, such as time series, regression, classification data sets to create forecasts based on various indicators. At the core of the platform architecture, along with technologies for backend and frontend development (HTML, JS, CSS, C#, .NET), MSSQL Server and TSQL for database management, are AI Microservices (Python, Flask); they are responsible for artificial intelligence services, in particular neural networks. In order to identify the optimal model, which is able to effectively solve regression problems based on the selected indicators, the study analyzed several configurations of neural networks on End-to-end machine learning platforms. Distinctive features of the architecture of the designed data analysis platform include its ability to dynamically switch between different machine learning models based on predefined indicators and criteria such as prediction accuracy and model selection. Improved interpretation of forecasts through the use of Explainable AI enables effective decision-making in the pharmaceutical industry. The functioning of the proposed instrumental decision-making base is demonstrated on the examples of forecasting trends in the consumption of pharmaceuticals by different groups in the pharmaceutical markets of different countries. Automating the model selection and prediction loss minimization process in a comprehensive data analysis platform (CDAP) improves forecast accuracy by approximately 15 % compared to traditional manually selected models.

Multiobjective design of automated order fulfillment systems: A case study for an existing surgical complex

We present a case study that introduces a systematic framework for designing an automated guided vehicle (AGV) order fulfillment system. The multi-stage framework integrated discrete event simulation (DES), super-efficiency data envelopment analysis (S-DEA) model, and Design of Experiments (DOE) methodologies for the design of an AGV system in an existing large-scale surgical complex within a comprehensive hospital in Singapore. Performance metrics derived from this model were consolidated into a single relative efficiency score using the S-DEA model. This efficiency score was then utilized in a DOE methodology, combined with a post-hoc model selection process, to robustly identify appropriate models for evaluating alternative system designs. The proposed framework optimizes AGV deployment by first identifying opportunities for AGVs to add value within the surgical system through a value-stream mapping approach. It then derives the optimal system design by balancing cost efficiency, manpower reallocation, and order fulfillment rates, facilitated by the S-DEA model augmented with the DOE approach and post-hoc model selection. Application of this framework to the surgical complex revealed several insights that can significantly enhance AGV efficiency prior to actual deployment. However, a key limitation of this case-study-based approach is its limited generalizability across different systems. Despite this, the methodology provides a robust foundation for AGV system design in similar healthcare environments.

Three-Dimensional Outdoor Object Detection in Quadrupedal Robots for Surveillance Navigations

Quadrupedal robots are confronted with the intricate challenge of navigating dynamic environments fraught with diverse and unpredictable scenarios. Effectively identifying and responding to obstacles is paramount for ensuring safe and reliable navigation. This paper introduces a pioneering method for 3D object detection, termed viewpoint feature histograms, which leverages the established paradigm of 2D detection in projection. By translating 2D bounding boxes into 3D object proposals, this approach not only enables the reuse of existing 2D detectors but also significantly increases the performance with less computation required, allowing for real-time detection. Our method is versatile, targeting both bird’s eye view objects (e.g., cars) and frontal view objects (e.g., pedestrians), accommodating various types of 2D object detectors. We showcase the efficacy of our approach through the integration of YOLO3D, utilizing LiDAR point clouds on the KITTI dataset, to achieve real-time efficiency aligned with the demands of autonomous vehicle navigation. Our model selection process, tailored to the specific needs of quadrupedal robots, emphasizes considerations such as model complexity, inference speed, and customization flexibility, achieving an accuracy of up to 99.93%. This research represents a significant advancement in enabling quadrupedal robots to navigate complex and dynamic environments with heightened precision and safety.

Exploring the power of data-driven models for groundwater system conceptualization: a case study of the Grazer Feld Aquifer, Austria

AbstractDeveloping a reliable conceptual model is crucial for analyzing groundwater systems. An essential part of the aquifer conceptualization is the identification of the hydrological stresses that control the hydraulic head fluctuations. By effectively capturing and understanding these stresses, the propagation of potential errors and uncertainties through subsequent modeling steps can be minimized. This study aims to test data-driven models as screening models for conceptualizing a groundwater system. The case study is applied to the Grazer Feld Aquifer in southeast Austria. Time series models are applied to: (1) identify the stresses likely influencing the observed head fluctuations and their spatial variability; (2) identify locations where a lack of understanding of head fluctuations exists; and (3) discuss the limitations and opportunities associated with data-driven models to support system conceptualization. Time series models were created for 144 monitoring wells where sufficient head observations were available during the calibration period (2005–2015). A total of 576 models were developed, incorporating the combinations of stresses: recharge, river level, and a step trend. Following the model selection process, each model was categorized based on its performance and divided into four groups. At 88 sites, recharge and river level variations were identified as the primary controlling stresses influencing head fluctuations. The inclusion of the step trend was found to be necessary at five sites to accurately simulate heads due to dam construction. The application of data-driven models in this study enhanced the identification of key aquifer stresses, facilitating a more informed understanding of the groundwater system.

Diagnosis of ophthalmologic diseases in canines based on images using neural networks for image segmentation

The primary challenge in diagnosing ocular diseases in canines based on images lies in developing an accurate and reliable machine learning method capable of effectively segmenting and diagnosing these conditions through image analysis. Addressing this challenge, the study focuses on developing and rigorously evaluating a machine learning model for diagnosing ocular diseases in canines, employing the U-Net neural network architecture as a foundational element of this investigation.Through this extensive evaluation, the authors identified a model that exhibited good reliability, achieving prediction scores with an Intersection over Union (IoU) exceeding 80 %, as measured by the Jaccard index. The research methodology encompassed a systematic exploration of various neural network backbones (VGG, ResNet, Inception, EfficientNet) and the U-Net model, combined with an extensive model selection process and an in-depth analysis of a custom training dataset consisting of historical images of different medical symptoms and diseases in dog eyes.The results indicate a fairly high degree of accuracy in the segmentation and diagnosis of ocular diseases in canines, demonstrating the model's effectiveness in real-world applications. In conclusion, this potentially makes a significant contribution to the field by utilizing advanced machine-learning techniques to develop image-based diagnostic routines in veterinary ophthalmology. This model's successful development and validation offer a promising new tool for veterinarians and pet owners, enhancing early disease detection and improving health outcomes for canine patients.

Modeling regional occupancy of fishes using acoustic telemetry: a model comparison framework applied to lake trout

Acoustic telemetry is a common tool used in fisheries management to estimate fish space use (i.e., occupancy) from a local habitat scale to entire systems. Numerous analytical models have been developed to estimate different aspects of fish movement from telemetry datasets, yet evaluations of model performance and comparisons among models are limited. Here, we develop a framework to evaluate model estimates of regional occupancy in large and fragmented systems using an acoustic receiver array in Lake Champlain. We simulated the tracks of 100 acoustically tagged fish using a random walk function and created detection events based on receiver positions and distance-based detection probability. Regional occupancy for the simulated data was estimated by six movement models that ranged in analytical complexity, and results were compared to the true distributions for each simulated track to evaluate model error. The six movement models included: (1) a basic residency index using detections alone; (2) a residency index using last-observation-carried-forward; (3) a centers of activity model; (4) linear and non-linear interpolations (i.e., least-cost paths); and (5 and 6) two dynamic Brownian bridge movement models generated using separate packages in R. We developed a model selection process to compare model performance and select the optimal analysis based on simulation error. This process showed significant differences in model performance among the six movement models based on model error. Overall, the model generating least-cost paths using linear and non-linear interpolations consistently provided the most accurate regional occupancy estimates. Based on these simulation results, we applied this model to a case study that evaluated patterns in the regional distribution of stocked lake trout (Salvelinus namaycush) in Lake Champlain, which demonstrated distinct regional occupancy of two stocked lake trout groups. These results demonstrate potential for large variability in interpretation of acoustic telemetry data for describing regional fish distribution dependent on the analytical method used.

A novel method to select time-varying multivariate time series models for the surveillance of infectious diseases

BackgroundDescribing the transmission dynamics of infectious diseases across different regions is crucial for effective disease surveillance. The multivariate time series (MTS) model has been widely adopted for constructing cross-regional infectious disease transmission networks due to its strengths in interpretability and predictive performance. Nevertheless, the assumption of constant parameters frequently disregards the dynamic shifts in disease transmission rates, thereby compromising the accuracy of early warnings. This study investigated the applicability of time-varying MTS models in multi-regional infectious disease monitoring and explored strategies for model selection.MethodsThis study focused on two prominent time-varying MTS models: the time-varying parameter-stochastic volatility-vector autoregression (TVP-SV-VAR) model and the time-varying VAR model using the generalized additive framework (tvvarGAM), and intended to explore and verify their applicable conditions for the surveillance of infectious diseases. For the first time, this study proposed the time delay coefficient and spatial sparsity indicators for model selection. These indicators quantify the temporal lags and spatial distribution of infectious disease data, respectively. Simulation study adopted from real-world infectious disease surveillance was carried out to compare model performances under various scenarios of spatio-temporal variation as well as random volatility. Meanwhile, we illustrated how the modelling process could help the surveillance of infectious diseases with an application to the influenza-like case in Sichuan Province, China.ResultsWhen the spatio-temporal variation was small (time delay coefficient: 0.1–0.2, spatial sparsity:0.1–0.3), the TVP-SV-VAR model was superior with smaller fitting residuals and standard errors of parameter estimation than those of the tvvarGAM model. In contrast, the tvvarGAM model was preferable when the spatio-temporal variation increased (time delay coefficient: 0.2–0.3, spatial sparsity: 0.6–0.9).ConclusionThis study emphasized the importance of considering spatio-temporal variations when selecting appropriate models for infectious disease surveillance. By incorporating our novel indicators—the time delay coefficient and spatial sparsity—into the model selection process, the study could enhance the accuracy and effectiveness of infectious disease monitoring efforts. This approach was not only valuable in the context of this study, but also has broader implications for improving time-varying MTS analyses in various applications.

Application of deep learning methods for beam size control during user operation at the Advanced Light Source

Past research at the Advanced Light Source (ALS) provided a proof-of-principle demonstration that deep learning methods could be effectively employed to compensate for the significant perturbations to the transverse electron beam size induced by user-controlled adjustments of the insertion devices. However, incorporating these methods into the ALS’ daily operations has faced notable challenges. The complexity of the system’s operational requirements and the significant upkeep demands has restricted their sustained application during user operation. Here, we introduce the development of a more robust neural network (NN)-based algorithm that utilizes a novel online fine-tuning approach and its systematic integration into the day-to-day machine operations. Our analysis emphasizes the process of NN model selection, demonstrates the superior performance of the NN-based method over traditional feedback methods, and examines the effectiveness and resilience of the new algorithm during user-operation scenarios. Published by the American Physical Society 2024

Machine Learning for Predictive Maintenance to Enhance Energy Efficiency in Industrial Operations

In the realm of industrial operations, optimizing energy usage is critical for both economic and environmental sustainability. Traditional approaches to maintenance often rely on fixed schedules or reactive responses to equipment failures, leading to inefficiencies and higher energy consumption. Predictive maintenance (PdM) offers a proactive solution by leveraging machine learning algorithms to predict equipment failures before they occur. This approach not only reduces downtime but also facilitates energy-efficient practices by enabling timely interventions and optimized operational strategies. This study explores the application of machine learning techniques for predictive maintenance in a manufacturing setting. Historical operational data, including equipment performance metrics and environmental conditions, are collected and preprocessed. Various machine learning models, such as support vector machines (SVM), random forests, and neural networks, are trained on this dataset to predict equipment failures and maintenance needs. Feature engineering and model selection processes are detailed to highlight the steps taken to enhance prediction accuracy and reliability. Through rigorous experimentation and validation, our approach demonstrates significant improvements in energy efficiency within industrial operations. By predicting maintenance needs in advance, downtime is minimized, and energy-intensive emergency repairs are avoided. Furthermore, the implementation of optimized maintenance schedules and operational strategies based on machine learning predictions leads to substantial reductions in overall energy consumption. Case studies and quantitative analyses underscore the efficacy of our methodology in enhancing both operational efficiency and energy sustainability in industrial settings.

Bootstrap Approximation of Model Selection Probabilities for Multimodel Inference Frameworks.

Most statistical modeling applications involve the consideration of a candidate collection of models based on various sets of explanatory variables. The candidate models may also differ in terms of the structural formulations for the systematic component and the posited probability distributions for the random component. A common practice is to use an information criterion to select a model from the collection that provides an optimal balance between fidelity to the data and parsimony. The analyst then typically proceeds as if the chosen model was the only model ever considered. However, such a practice fails to account for the variability inherent in the model selection process, which can lead to inappropriate inferential results and conclusions. In recent years, inferential methods have been proposed for multimodel frameworks that attempt to provide an appropriate accounting of modeling uncertainty. In the frequentist paradigm, such methods should ideally involve model selection probabilities, i.e., the relative frequencies of selection for each candidate model based on repeated sampling. Model selection probabilities can be conveniently approximated through bootstrapping. When the Akaike information criterion is employed, Akaike weights are also commonly used as a surrogate for selection probabilities. In this work, we show that the conventional bootstrap approach for approximating model selection probabilities is impacted by bias. We propose a simple correction to adjust for this bias. We also argue that Akaike weights do not provide adequate approximations for selection probabilities, although they do provide a crude gauge of model plausibility.

Navigating the confluence of econometrics and data science: Implications for economic analysis and policy

This paper explores the transformative integration of econometrics and data science, a synergy poised to redefine empirical research within economics. By merging traditional econometric methods with advanced data science techniques, such as machine learning algorithms and big data analytics, this interdisciplinary approach enables a deeper, more nuanced understanding of complex economic phenomena. We delve into the theoretical foundations underlying this integration, highlighting how machine learning algorithms like random forests and neural networks complement conventional regression analysis, thereby enhancing model complexity and predictive accuracy. The paper further discusses methodological advancements, including handling high-dimensional data, incorporating unstructured data through natural language processing, and the evolution of model selection processes empowered by machine learning. Practical applications are thoroughly examined across three pivotal areas: economic forecasting and policy analysis, financial markets and risk management, and social economic analysis and public policy, showcasing the significant contributions of this convergence to economic forecasting, policy formulation, and the assessment of public interventions. This comprehensive exploration underscores the potential of combining econometrics and data science to offer more precise and actionable insights for policymakers, researchers, and practitioners in the field of economics.

Subjectivity in Unsupervised Machine Learning Model Selection

Model selection is a necessary step in unsupervised machine learning. Despite numerous criteria and metrics, model selection remains subjective. A high degree of subjectivity may lead to questions about repeatability and reproducibility of various machine learning studies and doubts about the robustness of models deployed in the real world. Yet, the impact of modelers' preferences on model selection outcomes remains largely unexplored. This study uses the Hidden Markov Model as an example to investigate the subjectivity involved in model selection. We asked 33 participants and three Large Language Models (LLMs) to make model selections in three scenarios. Results revealed variability and inconsistencies in both the participants’ and the LLMs' choices, especially when different criteria and metrics disagree. Sources of subjectivity include varying opinions on the importance of different criteria and metrics, differing views on how parsimonious a model should be, and how the size of a dataset should influence model selection. The results underscore the importance of developing a more standardized way to document subjective choices made in model selection processes.

MLCompare: Platform to Compare Machine Learning Models

Abstract: The field of machine learning (ML) continues to advance, the demand for efficient and effective methods for model development and selection becomes increasingly pressing. Automated Machine Learning (AutoML) platforms have emerged as powerful tools to address this need by automating the process of model selection, hyperparameter tuning, and feature engineering. By integrating a wide array of ML techniques and leveraging cloud computing resources, the platform enables researchers and practitioners to efficiently explore and compare the performance of various algorithms across diverse datasets. Key features include a user-friendly interface, a robust evaluation framework encompassing traditional metrics and advanced techniques, and parallel processing capabilities. Through experiments on benchmark datasets from multiple domains, the platform demonstrates its ability to streamline algorithm selection and provide valuable insights into algorithm performance. This contribution advances the field of AutoML by facilitating informed decision-making and accelerating innovation in ML applications.

Empirical analysis of decision recommendation models for various processes from a pragmatic perspective

Decision recommendation models allow researchers and process designers to identify and implement high-efficiency processes in ambiguous situations. These models perform multipara metric analysis on the given process sets to recommend high-quality decisions that assist in improving process-based efficiency levels. A wide variety of models have been proposed by researchers for the implementation of such recommenders, and each of them varies in terms of their functional nuances, applicative advantages, internal operating characteristics, contextual limitations, and deployment-specific future scopes. Thus, it is difficult for researchers and process designers to identify optimal models for functionality-specific use. Therefore, they tend to validate multiple process models, which increase deployment time, cost and complexity levels. To overcome this ambiguity, a detailed survey of different decision process recommendation models is discussed in this text. Fuzzy logic, analytical hierarchical processing (AHP), the technique for order performance by similarity to ideal solution (TOPSIS), and their variants are highly useful for the recommendation of efficient decisions. Based on this survey, readers will be able to identify recently proposed decision recommendation models and functionality-specific models for their deployments. To further assist in the model selection process, this paper compares the reviewed models in terms of their computational complexity, recommendation efficiency, delay needed for recommendation, scalability and contextual accuracy. Based on this comparison, readers will be able to identify performance-specific models for their deployments. This paper also evaluates a novel decision recommendation rank metric (DRRM), which combines these parameters, to identify models that can optimally perform w.r.t. multiple process metrics. Referring to this parameter comparison, readers will be able to identify optimal recommendation models for enhancing the performance of their decision recommendations under real-time scenarios.

Determinants of health as predictors for differential antibody responses following SARS-CoV-2 primary and booster vaccination in an at-risk, longitudinal cohort.

Post vaccine immunity following COVID-19 mRNA vaccination may be driven by extrinsic, or controllable and intrinsic, or inherent health factors. Thus, we investigated the effects of extrinsic and intrinsic on the peak antibody response following COVID-19 primary vaccination and on the trajectory of peak antibody magnitude and durability over time. Participants in a longitudinal cohort attended visits every 3 months for up to 2 years following enrollment. At baseline, participants provided information on their demographics, recreational behaviors, and comorbid health conditions which guided our model selection process. Blood samples were collected for serum processing and spike antibody testing at each visit. Cross-sectional and longitudinal models (linear-mixed effects models) were generated to assess the relationship between selected intrinsic and extrinsic health factors on peak antibody following vaccination and to determine the influence of these predictors on antibody over time. Following cross-sectional analysis, we observed higher peak antibody titers after primary vaccination in females, those who reported recreational drug use, younger age, and prior COVID-19 history. Following booster vaccination, females and Hispanics had higher peak titers after the 3rd and 4th doses, respectively. Longitudinal models demonstrated that Moderna mRNA-1273 recipients, females, and those previously vaccinated had increased peak titers over time. Moreover, drug users and half-dose Moderna mRNA-1273 recipients had higher peak antibody titers over time following the first booster, while no predictive factors significantly affected post-second booster antibody responses. Overall, both intrinsic and extrinsic health factors play a significant role in shaping humoral immunogenicity after initial vaccination and the first booster. The absence of predictive factors for second booster immunogenicity suggests a more robust and consistent immune response after the second booster vaccine administration.

Experimental and computational analysis of elastomer membranes used in oscillating water column WECs

The study investigates the structural characterisation of flexible membranes used in oscillating water column (OWC) wave energy converters (WECs). Four commonly utilised elastomers – natural rubber, nitrile rubber, silicone, and latex – were subjected to a novel hyperelastic model selection process. A custom bulge test setup enabled the selection of second-order Mooney-Rivlin (SOMR) and Yeoh models for relevant accuracy (RMSE<0.018 MPa), stability and numerical validation. A 1:20 scale OWC model with latex was tested in a water tank to examine the effects of waves with a frequency range of 0.25–1.4 Hz and up to 0.24m amplitude. Water tank experiments demonstrated smooth frequency responses for OWC with membrane, beneficial for consistent power generation. A dry test rig was designed and built to replicate OWC inflation conditions and apply cyclic loadings up to 1.5 Hz, overcoming pressure limitations of the water tank, exploring wider material options, and validating numerical simulation. An optical motion capture system, Qualisys, supported the validation process by providing precise data on membrane deformation during experiments. Furthermore, finite element analysis (FEA) was utilised to conduct stress analysis and parametric studies, assessing the suitability of these materials for flexible OWC application.

A comprehensive study to estimate income and price elasticities of household electricity consumption using Auto-metrics

This study focuses on understanding family electricity consumption behaviors in response to income and price changes from 1994 to 2022 across 12 prominent European countries. We employ a unique econometric approach, Auto-selection Models, to analyze the nuances of energy demand elasticity. Our methodology includes the use of saturation techniques, which are highly effective in identifying anomalies and discontinuities in the data, ensuring the reliability of our results. The Auto-metrics method streamlines the model selection process and enhances the accuracy of elasticity predictions. We use Error Correction Models (ECMs) for each country to examine the long-term equilibrium relationships among key variables such as energy consumption, household income, electricity prices, and weather patterns, taking into account any observed anomalies and significant structural changes. The findings reveal varying levels of income and price elasticity across the countries, reflecting their unique economic and climatic conditions. The study's results hold significant implications for policymaking. By recognizing and adapting to the varied characteristics of electricity demand elasticity, energy policies can be more accurately tailored at both the national and European Union levels.

Predicting Construction Company Insolvent Failure: A Scientometric Analysis and Qualitative Review of Research Trends

The construction industry is infamous for its high insolvent failure rate because construction projects require complex processes, heavy investment, and long durations. However, there is a lack of a comprehensive framework and a requirement for such a framework in predicting the financial distress of construction firms. This paper reviews relevant literature to summarize the existing knowledge, identify current problems, and point out future research directions needed in this area using a scientometric analysis approach. Based on a total of 93 journal articles relating to predicting construction company failure extracted from multiple databases, this study conducts a holistic review in terms of chronological trends, journal sources, active researchers, frequent keywords, and most cited documents. Qualitative analysis is also provided to explore the data collection and processing procedures, model selection and development process, and detailed performance evaluation metrics. Four research gaps and future directions for predicting construction company failure are presented: selecting a broader data sample, incorporating more heterogeneous variables, balancing model predictability and interpretability, and quantifying the causality and intercorrelation of variables. This study provides a big picture of existing research on predicting construction company insolvent failure and presents outcomes that can help researchers to comprehend relevant literature, directing research policy-makers and editorial boards to adopt the promising themes for further research and development.

Are the Bayesian Information Criterion (BIC) and the Akaike Information Criterion (AIC) Applicable in Determining the Optimal Fit and Simplicity of Mechanistic Models?

ABSTRACT Over the past three decades, the discourse on the mechanistic approach to scientific modelling and explanation has notably sidestepped the topic of simplicity and fit within the process of model selection. This paper aims to rectify this disconnect by delving into the topic of simplicity and fit within the context of mechanistic explanations. More precisely, our primary objective is to address whether simplicity metrics hold any significance within mechanistic explanations. If they do, then our inquiry extends to the suitability of the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), and related criteria in determining the optimal balance of fit and simplicity of mechanistic models. As our main claims, we argue that mechanistic models inherently lend themselves to considerations of simplicity, and that the AIC and BIC and related criteria are applicable to some submodels of certain kinds of mechanistic models. However, these criteria and related criteria designed for curve fitting and causal modelling are of little help for a comparative assessment of full mechanistic models, and a fundamentally different approach is needed to make determinations of this kind.

A Review of Contact Models’ Properties for Discrete Element Simulation in Agricultural Engineering

In agricultural engineering, the discrete element simulation of the operational structure, object of movement, and force has become a standard method of modern agricultural equipment design. The selection and development of an appropriate contact model are critical factors affecting the accuracy of the process of the simulation calculation of the movement and force. Understanding how to choose or establish suitable contact models according to different research fields, objects, and purposes has become the focus of present research. This paper gives an overview of contact models for discrete element simulation, summarizes and analyzes the simulation calculation basis of different contact models, and focuses on the application status and scenarios of different models at this stage. It analyzes and summarizes the selection basis and application fields of contact models. The next direction in the development of discrete element simulation contact models should be the hybrid application of multicontact models and the precise development of specialized contact models. It is necessary to establish a standardized parameter-calibration process for different contact models to guarantee the accuracy of the models, to improve the application of computer arithmetic, and to establish an efficient and accurate simulation contact model selection and application in the field of agricultural engineering. Efficient and accurate simulation contact model selection, design theory, and calculation processes will improve the efficiency of modern agricultural machinery design.