Data-driven Evaluation Research Articles

Utilizing data analysis to identify the best players from the 2019 draft

This study seeks further standardization of the evaluation process of these players while recognizing that it is indeed inherently quite complex and subjective. Acknowledging the different skill sets in professional basketball, this research stresses the need for specialized approaches in evaluating players fully. Diversity and balance are what make a player fit as it assumes those who perform well in multiple positions, meaning they have both offensive and defensive skills are likely to be fitter. This research adopted a quantitative approach to analyze KPIs (games played, minutes per game, adjusted shooting percentage, rebounds, total points) in assessing player performance and return on investment. There was a very high correlation between minutes played and points scored from this analysis at a very high level of significance showing that one person can play many roles being versatile is key. The 2019 mock draft ranks players based on a combined scoring system, finding the highest value of players like Zion Williamson and Ja Morant. Research can improve draft strategies and objectives, and data-driven evaluation methods can boost player development and team success. It can help teams to find suitable players based on team playing styles.

Does one size fit all? Developing an evaluation strategy to assess large language models for patient safety event report analysis.

Collecting and analyzing patient safety event (PSE) reports is a key component to the improvement of patient safety yet report analysis has been challenging. Large language models (LLMs) may support analysis; however, PSE reports tend to be a hybrid of clinical and general language. We propose a data-driven evaluation strategy to assess LLM fit for report analysis. We identify target tokens and sentences from PSE reports and use perplexity to evaluate four LLMs comprehension of the target sentence. LLMs had statistically significantly different perplexity measures in six of seven event categories. Clinical models perform better with clinical narratives, often reported by nurses and physicians. General models perform better with colloquial language and communication themes. For LLMs to support PSE report analysis there must be a good fit between the language model and the nature of the text in reports. A single LLM approach may not be the most useful strategy.

A multi-criteria decision analysis framework for evaluating deep learning models in healthcare research

Selecting the appropriate deep learning (DL) model for healthcare research poses a significant challenge due to the diversity of evaluation criteria and the complex nature of health-related tasks, where a single metric like accuracy is often insufficient. Motivated by the need for a structured, multi-criteria approach, this study proposes a Multi-Criteria Decision Analysis (MCDA) framework using the Analytic Hierarchy Process (AHP). Our primary contribution is the development of a comprehensive decision-making framework that integrates multiple evaluation criteria, such as accuracy, sensitivity, specificity, and computational complexity, alongside empirical data from existing literature to systematically compare DL models. The framework was validated through a use case involving the selection of the best DL model for diagnosing COVID-19 using X-ray images, where we compared eight popular models, including ResNet34, SqueezeNet, and AlexNet, and it was also evaluated through comparative scenarios using traditional methods, including weighted sum, weighted average, and accuracy-based evaluation. Quantitative results show that SqueezeNet achieved the highest score in the AHP framework (88.64), while ResNet34 performed best in traditional methods such as weighted sum (588.49) and accuracy ranking (98.33%). A sensitivity analysis further demonstrated the impact of varying criteria weights, showing how changes in the importance of accuracy and precision, influenced model ranking. These findings highlight the flexibility and robustness of the AHP framework in addressing the complexities of model selection in healthcare research. The implications of this work suggest that a structured, data-driven evaluation approach can provide more nuanced and reliable insights compared to traditional methods like single-metric evaluations, ultimately supporting more informed decision-making in healthcare applications.

The Effectiveness of National Innovation and Startup Policy (NISP) in Cultivating Entrepreneurial Ecosystems within Management Institutions in Pune

This study investigates the effectiveness of the National Innovation and Startup Policy (NISP) in cultivating entrepreneurial ecosystems within management institutions in Pune. A mixed-method approach involving surveys and interviews with students, faculty, and NISP coordinators across 15 HEIs was employed. The findings reveal that NISP has fostered a supportive environment by promoting awareness, offering mentorship, and establishing infrastructure. However, challenges remain, including limited funding, lack of industry partnerships, and insufficient resources for startups. Analysis of student responses indicates a positive influence on entrepreneurial interest and confidence. However, concerns exist around access to funding, infrastructure, and specialized equipment. Additionally, diversifying engagement activities is crucial to reach a wider student base. Based on these results, recommendations are provided to enhance NISP's effectiveness. These include increased funding, fostering industry collaborations, expanding awareness campaigns, and providing high-value resources like seed funding and investor networks. Additionally, dedicated faculty mentors and standardized resource allocation across institutions are recommended. Finally, establishing data-driven evaluation mechanisms with student feedback loops is crucial for continuous improvement. By implementing these recommendations, NISP can be strengthened to cultivate robust entrepreneurial ecosystems within management institutions, fostering innovation and contributing to India's emergence as a global hub for entrepreneurship.

Causal message-passing for experiments with unknown and general network interference

Randomized experiments are a powerful methodology for data-driven evaluation of decisions or interventions. Yet, their validity may be undermined by network interference. This occurs when the treatment of one unit impacts not only its outcome but also that of connected units, biasing traditional treatment effect estimations. Our study introduces a framework to accommodate complex and unknown network interference, moving beyond specialized models in the existing literature. Our framework, termed causal message-passing, is grounded in high-dimensional approximate message-passing methodology. It is tailored for multiperiod experiments and is particularly effective in settings with many units and prevalent network interference. The framework models causal effects as a dynamic process where a treated unit's impact propagates through the network via neighboring units until equilibrium is reached. This approach allows us to approximate the dynamics of potential outcomes over time, enabling the extraction of valuable information before treatment effects reach equilibrium. Utilizing causal message-passing, we introduce a practical algorithm to estimate the total treatment effect, defined as the impact observed when all units are treated compared to the scenario where no unit receives treatment. We demonstrate the effectiveness of this approach across five numerical scenarios, each characterized by a distinct interference structure.

Public Health Policies and Programs for Alzheimer’s and Dementia: A Data-Driven Evaluation of Effectiveness and Areas for Improvement in the United States

Public Health Policies and Programs for Alzheimer’s and Dementia: A Data-Driven Evaluation of Effectiveness and Areas for Improvement in the United States

Smart Classrooms: Mengintegrasikan IoT dan AI untuk Pembelajaran yang Lebih Interaktif dan Terukur

This study examines the impact of integrating the Internet of Things (IoT) and Artificial Intelligence (AI) in Smart Classrooms on interactivity, personalization, and learning evaluation. Using a descriptive qualitative method, data were collected through in-depth interviews, participatory observation, and document analysis in schools implementing this technology. The findings show that IoT and AI enhance student and teacher interaction, enable personalized learning tailored to individual needs, and provide more measurable and data-driven evaluations. Although challenges in technology adaptation and data security were identified, this study affirms that Smart Classrooms have the potential to become a more effective and responsive model of future education, aligned with technological advancements in education. These findings are supported by relevant educational theories, such as Vygotsky’s social learning theory and Bloom’s data-driven evaluation. They are further reinforced by related literature demonstrating that these technologies can significantly enhance the quality of education.

Rethinking the mobile edge for vehicular services

The growing connected car market requires mobile network operators (MNOs) to rethink their network architecture to deliver ultra-reliable low-latency communications. In response, Multi-Access Edge Computing (MEC) has emerged as a solution, enabling the deployment of computing resources at the network edge. For MNOs to tap into the potential benefits of MEC, they need to transform their networks accordingly. Consequently, the primary objective of this study is to design a realistic MEC architecture and corresponding optimal deployment strategy – deciding on the placement and configuration of computing resources – as opposed to prior studies focusing on MEC run-time management and orchestration (e.g., service placement, computation offloading, and user allocation). To cater to the heterogeneous demands of vehicular services, we propose a multi-tier MEC architecture aligned with 5G and Beyond-5G radio access network deployments. Therefore, we frame MEC deployment as an optimization problem within this architecture, assuming 3 MEC tiers. Our data-driven evaluation, grounded in realistic assumptions about network architecture, usage, latency, and cost models, relies on datasets from a major MNO in the UK. We show the benefits of adopting a 3-tier MEC architecture over single-tier (centralized or distributed) architectures for heterogeneous vehicular services, in terms of deployment cost, energy consumption, and robustness.

Beyond competence and retention: Developing a comprehensive evaluation framework for a rural primary care nurse practitioner residency program.

With the rise in nurse practitioner (NP) residency programs, evaluations have largely focused on retention and competency completion for residents. There is a need for expanded evaluation to ensure the sustainability of NP residency programs, to ensure timely adaptations to address resident satisfaction, and to solidify a long-term pathway of NPs well prepared for rural practice. We created a family nurse practitioner (FNP) residency program with a comprehensive evaluation framework to prepare residents for practice in rural settings. The evaluation framework was developed through collaborative engagement of an external evaluation team, program leadership, and clinical site representatives. The evaluation framework of the FNP residency program combined resident assessment and holistic program evaluation, using a rapid continuous quality improvement (QI) approach. The evaluation considered three distinct perspectives: the resident, the peer coach, and the clinical site. The rapid continuous QI approach allowed program leadership to respond swiftly to programmatic challenges, improve the residency program in response to residents' reported experiences, and emphasize sustainability for continued program impact, while assessing residents' learning and performance. The program's data-driven evaluation approach has demonstrated its success in meeting the goals of the Health Resources and Services Administration funding by increasing the number of primary care providers in rural settings. The program's expansion and continued success have further validated the efficacy of this evaluation framework in assessing, improving, and ensuring the sustainability of APRN residency programs. This article calls for the adoption of similar evaluation strategies in future residency programs to promote their long-term success and impact in rural health care settings.

Short Video Platform Advertising Marketing Ideas and Logic--Taking KANS as an Example

With the rapid rise of mobile Internet, short video platforms have captured massive user attention. In 2023, short video users exceeded 750 million, shifting advertising revenue from traditional to new media. Short video marketing, with its rich content and immersive experiences, has become crucial for brand promotion. Young users' creativity is pivotal, and short video ads blend seamlessly with content to enhance brand recognition. However, challenges still exist. Content homogenization, low user participation may impact brand loyalty. Ad effectiveness should be evaluated properly. This paper examines KANS's short video marketing, highlighting strategies like "short drama + advertising," and addressing issues through creative planning and interactive engagement. KANS's success, aided by precise audience targeting and innovative tactics, demonstrates the potential of short video advertising. However, continuous improvement in content uniqueness, user engagement, and data-driven evaluation is necessary for sustained effectiveness. The study provides suggestions for brands to optimize their short video marketing strategies, ensuring alignment with user needs and market dynamics.

Data-driven evaluation for quantifying energy resilience in distribution systems with microgrids and P2P energy trading

Electrical distribution systems (EDS) face the major issue of complete interruption of power when the supply end of the electrical grid encounters extreme events. The integration of distributed energy resources (DERs) at the residential level has opened the path to building energy resilience within these systems, enabling them to withstand such incidents at the load end. To further enhance energy resilience, peer-to-peer (P2P) energy trading networks could be established at the community level. While several researchers have developed models to implement energy trading in the P2P network with DERs, enabling prosumers to benefit from cost savings, the impact of both DERs and P2P energy trading on the system’s resilience has not been evaluated with a quantifiable measure. In this work, we propose a methodology introducing a quantifiable measure for evaluating energy resilience in electrical distribution systems with DERs and P2P energy trading by calculating the percolation threshold (PT) using complex networks for both univariate and multivariate data. We employ cooperative game theory to model P2P energy trading, ensuring that all prosumers participate rationally. We consider the formation of microgrids in a standard IEEE-123 node test feeder system integrated with renewable energy sources. The improvement in energy resilience with trading of energy in the P2P network is analyzed for the most resilient microgrid in the distribution system. The results demonstrate a 67.91% total cost benefit and a 25.07% improvement in the resilience of the microgrid due to the integration of DERs and P2P energy trading for a period of one year.

The New Mixed Teaching Exploration of Low-Voltage Electrician Training Course under the Background of New Engineering

In the context of New Engineering, the exploration of a novel blended teaching approach for low-voltage electrician training courses aims to address the challenges of modern engineering education by integrating New Engineering concepts and modern teaching technologies to achieve comprehensive innovation in course content and teaching methods. This strategy leverages the advantages of combining online and offline learning to provide a diverse and personalized learning experience, thereby enhancing students' knowledge and practical skills. Online course resources, including interactive courseware, virtual experiments, and simulations, enrich the learning content and increase students' interest and engagement. Offline classes focus on interactivity and practical application through discussions, case studies, and project collaborations, helping students deepen their understanding and apply their knowledge. Flexible scheduling allows students to learn at their own pace, improving learning efficiency and effectiveness. Personalized learning path design, through preliminary surveys and assessments, helps students set individualized learning goals. Utilizing big data analysis and intelligent recommendation algorithms, the system provides the most suitable learning resources. Students can autonomously adjust their learning paths, fostering self-directed learning, while teachers offer targeted guidance and support to ensure the successful completion of learning tasks. Interactive and collaborative learning environments are strengthened through online and offline interaction mechanisms, group collaborative learning, and the use of various interactive tools, enhancing classroom interaction and participation. The establishment of online learning communities promotes resource sharing and experience exchange, creating a positive learning atmosphere. Assessment and feedback employ formative and practical assessments through online tests, assignments, and project work, comprehensively reflecting students' learning progress and outcomes. Multi-dimensional feedback mechanisms and data-driven evaluation provide detailed learning analysis and improvement suggestions for both teachers and students, enhancing teaching effectiveness.

Data-driven evaluation of building materials using Ground Penetrating Radar

Ground Penetrating Radar is a widely used technology in the nondestructive evaluation and monitoring of structures during repair and maintenance phases. Within this domain, a critical focus lies in assessing layered structures such as building envelopes. Evaluating attributes like thickness and material type in layered structures traditionally demands labor-intensive manual efforts involving calculations of dielectric properties and extensive signal processing. This study addresses these challenges by leveraging machine learning data-driven models that harness the entire recorded waveform, presenting a potential breakthrough for expediting diagnostic monitoring within the construction industry. We evaluate both supervised and unsupervised machine learning models for classification and regression tasks predicting the properties of common building materials. Experimental GPR A-scans serve as input for these models, with a thorough evaluation of both instance-based and parametric modeling approaches. A detection accuracy of 100% is achieved for identifying outlier scans, while material classification accuracy is 85%. Layer thickness predictions also had a high accuracy, with a typical error of 5%. Additionally, we explore the impact of feature learning and other preprocessing strategies on model performance. Our findings demonstrate the suitability of standard data-driven models for a spectrum of supervised learning tasks for layered structure diagnostics. However, we emphasize the importance of careful attention to the distribution of training data and its relevance to intended use in the field.

Causal Machine Learning in Marketing

This study reviews three primary purposes of causal machine learning (CML) in marketing, merging impact evaluation of marketing interventions with machine learning algorithms for learning statistical patterns from data. Firstly, CML enables more credible impact evaluation by considering important control variables that simultaneously influence the intervention and business outcomes (such as sales) in a data-driven manner. Secondly, it facilitates the data-driven detection of customer segments for which a marketing intervention is particularly effective or ineffective, a process known as effect moderation or heterogeneity analysis. Thirdly, closely related to the second point, it allows for optimal customer segmentation into groups that should and should not be targeted by the intervention to maximize overall effectiveness. The discussion is grounded in recent empirical applications, all of which aim to enhance decision support in marketing by leveraging data-driven evaluation and optimization of interventions across different customer groups.

Research on early identification of burning status in a fire area in Xinjiang based on data-driven

The early combustion stage in coalfield fire areas (CFA) is the initial phase of coal fire development and spread, making the effective identification of the combustion state crucial. To effectively identify the early combustion state of CFA, this study investigates the CFA in Xinjiang, China. Based on the abnormal characteristics of shallow soil and surface space, a data-driven evaluation method for identifying early combustion states of CFA is proposed. Construct a soil-gas dual-layer early combustion state assessment model of CFA. The evaluation model was applied and validated in the coal fire areas of Xinjiang, China.The results show that there are obvious differences in the spatial distribution of auto-ignition gas, temperature, humidity, surface vegetation coverage and other indicators in the shallow soil and surface space of the coal fire area. The early combustion states of coal fires in areas A, B, and C in the survey area are all in the combustion stage, the combustion state in area D is in the oxidation stage. The accuracy of the evaluation method is bidirectionally verified by the radon measurement method and the infrared thermal imaging detection method. It provides a theoretical basis for controlling the development and spread of coalfield fire areas.

Justice, Equity, Diversity, Inclusion, and Belonging (JEDI-B) in Transition to Practice Programs.

Justice, equity, diversity, inclusion, and belonging (JEDI-B) are essential for creating safe and productive professional environments. Clear definitions of JEDI-B terms are essential for collective understanding and organizational buy-in. Institutional and transition to practice program efforts should focus on the development of inclusive cultures, diverse curriculum, and data-driven evaluation of JEDI-B policies, processes, and practices. Engagement in bias recognition and the unlearning of those biases is also vital. Despite current legislative challenges, the integration of JEDI-B principles remains a vital component in enhancing health care experiences for patients and fostering inclusive workplaces. [J Contin Educ Nurs. 2024;55(6):273-275.].

VOC data-driven evaluation of vehicle cabin odor: from ANN to CNN-BiLSTM.

Emissions of volatile organic compounds (VOCs) in vehicles represent a significant problem, causing unpleasant odors. To mitigate VOCs and odors in vehicles, it is critical to choose interior parts with low odor and VOC emissions. However, prevailing odor evaluation methods are subjective, costly, and potentially harmful to the health of evaluators. In this study, we analyzed 139 automotive interior parts and 92 vehicles, establishing a cost-effective, data-driven method for odor evaluation. The contents of benzene, toluene, ethylbenzene, xylene, styrene, formaldehyde, acetaldehyde, acrolein, and total volatile organic compounds (TVOC) were detected by thermal desorption gas chromatography-mass spectrometry (TD-GC/MS) and high-performance liquid chromatography with an ultraviolet detector (HPLC-UV). Professional odor evaluators assessed the odors, identifying intensity levels from 2.0 to 4.5 in interior parts and 2.5 to 3.5 in whole vehicles. Leveraging this data, we applied four supervised learning algorithms to develop predictive models for the odor intensity of both interior parts and entire vehicles. During model training, we implemented early stopping techniques for the artificial neural network (ANN) and convolutional neural network-bidirectional long short-term memory (CNN-BiLSTM) models, while optimizing the support vector machine (SVM) and extreme gradient boosting (XGBoost) models using the GridSearch algorithm. The evaluation results reveal that the CNN-BiLSTM model performs the best, achieving an average accuracy of 89% for unknown samples within an odor intensity level of 0.5. The root mean square error (RMSE) is 0.24, and the mean absolute error (MAE) is 0.08. The model also underwent a sevenfold cross-validation, achieving an accuracy of 83.43%. Additionally, we employed SHapley Additive exPlanations (SHAP) for the interpretative analysis of the model, which confirmed the consistency of each VOC's odor contribution with human olfactory rules. By predicting odors based on VOCs through supervised learning, this study reduces the costs and enhances the efficiency and applicability of odor assessment across various vehicle interiors.

313 Data-Driven Evaluation of Community Health Worker Program

OBJECTIVES/GOALS: Thiswork is an evidential study that demonstrates the positive impactof integrating Community Health Workers (CHWs) and SocialDeterminants of Health on an important health outcome, notably in decreasing the 30-day unplanned hospital ED readmissions at Sinai Health Systems. METHODS/STUDY POPULATION: Using datafrom the Sinai Urban Health Institute (SUHI), we compare predictingthe readmissions of patients with and without data pertainingto Social Determinants of Health (SDoH). We thoroughly describe the data cleaning and data pre-processing, done in collaboration with experts in community health. We use a fundamental and ubiquitous classifier in Random Forest for its feature characterization capability in order to translate models results into insights and recommendations for the CHW program. RESULTS/ANTICIPATED RESULTS: We show that when patients are simply engaged byCHWs, regardless of the content of those conversations, we canincrease the predictive accuracy of our classifier by 5%. We usethis result to make recommendations for improving patient careand discuss limitations and future work. Importantly our workpoints directly to the human connection between patients andCHWs as an important feature in the readmission rate. DISCUSSION/SIGNIFICANCE: Our work shows that the predictive capabilities of the classifier increases with CHW logs and SDoH survey data, highlighting the benefit of collecting this information. This is the first step in early identification of such patients so that CHWs are focusing on and providing resources to patients who will most benefit from the program.

A quantitative assessment of the status of benthic communities on US Atlantic coral reefs using a novel standardized approach

As coral reefs decline globally, the need for an objective approach to quantify the status and trends of corals has become increasingly important. Empirical data on predisturbance conditions are rare, and integrating data from multiple and disparate survey designs and methods can be analytically challenging. Our goal was to conduct a holistic, data-driven evaluation of the status of corals and benthic communities in US Atlantic coral reef jurisdictions: Florida, Flower Garden Banks, Puerto Rico, and the US Virgin Islands. A quantitative approach based upon standardized data was used to compare the change in multiple indicators of coral condition (hard coral, macroalgae, and crustose coralline algae cover, coral density, and old mortality) from historic to current conditions in each geographic region. For each indicator, historic, reference baseline conditions from long-term monitoring data or literature data were first identified, reviewed, and classified on a categorical scale from Very Good to Critical by regional experts to account for condition changes that pre-dated current monitoring data. A reference-centering approach then allowed for categorization of statistical changes from historic to current conditions on the same scale to produce results that could be communicated to a broad audience. Our findings show continued declines for multiple indicators in all regions except Flower Garden Banks, illustrate particularly dire declines from regions that had been impacted by Stony Coral Tissue Loss Disease at the most recent monitoring included in this study, and demonstrate the increasingly critical need for effective coral reef conservation.

Comparison of recruitment methodologies for clinical trials: Results from the time for living and caring (TLC) intervention study

BackgroundRecruiting participants for research studies is a critical yet challenging task. Community-engaged recruitment strategies have gained prominence as effective means to engage diverse populations and ensure the representativeness of study samples. This case study aims to investigate the cost and effectiveness of various recruitment methods in enhancing research participation. MethodsA comparative approach was employed to assess the outcomes of five different recruitment strategies used in the Time for Living & Caring (TLC) research study. Data on recruitment success, participant demographics, and retention rates were collected and analyzed using descriptive statistics, including ANOVA and Chi-squares, to statistically compare the outcomes associated with 5 different recruitment methodologies. The recruitment methodologies included two community-engaged strategies (community partner referral and community-based recruiters), a clinical database, social media, and word-of-mouth referral. ConclusionThe meta-data used to build this methodological case study describe different recruitment methodologies that may be used for clinical trials. This data-driven evaluation provides examples and considerations for researchers when developing budgets and proposals for future clinical trials. The primary finding is that there are tradeoffs in terms of cost, time, labor, and ultimately the representativeness of the sample, based on the type of recruitment methodology chosen.