Validity Problems Research Articles

G4 & the balanced metric family – a novel approach to solving binary classification problems in medical device validation & verification studies

BackgroundIn medical device validation and verification studies, the area under the receiver operating characteristic curve (AUROC) is often used as a primary endpoint despite multiple reports showing its limitations. Hence, researchers are encouraged to consider alternative metrics as primary endpoints. A new metric called G4 is presented, which is the geometric mean of sensitivity, specificity, the positive predictive value, and the negative predictive value. G4 is part of a balanced metric family which includes the Unified Performance Measure (also known as P4) and the Matthews’ Correlation Coefficient (MCC). The purpose of this manuscript is to unveil the benefits of using G4 together with the balanced metric family when analyzing the overall performance of binary classifiers.ResultsSimulated datasets encompassing different prevalence rates of the minority class were analyzed under a multi-reader-multi-case study design. In addition, data from an independently published study that tested the performance of a unique ultrasound artificial intelligence algorithm in the context of breast cancer detection was also considered. Within each dataset, AUROC was reported alongside the balanced metric family for comparison. When the dataset prevalence and bias of the minority class approached 50%, all three balanced metrics provided equivalent interpretations of an AI’s performance. As the prevalence rate increased / decreased and the data became more imbalanced, AUROC tended to overvalue / undervalue the true classifier performance, while the balanced metric family was resistant to such imbalance. Under certain circumstances where data imbalance was strong (minority-class prevalence < 10%), MCC was preferred for standalone assessments while P4 provided a stronger effect size when evaluating between-groups analyses. G4 acted as a middle ground for maximizing both standalone assessments and between-groups analyses.ConclusionsUse of AUROC as the primary endpoint in binary classification problems provides misleading results as the dataset becomes more imbalanced. This is explicitly noticed when incorporating AUROC in medical device validation and verification studies. G4, P4, and MCC do not share this limitation and paint a more complete picture of a medical device’s performance in a clinical setting. Therefore, researchers are encouraged to explore the balanced metric family when evaluating binary classification problems.

Validity of Problem Based Learning Oriented E-LAPD to Improve Chemical Literacy Skills of High School Students

This study aims to produce valid Problem Based Learning (PBL) oriented E-LAPD to improve students' chemical literacy skills on acid-base material. This research method used development research with the 4D model which includes define, design, develop, and disseminate. However, it is limited to the development stage. The subjects of this study were phase F students of class XI SMAN 1 Krembung Sidoarjo as many as 36 students. The research instrument used in collecting data was a validation sheet. Data analysis techniques in this study used quantitative analysis techniques through the provision of validation questionnaires to validators. The results of this study indicated that the Problem Based Learning (PBL) oriented E-LAPD to improve students' chemical literacy skills on acid-base material was declared feasible to use because it got a mode score of 4 with valid criteria based on content and construct validity which includes aspects of language, presentation, and graphics.

Wind farm power optimization using system identification

The wake effect reduces the total power production of wind farms. This paper presents a method for wind farm power optimization through wake effect reduction. The proposed method optimizes the yaw angle offsets and de-rating settings of all turbines to maximize total power generation. The optimization approach is gradient-based, with gradients at each iteration obtained through system identification using field test data, eliminating the need for physical models. In system identification, test signal design, model estimation and model validation problems are solved in a systematic manner; in the gradient-based optimization, in order to achieve fast convergence, methods for initial value and initial step-size determination, variable step-size iteration and iteration termination are developed. The method is verified using the FLORIS wind farm model developed by National Renewable Energy Laboratory (NREL), USA. The studied wind farm consists of 80 wind turbines configured similarly to the Horns Rev I offshore wind farm in Denmark. The result of the developed optimization method is highly consistent with those obtained using FLORIS's built-in optimization tool.

On the Use of SuperDARN Ground Backscatter Measurements for Ionospheric Propagation Model Validation

AbstractPrior to use in operational systems, it is essential to validate ionospheric models in a manner relevant to their intended application to ensure satisfactory performance. For Over‐the‐Horizon radars (OTHR) operating in the high‐frequency (HF) band (3–30 MHz), the problem of model validation is severe when used in Coordinate Registration (CR) and Frequency Management Systems (FMS). It is imperative that the full error characteristics of models is well understood in these applications due to the critical relationship they impose on system performance. To better understand model performance in the context of OTHR, we introduce an ionospheric model validation technique using the oblique ground backscatter measurements in soundings from the Super Dual Auroral Radar Network (SuperDARN). Analysis is performed in terms of the F‐region leading edge (LE) errors and assessment of range‐elevation distributions using calibrated interferometer data. This technique is demonstrated by validating the International Reference Ionosphere (IRI) 2016 for January and June in both 2014 and 2018. LE RMS errors of 100–400 km and 400–800 km are observed for winter and summer months, respectively. Evening errors regularly exceeding 1,000 km across all months are identified. Ionosonde driven corrections to the IRI‐2016 peak parameters provide improvements of 200–800 km to the LE, with the greatest improvements observed during the nighttime. Diagnostics of echo distributions indicate consistent underestimates in model NmF2 during the daytime hours of June 2014 due to offsets of −8° being observed in modeled elevation angles at 18:00 and 21:00 UT.

Automatic Detection of Anomalies in Post-Processed Data Applied to UTC Time Transfer Links.

In this article, we present a method for identifying anomalies, particularly time steps, which can affect data. Recognition of these anomalies is essential for understanding the intrinsic nature of problems that may occasionally affect the data, and for guaranteeing system reliability and accuracy. The tool presented, based on the Kalman filter, is optimized to work with post-processed data which means that the dataset is available at the time the algorithm is run. The main aim is to retain as much data as possible, while detecting anomalies, and avoid deleting valuable data. The originality of this tool with respect to the already existing Kalman-filter-based tools for detecting anomalies is substantial, because its objective is not only to enable the system to run but also to avoid unnecessary deletion of valuable data. This tool is designed to accurately determine the date of occurrence and magnitude of these anomalies, focusing on time steps. The tool presented will be applied, by way of example, to the time links used in Coordinated Universal Time (UTC) as calculated by Bureau International des Poids et Measures (BIPM), Paris, France. In addition, the algorithm developed will enable the BIPM's Time Department to be rapidly alerted to unexpected behavior that may compromise UTC performance. To guarantee the reliability and accuracy of UTC, rigorous data validation and rapid problem identification are essential.

The mean field limit for the Vlasov–Poisson system with a point charge

In this paper, we are concerned with the validity problem for the repulsive Vlasov–Poisson system when the initial datum is the sum of a diffuse density and a point charge in the three dimensional space. Our result leads to a derivation of this system from the regularized microscopic N-plasma particle and one point charge system when the Coulomb force between plasma particles needs to be modified with the modified size N − 1 3 + ϵ while the Coulomb force between plasma and point charge does not need to be modified. More precisely, we obtain the convergence rate between the empirical measure associated to the regularized particle system and the solution of the Vlasov–Poisson system with a point charge in terms of the Wasserstein distance.

Pengembangan Media Pembelajaran Berbasis Augmented Reality pada Materi Bangun Ruang dengan Model Problem Based Learning (PBL) di Kelas IV Sekolah Dasar

The research is motivated by the fact that not all teachers use technology-based media in teaching and learning activities, especially in learning about building cube and block spaces. Teachers use more objects around students as media. This research aims to develop learning media based on augmented reality technology on classroom building materials in grade IV elementary school using a valid, practical and effective Problem Based Learning model. The type of research used in this study is development research using the ADDIE (Analysis, Design, Develop, Implement and Evaluate) model. The subjects of product trials and research data collection were grade IV students of SDN 08 Surau Gadang. The validation instrument used in this study is in the form of a validation sheet which aims to determine the validity or not of learning media based on Augmented Reality technology on space building materials. The analysis technique in this study uses the validity of learning media and the practicality of learning media. The results of the study show that the development of learning media based on augmented reality technology on building materials for cubes, blocks and prisms with the Problem Based Learning (PbL) model has been developed using the ADDIE development model and obtained an average result from three expert validators with a percentage of 81.33% which is included in the "Very Valid" category. The practicality of teachers' response to learning media based on Augmented Reality technology in building materials for cubes, blocks and triangular prisms in grade IV of elementary school obtained a percentage result of 90% with the very practical category at SDN 09 Surau Gadang and 96% with the very practical category at SDN 08 Surau Gadang. The results of the effectiveness test at the first meeting were obtained with a learning outcome percentage of 82.16%. Then at the second meeting, the percentage of learning outcomes increased by 88.16% at SDN 09 Surau Gadang. Then at SDN 08 Surau Gadang, the results of the effectiveness test at the first meeting were 87.19% and the results of the effectiveness test increased at the second meeting by 93%.

Teaching Exams: HSI Candidates’ Use of an Adaptive Online Test Preparation Program

ABSTRACT Supporters of teaching exams say they create better outcomes, but teacher shortages, equity issues, and validity and reliability problems indicate more research is warranted. This quantitative study investigated how an adaptive online test preparation program influenced TExES EC-6 exam performance for teacher candidates at a Hispanic Serving Institution (HSI). The framework was based on self-efficacy and critical pedagogy, and descriptive and inferential statistics were used to compare performance. The institution where the study was performed includes a student population that is 93% Hispanic and 52.27% first-generation. The study began with 324 candidates’ but was eventually narrowed to 77 participants. The program was most effective for those using it for 10–19 hours, while those with baseline scores below 60% benefitted the most. Implications include providing instructional sessions about the program, asking students who have passed to serve as mentors, and researching the long-term retention of content knowledge.

Development of Electronic Teaching Materials Based on Contextual Teaching and Learning (CTL) Approach to Improve High School Students' Mathematical Problem Solving Skills

The aim of this research is to develop approach-based electronic teaching materials Contextual Teaching And Learning (CTL) on systems of equations and linear inequalities to improve high school students' valid and practical mathematical problem solving abilities. This research method uses research and development with a 4D model. Data was collected using questionnaires, observations and interviews. The data analysis technique used is quantitative descriptive analysis. Teaching materials are said to be valid and practical if the score obtained is more than 70%. The results of this research show that electronic teaching materials are based The CTL approach was developed to increase mathematical problem solving abilities for high school students get score mean 89% with very valid category. The results of student responses in the limited trial obtained a score of 90.28% in the very practical category. The student response results in the field trial were 90.35% with very practical criteria. The teacher response results were 94.14% with the very practical category and the observation sheet 90.6% with the very practical criteria. From the results of validity and practicality, it can be seen that the teaching materials developed are suitable for use as learning resources.

Step Forward Cross Validation for Bioactivity Prediction: Out of Distribution Validation in Drug Discovery.

Recent advances in machine learning methods for materials science have significantly enhanced accurate predictions of the properties of novel materials. Here, we explore whether these advances can be adapted to drug discovery by addressing the problem of prospective validation - the assessment of the performance of a method on out-of-distribution data. First, we tested whether k-fold n-step forward cross-validation could improve the accuracy of out-of-distribution small molecule bioactivity predictions. We found that it is more helpful than conventional random split cross-validation in describing the accuracy of a model in real-world drug discovery settings. We also analyzed discovery yield and novelty error, finding that these two metrics provide an understanding of the applicability domain of models and an assessment of their ability to predict molecules with desirable bioactivity compared to other small molecules. Based on these results, we recommend incorporating a k-fold n-step forward cross-validation and these metrics when building state-of-the-art models for bioactivity prediction in drug discovery.

Statistical Validation of Multiple Related Data Sets—Case Study Using Interstellar Boundary Explorer Satellite Data

Space scientists often face the question of whether data collected by different instruments are measurements of the same source population. This paper proposes a statistical validation method for evaluating the agreement between such related data sets. It offers a detailed case study focused on validating a new data set from the Interstellar Boundary Explorer (IBEX) mission, which serves as a practical how-to guide for similar analyses. Since 2008, the IBEX satellite has been gathering data on heliospheric energetic neutral atoms (ENAs) while being exposed to various sources of background noise, such as cosmic rays and solar energetic particles. The IBEX mission initially released only a qualified triple-coincidence (qABC) data product, which was designed to provide observations of ENAs free of background contamination. Further measurements revealed that the qABC data were in fact susceptible to contamination, having relatively low ENA counts and high background rates. To mitigate this issue, the mission team recently considered releasing a certain qualified double-coincidence (qBC) data product, which has roughly twice the detection rate of the qABC data product. This paper presents a simulation-based validation of the new qBC data product against the already-released qABC data product. The results show that the qBCs can plausibly be said to be measuring the same source population as the qABCs up to an average absolute deviation of 3.6%. Visual diagnostics provide additional confirmation of source rate coherence across data products. The framework introduced here is general and can be applied to other validation problems both within and outside the field of space physics.

A multiphysics coupling framework for exascale simulation of fracture evolution in subsurface energy applications

Predicting the evolution of fractured media is challenging due to coupled thermal, hydrological, chemical and mechanical processes that occur over a broad range of spatial scales, from the microscopic pore scale to field scale. We present a software framework and scientific workflow that couples the pore scale flow and reactive transport simulator Chombo-Crunch with the field scale geomechanics solver in GEOS to simulate fracture evolution in subsurface fluid-rock systems. This new multiphysics coupling capability comprises several novel features. An HDF5 data schema for coupling fracture positions between the two codes is employed and leverages the coarse resolution of the GEOS mechanics solver which limits the size of data coupled, and is, thus, not taxed by data resulting from the high resolution pore scale Chombo-Crunch solver. The coupling framework requires tracking of both before and after coarse nodal positions in GEOS as well as the resolved embedded boundary in Chombo-Crunch. We accomplished this by developing an approach to geometry generation that tracks the fracture interface between the two different methodologies. The GEOS quadrilateral mesh is converted to triangles which are organized into bins and an accessible tree structure; the nodes are then mapped to the Chombo representation using a continuous signed distance function that determines locations inside, on and outside of the fracture boundary. The GEOS positions are retained in memory on the Chombo-Crunch side of the coupling. The time stepping cadence for coupled multiphysics processes of flow, transport, reactions and mechanics is stable and demonstrates temporal reach to experimental time scales. The approach is validated by demonstration of 9 days of simulated time of a core flood experiment with fracture aperture evolution due to invasion of carbonated brine in wellbore-cement and sandstone. We also demonstrate usage of exascale computing resources by simulating a high resolution version of the validation problem on OLCF Frontier.

Metarouting with automatic tunneling in multilayer networks

Metarouting allows for the modeling of routing protocols using an algebraic structure called routing algebra. Routing protocols requiring design or validation can easily be modeled using this approach. To date, however, existing research on routing algebras has mainly focused on applying this approach to routing protocols that are generally used in networks which have a single addressing and forwarding protocol. The basic algebraic structures used in such contexts are semirings, Sobrinho’s algebras and algebras of endomorphisms. In this paper, we propose the modification of these existing routing algebras to deal with networks that contain multiple forwarding protocols where tunnels are omnipresent. To achieve this, we define new algebraic structures derived from the three aforementioned ones, in order to model the generalized routing problem with automatic tunneling entitled valid paths algebra. All of our routing algebras are defined as semi-direct products of two structures, the well-known shortest paths algebra and the proposed valid paths algebra. These new algebras are isotonic and non-monotonic with a partial order. We propose a fixed point for those new algebras and we prove the iterative convergence to the optimal solution of the valid shortest paths problem.

Bayesian Safety Validation for Failure Probability Estimation of Black-Box Systems

Estimating the probability of failure is an important step in the certification of safety-critical systems. Efficient estimation methods are often needed due to the challenges posed by high-dimensional input spaces, risky test scenarios, and computationally expensive simulators. This work frames the problem of black-box safety validation as a Bayesian optimization problem and introduces a method that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce three acquisition functions that aim to reduce uncertainty by covering the design space, optimize the analytically derived failure boundaries, and sample the predicted failure regions. Results show this Bayesian safety validation approach provides a more accurate estimate of failure probability with orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate this approach on three test problems, a stochastic decision-making system, and a neural-network-based runway detection system. This work is open-sourced (https://github.com/sisl/BayesianSafetyValidation.jl) and is currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.

Hybrid lattice Boltzmann method using Cartesian and body-fitted grids for turbomachinery aeroacoustic simulations

In this study, we developed a hybrid approach using the lattice Boltzmann method (LBM) and finite-volume lattice Boltzmann method (FVLBM) to perform efficient aeroacoustic simulations with moving boundaries. An entire domain, including the flow and acoustic fields, was computed using the standard LBM with a Cartesian grid. Local domains around the moving objects were computed using the FVLBM with body-fitted grids. These simulations were coupled using the direct forcing method to consider moving boundaries. The hybrid method was validated for several problems including turbulent flows and flow-induced sounds under low-Mach-number conditions. These validation problems covered flows with Reynolds numbers up to 2.0×105 and Mach numbers less than 0.2. In the simulations of the aeolian tone generated from stationary and rotating cylinders, the hybrid method results were consistent with those of the conventional methods. In the simulation of the turbulent flow and broadband sound of the isolated airfoil, the hybrid method was 15 times faster than the standard LBM and produced results consistent with the experimental results. Furthermore, the application of a cross-flow fan demonstrated that the hybrid method successfully simulated the flow and acoustic fields around the rotor.

Reasoning on property graphs with graph generating dependencies

Data dependencies are a key concept in data management and have been researched in data integration, data quality and query optimization. With the increasing use of graph-structured data in diverse applications, there is also an increasing interest in the study of graph data dependencies. In this scenario, different classes of graph data dependencies have been proposed in the literature. In this work, we study the class of Graph Generating Dependencies (GGDs). Graph Generating Dependencies (GGDs) informally express constraints between two (possibly different) graph patterns which enforce relationships on both graph's data (via property value constraints) and its structure (via topological constraints). While most of previously proposed classes of graph data dependencies focus on generalizing equality-generating dependencies for graph data, Graph Generating Dependencies (GGDs) can express tuple- and equality-generating dependencies on property graphs, both of which find broad application in graph data management. Given this new class of dependency, in this paper, we discuss the reasoning behind GGDs on Property Graphs. We propose algorithms to solve three main reasoning problems: the satisfiability, implication, and validation problems for GGDs and analyze their complexity. By studying these problems, we can understand the expressiveness and the limitations of GGDs in practical applications. To demonstrate the practical use of GGDs, we propose an algorithm that finds inconsistencies in data through validation of GGDs. Our experiments show that even though the validation of GGDs has high computational complexity, GGDs can be used to find data inconsistencies in a feasible execution time on both synthetic and real-world data.

PENGEMBANGAN MODUL KAJIAN MATEMATIKA SEKOLAH BERBASIS PROBLEM BASED LEARNING

The aim of this research is to develop a School Mathematics Study module based on Problem Based Learning that is valid and effective. This Problem Based Learning-based module emphasizes problem solving in understanding mathematical concepts. The School Mathematics Study Module based on Problem Based Learning refers to the 4D development model, which is only limited to the Develop stage. The data collection instrument consists of a validation sheet and problem solving test questions. The trial was carried out on students of the Universitas Insan Budi Utomo Mathematics Education study program who were taking the School Mathematics Studies course. The results of data analysis show that the module fulfill the criteria for validity and effectiveness. Expert validation and module testing. From the assessment results from two experts, an average score of 3,2 was obtained, which is a very valid criterion.

Joint Situational Assessment‐Hierarchical Decision‐Making Framework for Maneuver Intent Decisions

Decision‐making in unmanned combat aerial vehicles (UCAVs) presents a multifaceted challenge because of the complexity and dynamics of the flight environment, which leads to hurdles in training convergence, low decision validity, and the dimensionality catastrophe for decision‐making neural networks. A novel framework is proposed to address breaking down the complicated decision issues, which combines the strengths of graph convolutional networks in relation extraction with the ability of hierarchical reinforcement learning. To solve the problem of decision validity under high‐dimensional inputs, the joint framework is applied to the Maneuver Intent's decision, and a maneuver library‐based state space design method is suggested. The joint framework executes adaptable strategies and flight maneuvers to address the issue of training non‐convergence or task failure due to difficult‐to‐obtain reward signals across various scenarios. Then, the recurrent curriculum training and cross‐entropy rewards are designed to train decisions on different sub‐strategies. The experimental evaluation demonstrated more flexibility and adaptability in decision‐making problems under complex tasks compared to rule‐based and reinforcement learning baseline methods. The method proposed in this article provides a novel approach to resolving intricate decision problems, and which has certain theoretical significance and reference value for engineering applications.

Pengembangan Proyek Pemrograman Medıa Pembelajaran Interaktıf Untuk Tugas Pemecahan Masalah Pelatihan Berpıkır Komputasıonal Mahasiswa Dengan Scratch

Media is needed in learning to attract and make it easier for students to understand the material being taught. Interactive learning media for computational thinking material through Scratch programming as an example of a block-based programming project assignment by applying computational thinking concepts for PGSD student training has not been researched in Indonesia. The research carried out aims to develop an example of a final programming project for interactive media learning computational thinking for PGSD student programming training as a valid and practical problem solving task using the Scratch program as well as the level of competency of the programming project being created. The research carried out is of the research and development type using the ADDIE model. The research participants were 40 students from a private university in West Sumatra. Data was collected using observation, interview, artifact analysis, and questionnaire techniques. The research instrument was a questionnaire with a Likert scale filled in by material, media and research subject experts. Analysis of research data was carried out in a qualitative descriptive manner. From the development research carried out, it was found that the final project example of interactive media programming for computational thinking learning which was developed for PGSD students' computational thinking training as a problem solving task received a very good score with a score of 85.5% from the validation of material experts, media and students, 85. 2% for product practicality, and makes it easier for students to created the program because the difficulty is at intermediate level.

Designing a Blockchain-based SME transaction model using the Federated Byzantine Agreement to shorten transaction validation times

Since the world experienced the Covid-19 pandemic, the role of MSMEs in the e-commerce sector has grown rapidly and contributed 50% of Indonesia's GDP. However, investors and some consumers still underestimate MSMEs. They have a low level of trust in MSMEs. This is due to the lack of a transparent system so that cases of fraud against MSME products often occur. This issue of trust and transparency needs to be resolved with technological solutions. One technology that can solve this problem is Blockchain. However, currently there are still validation problems if it is to be implemented in MSMEs. This research aims to resolve validation time constraints in the implementation of MSMEs. This research builds a Blockchain-based MSME transaction model using Federated Byzantine Agreement (FBA) algorithm to solve validation time problems. The result shows that FBA is able to complete transactions with the fastest validation time compared to Proof of Work (POW), Proof of State (POS), and Proof of Authority (POA).