Selection Of Metrics Research Articles

Searching for Consistency in Attendance Data Recording, Reporting, and Utilization in the USA

According to the United States Department of Education (USDOE), 16% or over eight million kindergarten through twelfth grade students in the US missed 10% or more school days during the 2017−2018 school year. This is approximately 18 of 180 days required. We know this because schools are mandated to report their attendance data to their respective states and to the USDOE. There are concerns around accuracy and consistency because each state is allowed to compile data in their own way and report only select metrics to the USDOE to comply with federal guidelines. The consistency on federal metrics, nonetheless, allows for similar analyses at the federal and state levels and comparisons across states. To best understand what is reported, we report on data compiled by the National Center for Education Statistics (NCES) housed in the Institute of Education Sciences, the science branch of the USDOE, and describe how attendance data are collected, reported, and used at the national level. We share similar findings for two representative US states − Connecticut and Indiana − to highlight similarities and differences between them, and their “best practices.” Key results from these multiple levels of analyses are then discussed, with the goal of informing research, practice, and policy related to school attendance, so that students of all ages and from all backgrounds are provided the opportunity to obtain optimal benefits from schooling throughout their school careers.

Development of an empirical model for the prediction of the noise reduction coefficient for thin and low-density fibrous materials

This paper presents the development of an empirical noise reduction coefficient model for the prediction of low-density, less than 50 kg/m3, thin, less than 20 mm thick, fibrous materials using multiple linear regression. The purpose of this empirical model is to assist design engineers, working with thin and low-density materials, efficiently and effectively select the most appropriate material for the design. Therefore, several models were developed using software such as Statistical Analysis System. Thereafter, the models were compared using an internal and external data set. A selection metric was developed to assist in the objective selection of the best model. It was found that the log model performed the best overall and thus was selected as the model of choice.

Pareto Dominance Archive and Coordinated Selection Strategy-Based Many-Objective Optimizer for Protein Structure Prediction.

Protein structure prediction (PSP) is predicting the three-dimensional of protein from its amino acid sequence only based on the information hidden in the protein sequence. One of the efficient tools to describe this information is protein energy functions. Despite the advancements in biology and computer science, PSP is still a challenging problem due to its large protein conformation space and inaccurate energy functions. In this study, PSP is treated as a many-objective optimization problem and four conflicting energy functions are used as different objectives to be optimized. A novel Pareto-dominance-archive and Coordinated-selection-strategy-based Many-objective-optimizer (PCM) is proposed to perform the conformation search. In it, convergence and diversity-based selection metrics are used to enable PCM to find near-native proteins with well-distributed energy values, while a Pareto-dominance-based archive is proposed to save more potential conformations that can guide the search to more promising conformation areas. The experimental results on thirty-four benchmark proteins demonstrate the significant superiority of PCM in comparison with other single, multiple, and many-objective evolutionary algorithms. Additionally, the inherent characteristics of iterative search of PCM can also give more insights into the dynamic progress of protein folding besides the final predicted static tertiary structure. All these confirm that PCM is a fast, easy-to-use, and fruitful solution generation method for PSP.

Graphene oxide/cellulose nanofibril composite: A high-performance catalyst for the fabrication of an electrochemical sensor for quantification of p-nitrophenol, a hazardous water pollutant

The detection and quantification of p-Nitrophenol in environmental samples are important for understanding the extent and impact of environmental pollution, protecting human health, ensuring regulatory compliance, and guiding remediation efforts. The main objective of this work was to investigate the electrochemical performance of a graphene oxide/cellulose nanofibril composite (GO/CNF) modified carbon paste electrode (GO/CNF/CPE) for the sensitive and reliable detection of p-nitrophenol in water samples. The transmission electron microscopy (TEM) technique was employed to enlighten the structure of nanocomposites. The electrochemical behavior of the fabricated electrochemical sensor was characterized via differential pulse voltammetry (DPV), linear sweep voltammetry (LSV), and electrochemical impedance spectroscopy (EIS). Under optimized analytical conditions, the peak current of the analyte showed a wide linear relationship with its concentration in a range of 3.0 nM–210 μM with a low amount of the limit of detection (LOD) value of 0.8 nM determined by the DPV method. The proposed electrochemical sensor demonstrated excellent sensitivity, selectivity, and accuracy metrics in real sample analysis of p-nitrophenol.

Formal evaluation of construction safety performance metrics and a case for a balanced approach

Introduction: Measuring safety performance is crucial to making informed decisions that improve construction safety management. Traditional approaches to construction safety performance measurement primarily focus on injury and fatality rates, but researchers have recently proposed and tested alternative metrics such as safety leading indicators and safety climate assessments. Although researchers tend to extol the benefits of alternative metrics, they are studied in isolation and the potential weaknesses are rarely discussed, leaving a critical gap in knowledge. Method: To address this limitation, this study aimed to evaluate existing safety performance against a set of pre-determined criteria and explore how multiple metrics may be used together to optimize strengths and offset weaknesses. For a well-rounded evaluation, the study included three evidence-based assessment criteria (i.e., the extent to which the metric is predictive, objective, and valid) and three subjective criteria (i.e., the extent to which each metric is clear, functional, and important). The evidence-based criteria were evaluated using a structured review of existing empirical evidence in literature, while the subjective criteria were evaluated using expert opinion solicited through the Delphi method. Results: The results revealed that no construction safety performance measurement metric is strong in all evaluation criteria, but many weaknesses may be addressed through research and development. It was also demonstrated that combining multiple complementary metrics may result in a more complete evaluation of the safety systems because multiple metrics offset respective strengths and weaknesses. Practical Applications: The study provides a holistic understanding of construction safety measurement that may guide safety professionals in their selection of metrics and assist researchers who seek more reliable dependent variables for intervention testing and safety performance trending.

Performance measurement of crypto funds

Crypto funds (CFs) are a growing intermediary in cryptocurrency markets. We evaluate CF performance using metrics based on alphas, value at risk, lower partial moments, and maximum drawdown. The performance of actively managed CFs is heterogeneous: While the average fund in our sample does not outperform the overall cryptocurrency market, there seem to be some few funds with superior skills. Given the non-normal nature of fund returns, the choice of the performance measure affects the rank orders of funds. Compared to the Sharpe ratio, the most commonly applied metric in the asset management practice, performance measures based on alphas and maximum drawdown lead to diverging fund rankings. Depending on their ranking order of preferences, CF investors should consider a bundle of metrics for fund selection and performance measurement.

Abstract LB062: Identifying novel targets for CAR-T therapies from single cell RNA-sequencing data

Abstract Chimeric antigen receptor T (CAR-T) cell therapies have revolutionized cancer treatment. While CAR-T has yielded tremendous clinical success for patients with liquid tumors, its potential remains to be unleashed against solid tumors. One key challenge is identifying optimal targets for these therapies: cell surface proteins that are expressed highly and uniformly by a tumor’s constituent malignant cells, and minimally so by healthy tissues. Employing a systematic, data-driven analysis, we first charted the landscape of existing CAR-T targets in the clinic, identifying the leading targets in each indication based on tumor selectivity and safety metrics. Next, from patient tumor single cell transcriptomics data, we performed a genome wide search across many different solid tumor types to identify new and candidate CAR-T targets with better selectivity and safety scores than extant ones. Remarkably, in almost all indications, we could not find such better targets, testifying to the near optimality of the current target space, at least in accordance with our measures. However, one striking exception is HPV-negative head and neck squamous cell carcinoma (HNSC), for which there is currently a dearth of existing CAR-T targets in clinics. Specifically, our investigation has discovered 20 novel CAR-T targets for treating HNSC and one for treating glioblastoma more precisely and safely. Citation Format: Sanna Madan, Sanju Sinha, Alejandro A. Schäffer, Eytan Ruppin. Identifying novel targets for CAR-T therapies from single cell RNA-sequencing data [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 2 (Clinical Trials and Late-Breaking Research); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(8_Suppl):Abstract nr LB062.

Evaluating and sustaining Coordinated Specialty Care for a recent onset of psychosis in non-academic-affiliated community mental healthcare settings

PurposeTo improve sustainability of Coordinated Specialty Care (CSC) for a recent onset of psychosis, a better understanding is needed regarding how non-academic-affiliated community mental health centers blend CSC service elements and select key performance metrics to evaluate their approach. MethodsA quality and evaluation team embedded within a large community mental health center partnered with CSC site leadership to implement CSC and design a program evaluation strategy informed by CSC research literature. Clinical, family, vocational, and psychiatry services participation, exits, key performance indicators, and standardized measures were examined for participants (n = 47) enrolled for 12-months. ResultsMean service participation was 55 h (SD = 23.5) in the first 12-months (approximately 4.70 h/month). All participated in clinical; 87% in psychiatry; 67% in vocational; and 57% in family services. Sixty-one percent had planned service exits; 39% had unplanned exits. Across the 12-months, 83% were employed or in school; 72% were not psychiatric hospitalized. ConclusionsCSC participation and outcomes were similar to the limited research examining both together. Understanding service participation and provider adjustments to sustain CSC is critical in community mental healthcare settings that rely on fee-for-service billing mechanisms. Findings have implications for national CSC data harmonization and sustainability efforts.

Stream power indices correspond poorly with observations of alluvial river channel adjustment

AbstractA variety of stream power‐based approaches for predicting catchment‐scale alluvial channel adjustment have been developed. There is an international interest in applying these to inform river catchment management. However, there is some uncertainty regarding their ability to make consistently accurate predictions. This study evaluates the performance of a range of stream power indices for predicting observed channel adjustment. Remotely sensed data were used to generate 33 different stream power indices every 50 m across the networks of five test catchments. The performances of the indices were evaluated by comparing them against observations of erosion and deposition‐dominated channels extracted from the UK's River Habitat Survey (RHS) database. A selection of metrics for evaluating the performance of the indices were calculated. The key finding from this study is that the stream power indices were poorly associated with the observations of alluvial channel adjustment. It is not clear whether this poor association is due to limitations with stream power indices or the suitability of the RHS observations. However, this is not the first study to find a weak association between stream power and observed adjustment. Therefore, caution is recommended to anyone hoping to take advantage of the practicability of stream power indices until further testing is applied using alternative observation datasets. An additional finding from this study is the inconsistency of outcomes between different measures of model performance. It is recommended that future studies also employ multiple model performance measures rather than relying on accuracy alone.

Enhancing the defectiveness prediction of methods and classes via JIT

ContextDefect prediction can help at prioritizing testing tasks by, for instance, ranking a list of items (methods and classes) according to their likelihood to be defective. While many studies investigated how to predict the defectiveness of commits, methods, or classes separately, no study investigated how these predictions differ or benefit each other. Specifically, at the end of a release, before the code is shipped to production, testing can be aided by ranking methods or classes, and we do not know which of the two approaches is more accurate. Moreover, every commit touches one or more methods in one or more classes; hence, the likelihood of a method and a class being defective can be associated with the likelihood of the touching commits being defective. Thus, it is reasonable to assume that the accuracy of methods-defectiveness-predictions (MDP) and the class-defectiveness-predictions (CDP) are increased by leveraging commits-defectiveness-predictions (aka JIT).ObjectiveThe contribution of this paper is fourfold: (i) We compare methods and classes in terms of defectiveness and (ii) of accuracy in defectiveness prediction, (iii) we propose and evaluate a first and simple approach that leverages JIT to increase MDP accuracy and (iv) CDP accuracy.MethodWe analyse accuracy using two types of metrics (threshold-independent and effort-aware). We also use feature selection metrics, nine machine learning defect prediction classifiers, more than 2.000 defects related to 38 releases of nine open source projects from the Apache ecosystem. Our results are based on a ground truth with a total of 285,139 data points and 46 features among commits, methods and classes.ResultsOur results show that leveraging JIT by using a simple median approach increases the accuracy of MDP by an average of 17% AUC and 46% PofB10 while it increases the accuracy of CDP by an average of 31% AUC and 38% PofB20.ConclusionsFrom a practitioner’s perspective, it is better to predict and rank defective methods than defective classes. From a researcher’s perspective, there is a high potential for leveraging statement-defectiveness-prediction (SDP) to aid MDP and CDP.

Genetic Parameters of Growth Traits and Quantitative Genetic Metrics for Selection and Conservation of Mecheri Sheep of Tamil Nadu.

Determining the genetic and non-genetic sources of variation in a breed is vital for the formulation of strategies for its conservation and improvement. The present study was aimed at estimating the (co)variance components and genetic parameters of Mecheri sheep by fitting six different animal models in the restricted maximum likelihood method, with a preliminary investigation on the performance of animals for non-genetic sources of variation. A total of 2616 lambs were studied, and varying levels of significance were found for the effects of period, season, parity of dam, and birth type on different body-weight traits. Direct heritability estimates derived from the best animal model for body weight at birth, three months, six months, nine months, and twelve months were 0.21, 0.24, 0.10, 0.15, and 0.09, respectively, and the maternal heritability of the corresponding traits was 0.12, 0.05, 0.04, 0.04, and 0.04, respectively. The genetic correlations between the body-weight traits were all positive and moderate-to-strong, except for the correlation between birth weight and the other body-weight traits. The significance of non-genetic factors studied in this work demanded a correction to improve the accuracy of the direct selection of lambs for body-weight traits. The estimated genetic parameters identified the weaning weight as a selection criterion for the improvement in body weight of Mecheri lambs at different ages. Inbred individuals accounted for approximately 13% of the total population in the Mecheri sheep population studied. There were 877 founders in the population, and the actual effective population size was 128.48. The population's mean generation interval was 3.26. The mean inbreeding values ranged from 0.005 to 0.010 across generations. The population's average relatedness ranged from 0.001 to 0.014 across generations. Individual inbreeding was found to be 0.45 per cent for the entire population and 3.4 per cent for the inbred population.

On some pitfalls of the log-linear modeling framework for capture-recapture studies in disease surveillance

Abstract In epidemiological studies, the capture-recapture (CRC) method is a powerful tool that can be used to estimate the number of diseased cases or potentially disease prevalence based on data from overlapping surveillance systems. Estimators derived from log-linear models are widely applied by epidemiologists when analyzing CRC data. The popularity of the log-linear model framework is largely associated with its accessibility and the fact that interaction terms can allow for certain types of dependency among data streams. In this work, we shed new light on significant pitfalls associated with the log-linear model framework in the context of CRC using real data examples and simulation studies. First, we demonstrate that the log-linear model paradigm is highly exclusionary. That is, it can exclude, by design, many possible estimates that are potentially consistent with the observed data. Second, we clarify the ways in which regularly used model selection metrics (e.g., information criteria) are fundamentally deceiving in the effort to select a “best” model in this setting. By focusing attention on these important cautionary points and on the fundamental untestable dependency assumption made when fitting a log-linear model to CRC data, we hope to improve the quality of and transparency associated with subsequent surveillance-based CRC estimates of case counts.

Explainable Data-Driven Method Combined with Bayesian Filtering for Remaining Useful Lifetime Prediction of Aircraft Engines Using NASA CMAPSS Datasets

An aircraft engine is expected to have a high-reliability system as a safety-critical asset. A scheduled maintenance strategy based on statistical calculation has been employed as the current practice to achieve the reliability requirement. Any improvement to this maintenance interval is made after significant reliability issues arise (such as flight delays and high component removals). Several publications and research studies have been conducted related to this issue, one of them involves performing simulations and providing aircraft operation datasets. The recently published NASA CMAPPS datasets have been utilised in this paper since they simulate flight data recording from various measurements. A prognostics model can be developed by analysing these datasets and predicting the engine’s reliability before failure. However, the state-of-the-art prognostics techniques published in the literature using these NASA CMAPPS datasets are mainly purely data-driven. These techniques mainly deal with a “black box” process which does not include uncertainty quantification (UQ). These two factors are barriers to prognostics applications, particularly in the aviation industry. To tackle these issues, this paper aims at developing explainable and transparent algorithms and a software tool to compute the engine health, estimate engine end of life (EoL), and eventually predict its remaining useful life (RUL). The proposed algorithms use hybrid metrics for feature selection, employ logistic regression for health index estimation, and unscented Kalman filter (UKF) to update the prognostics model for predicting the RUL in a recursive fashion. Among the available datasets, dataset 02 is chosen because it has been widely used and is an ideal candidate for result comparison and dataset 03 is employed as a new state-of-the-art. As a result, the proposed algorithms yield 34.5–55.6% better performance in terms of the root mean squared error (RMSE) compared with the previous work. More importantly, the proposed method is transparent and it quantifies the uncertainty during the prediction process.

How complex is professional academic writing? A corpus-based analysis of research articles in 'hard' and 'soft' disciplines

This study focuses on the analysis of linguistic complexity in professional academic writing in light of the empirical evidence provided by a 1,597,000-word corpus of ‘hard’ (life and physical sciences) and ‘soft’ (arts and social) scientific research articles published in leading peer-review journals. Specifically, this investigation aims both to describe the complexity features of texts written by professional authors and to test the hypothesis that linguistic complexity varies across disciplines. Since previous studies have revealed that automatic complexity indices do not sufficiently succeed in providing a comprehensive description of complexity of texts, in this paper complexity has been measured in two ways: quantitatively through the indexes provided by Lu’s (2010) L2 Syntactic Complexity Analyser, and through the more qualitative analysis of a selection of metrics associated with clausal and phrasal complexity in seminal studies. The data show, first, that syntactic complexity indices (basically, strategies of coordination and subordination) are statistically relevant to the characterisation of specifically the soft-science disciplines; second, that there is a continuum across subdisciplines within the broad distinction of soft versus hard genres; and, third, that the soft genre demonstrates a more stable productivity of clausal-complexity strategies, while phrasal-complexity features are more pervasive in the hard-science subcorpus.

Heterogeneity in choice experiment data: A Bayesian investigation

Discrete mixture (DM) models recognize the presence of heterogeneity across individuals in a given population. In the context of a public land use discrete choice experiment, we use DM models to allow for respondent behavior to probabilistically mix over multiple competing process heuristics. We pairwise combine the Random Utility Model (RUM), Contextual Concavity Model (CCM), and Random Regret Minimization (RRM) heuristic into three DM models, in which the probability of an individual adhering to a particular heuristic is modeled as a function of sociodemographic characteristics. We present a comprehensive Bayesian analysis for which we explicitly describe prior selection, inferential procedures, and model comparison metrics. We use a fully Bayesian information criterion to rank the models. We find evidence that responses are best modeled using random regret. After accounting for preference heterogeneity, the DM models estimate two latent groups of decision makers. For the DM models, we develop a novel algorithm to calculate posterior-weighted willingness to pay estimates for improvements in different public park amenities in Polk County, Iowa.

Cut Selection in MILP based on Supervised Learning

With the increasing popularity of deep learning techniques, there has been a growing interest in combining learning methods with Mixed-Integer Linear Programming (MILP) solving processes. A promising approach is to incorporate the learning model as a module in traditional methods. Cut selection is a fundamental subroutine in modern MILP solvers used to select a subset of generated cuts and enhance solver performance. In this work, we present a supervised learning framework to improve the effectiveness of cut selection. Cut selection scoring rules are typically weighted sums of different metrics, and we have developed weighted cut selection metrics based on Machine Learning (ML) techniques for different problems. We propose a novel Neural Network (NN) architecture that incorporates a Graph Convolutional Neural Network (GCN) with a self-attention mechanism to determine appropriate weights. The resulting model serves as a component of the solver and is evaluated through controlled experiments on real-world MILPs. The numerical results demonstrate that our approach outperforms the standard SCIP cut selection strategy, especially on datasets containing the same class of problems.

Low-complexity energy-aware sensor selection for noise reduction in distributed microphone networks

Noise reduction (NR) is a necessary front-end in many audio applications for improving signal quality. It was shown that sparsity-promoting sensor selection potentially makes a trade-off between energy consumption and NR performance, which is rather important for large-scale wireless acoustic sensor networks (WASNs), where many sensors contribute negligibly to NR but energy consumption affects the lifetime of WASNs. This paper presents a sensor selection approach for beamforming-based NR by minimizing the total energy consumption and constraining the output noise variance. Motivated by the optimal semi-definite programming (SDP) solution and the utility-based method, we propose three low-complexity selection metrics: weighted utility, gradient, and weighted input signal-to-noise ratio (SNR). It is shown that the proposed weighted utility and gradient-based methods are near-optimal in performance but much faster than the SDP-based method, and the weighted SNR method has the lowest time complexity with a tiny performance sacrifice. Numerical results using a simulated WASN validate the superiority of the proposed approaches over conventional methods.

Joint Learning-based Heterogeneous Graph Attention Network for Timeline Summarization

Timeline summarization (TLS) is defined as a task for summarizing events in chronological order, which gives readers a comprehensive understanding of an evolutionary story. Previous studies on the timeline summarization (TLS) task ignored the information interaction between sentences and dates, and adopted pre-defined unlearnable representations for them, which significantly degrade the performance. They also considered date selection and event detection as two independent tasks, which makes it impossible to integrate their advantages and obtain a globally optimal summary. In this paper, we present a {joint learning-based heterogeneous graph attention network for TLS (HeterTls), in which date selection and event detection are combined into a unified framework to improve the extraction accuracy and remove redundant sentences simultaneously. Our heterogeneous graph involves multiple types of nodes, the representations of which are iteratively learned across the heterogeneous graph attention layer. We evaluated our model on four datasets, and found that it significantly outperformed the current state-of-the-art baselines with regard to ROUGE scores and date selection metrics.

B-AMA: A Python-coded protocol to enhance the application of data-driven models in hydrology

In this manuscript, we present B-AMA (Basic dAta-driven Models for All), an easy, flexible, fully coded Python-written protocol for the application of data-driven models (DDM) in hydrology. The protocol, which is open source and freely available for academic and non-commercial purposes, has been realized to allow early career scientists, with a basic background in programming, to develop DDM ensuring that no stones are left unturned through their implementation. B-AMA embeds data splitting, feature selection, hyperparameter optimization, and performance metrics. A Jupyter notebook with a practical workflow is available to guide the users through the protocol employment, while visualization tools allow efficient investigation and communication of results. We tested B-AMA across four hydrological applications to explore DDM applicability across temporal resolutions, time series lengths, and autocorrelations. B-AMA showed great accuracy and reasonable computational time, making the protocol ideal for educational purposes and for the development of DDM-based forecasts of hydrological time-series.

ERGCN: Data enhancement-based robust graph convolutional network against adversarial attacks

With recent advancements, graph neural networks (GNNs) have shown considerable potential for various graph-related tasks, and their applications have gained considerable attention. However, adversarial attacks can significantly degrade the performance of GNNs, hindering their deployment in critical real-world tasks. GNNs must be robust against adversarial attacks, in which imperceptible adversarial perturbations are introduced to induce serious security issues. To achieve this goal, we propose a robust graph convolutional network, ERGCN, for node classification via data enhancement. ERGCN simultaneously utilizes properties from the “data domain” and “model space” as guidance. Based on the feature smoothness assumption, a graph structure enhancement (GSE) mechanism is proposed to improve the structural reliability of input graphs. Moreover, inspired by self-training methods that assign pseudo-labels to unlabeled training samples and use them to optimize the target model iteratively, a reliable node selection metric, model boundary distance (MBD), is defined based on the distance from training samples to model decision boundary. Finally, a self-training-based robust graph convolutional network is proposed for node classification. Extensive experiments on three public datasets demonstrate the superiority of our model over existing state-of-the-art methods. Our study provides a solution for trustworthy graph machine learning systems in adversarial environments. The code is available at https://github.com/star4455/ERGCN.