Reliability Metrics Research Articles

Adherence to Insulin Therapy Among Children with Type 1 Diabetes: Reliability and Validity of the Arabic Version of the 4-Item Morisky Medication Adherence Scale

BackgroundInadequate adherence to insulin is a major concern, necessitating the use of reliable and valid metrics for assessing adherence. Up to date, there are no Arabic validated tools assessing adherence to insulin therapy among children with type 1 diabetes (T1DM). Thus, the aim of this study is to evaluate the psychometric properties of an Arabic version of the four-item Morisky Green Levine Medication Adherence Scale (MGLS-4) as a self-reported measure of adherence to insulin among a cohort of Egyptian children with T1DM.MethodsThe MGLS-4 was translated using forward and backward translation. The Cronbach’s alpha was used to assess reliability. Criterion validity of the scale was tested by examining the correlation coefficients between the compliance score (level of adherence) and the HbA1c levels.ResultsA total of 400 patients completed the Arabic version of MGLS-4. 26.25% of the studied cohort was found to be non-adherent to insulin therapy; non-adherent patients were significantly older (P=0.001). Decreased maternal education level, decreased frequency of blood glucose monitoring and prolonged disease duration best predicted the occurrence of non-adherence among the studied cohort. The internal consistency of the current version showed good reliability (Cronbach’s alpha = 0.857). The adherence score and adherence level showed very strong correlation with HbA1c level (rho = 0.830, P < 0.001 and rho = 0.808, P < 0.001, respectively).ConclusionThe Arabic version of MGLS-4 showed good reliability and validity as a self-administered tool for assessing adherence to insulin in pediatric patients with T1DM.

Mission Effective Capacity—A Novel Dependability Metric: A Study Case of Multiconnectivity-Enabled URLLC for IIoT

Various industrial Internet of Things applications demand execution periods throughout which no communication failure is tolerated. However, the classical understanding of reliability in the context of ultra-reliable low-latency communication (URLLC) does not reflect on the time-varying characteristics of the wireless channel. In this article, we introduce a novel mission reliability and mission effective capacity metric that takes these phenomena medium into account, while specifically studying multiconnectivity (MC)-enabled industrial radio systems. We assume uplink short packet transmission with no channel state information at URLLC user (the transmitter) and sporadic traffic arrival. Moreover, we leverage the existing framework of dependability theory and provide closed-form expressions (CFEs) for the mission reliability of the MC system using the maximal-ratio combining scheme. We do so by utilizing the mean time to first failure, which is the expected time of failure occurring for the first time. Moreover, we also derive exact CFEs for second-order statistics, such as level crossing rate and average fade duration, showing how fades are distributed in fading channels with respect to time. Furthermore, the design throughput maximization problem under the mission reliability constraint is solved numerically through the cross-entropy method.

Validity of Off-the-Shelf Automatic Speech Recognition for Assessing Speech Intelligibility and Speech Severity in Speakers With Amyotrophic Lateral Sclerosis.

There is increasing interest in using automatic speech recognition (ASR) systems to evaluate impairment severity or speech intelligibility in speakers with dysarthria. We assessed the clinical validity of one currently available off-the-shelf (OTS) ASR system (i.e., a Google Cloud ASR API) for indexing sentence-level speech intelligibility and impairment severity in individuals with amyotrophic lateral sclerosis (ALS), and we provided guidance for potential users of such systems in research and clinic. Using speech samples collected from 52 individuals with ALS and 20 healthy control speakers, we compared word recognition rate (WRR) from the commercially available Google Cloud ASR API (Machine WRR) to clinician-provided judgments of impairment severity, as well as sentence intelligibility (Human WRR). We assessed the internal reliability of Machine and Human WRR by comparing the standard deviation of WRR across sentences to the minimally detectable change (MDC), a clinical benchmark that indicates whether results are within measurement error. We also evaluated Machine and Human WRR diagnostic accuracy for classifying speakers into clinically established categories. Human WRR achieved better accuracy than Machine WRR when indexing speech severity, and, although related, Human and Machine WRR were not strongly correlated. When the speech signal was mixed with noise (noise-augmented ASR) to reduce a ceiling effect, Machine WRR performance improved. Internal reliability metrics were worse for Machine than Human WRR, particularly for typical and mildly impaired severity groups, although sentence length significantly impacted both Machine and Human WRRs. Results indicated that the OTS ASR system was inadequate for early detection of speech impairment and grading overall speech severity. While Machine and Human WRR were correlated, ASR should not be used as a one-to-one proxy for transcription speech intelligibility or clinician severity ratings. Overall, findings suggested that the tested OTS ASR system, Google Cloud ASR, has limited utility for grading clinical speech impairment in speakers with ALS.

High Volume Freeway Travel Time Reliability and the COVID-19 Pandemic

This paper examines the effect of the COVID-19 pandemic on travel time reliability (TTR) performance on two high traffic volume freeway corridors (I-90 and I-405) in the U.S.A. Specifically, the travel time distributions (TTDs) during and before the pandemic are compared. The paper also examines which TTR metrics best capture the effect of the pandemic on reliability performance. There were statistically significant differences, at the 95% confidence level, between TTD in 2020 and corresponding TTDs in 2018 and 2019. Not surprisingly, all measures of central tendency and all measures of dispersions were reduced during the pandemic. Consequently, it was concluded that TTR performance improved during the pandemic regardless of what TTR definition was used. Travel time index (TTI), planning time index (PTI), and the level of travel time reliability (LOTTR) metrics improved during the pandemic, albeit at different rates. In contrast, the buffer index and coefficient of variation increased. In other words, whether an analyst would identify that TTR improved or decreased during the pandemic, and by how much, would be a function of which TTR metric was applied. Not surprisingly, the more congested the roadway section, the greater the impact the pandemic-related interventions had on TTR. It was found that, practically speaking, there are no differences in TTI, PTI, or LOTTR values when TTDs are formulated using a 5 min or 15 min aggregation interval. It was concluded that analysts need to have a deep understanding of the underlying TTD and the various TTR definitions when evaluating changes in freeway systems’ TTR.

Measuring the Effect of Using a Borderline Students' Characteristics Model on Reliability of Objective Structured Clinical Examination.

Defining borderline group is a crucial step when applying standard setting methods in objective structured clinical examination (OSCE). However, it is the most challenging and demanding task. This study aimed to measure the effect of using a model describing characteristics of borderline students on the reliability and metrics of OSCE. This model was adopted from a qualitative study based on conducted semi-structured interviews with experienced raters. The model identifies several themes categorized under four items which are gathering patient information, examining patients, communicating with patients, and general personal characteristics. In the current study, two groups of evaluators were investigated: one as the experimental group that received orientation about the used model and the other as the control group that did not receive any orientation. We applied the model in two mirrored OSCE circuits. Using the model enhanced raters' global rating. Consequently, the cut scores between the two OSCE circuits were different, and the examination reliability and quality metrics were improved.

Belief reliability analysis of multi-state deteriorating systems under epistemic uncertainty

Due to the growing complexity of multi-state deteriorating systems and the lack of data, information and knowledge, epistemic uncertainty cannot be ignored anymore. Although there already exist some reliability metrics to deal with epistemic uncertainty problems in multi-state deteriorating systems, the final results provided by these methods are always probability intervals, which will suffer from the interval extension challenge in the subsequent operation. In this paper, we propose a reliability analysis method for the multi-state deteriorating system based on belief reliability theory, which can provide more conservative estimations and avoid interval extension problems. To describe the state transition process of multi-state deteriorating systems under the effect of epistemic uncertainty, a mathematical model called uncertain state transition chain is proposed. Under the framework of uncertainty theory, a total uncertainty inequality is given to assist the calculation of belief reliability. Using the proposed model and inequality, an algorithm of belief reliability for multi-state deteriorating systems considering the uncertainty of inspection time is provided. As a numerical example, the calculating process, the evaluation result and some discussions of a specific deteriorating system are given at last.

Proving and improving the reliability of infant research with neuroadaptive Bayesian optimization

AbstractThe field of infant research is not immune from the reproducibility crisis in cognitive science and psychology. In their recent methodological article, Byers‐Heinlein et al. (2021) invited infant researchers to commit to produce robust findings by reporting reliability metrics for their variables of interest, improving data quality and quantity, and moving towards more sophisticated paradigms and analyses. We present a novel artificial intelligence‐enriched individualized approach that, in our view, is particularly promising to shed new light on infant and child development and promote good research practice in the field; neuroadaptive Bayesian optimization (NBO). NBO is a transformative method where the collected brain or behavioural data are processed in real time and used to identify the stimuli that maximize the individual's response. Applying NBO to infant research goes in the direction proposed by Byers‐Heinlein et al. (2021) and further, the method requires careful a priori choices that effectively correspond to preregistering the experimental design and analytic pipeline. In this commentary, we examine how the NBO approach embeds the six proposed solutions for more reliable infant research, encouraging transparency of the planned analyses and robustness of findings.

Measuring the reliability of SDG 7: the reasons, timing, and fairness of outage distribution for household electricity access solutions

The United Nations identifies ensuring ‘access to affordable, reliable, sustainable and modern energy for all’ as one of its Sustainable Development Goals for 2030. This article focuses on the comparatively under-investigated question of reliability within the broader goal. We empirically study experienced household electricity reliability using common frameworks in key countries such as Tanzania, Kenya, and India. Datasets represent a diverse set of technologies including solar home systems (SHS), solar pico-grids, and national electricity grids. First, the prevailing reliability metrics—SAIDI and SAIFI—are measured for all datasets. Informed by critical assessments, this article then proposes a suite of new metrics that facilitate improved reliability comparisons by considering the reasons, timing, and fairness of outage distribution. Analyses using our proposed metrics reveal key policy implications for addressing energy poverty in the Global South. Acknowledging that the systems studied provide different capacity, affordability, and carbon footprints, we find that on average, SHS provided comparable hours of lighting to local grid connections, however SHS outages were less equally distributed than those from other sources. In addition, calculations of grid reliability were highly sensitive to measurement techniques and assumptions used, necessitating high resolution data for policy decisions. Finally, economically driven outages conspicuous in pre-paid SHS systems (i.e., disconnections for non-payment) composed a significant portion of experienced unreliability. These findings quantify the important contribution of demand-side affordability to experienced household reliability, thereby allowing for a comprehensive understanding of the reliability of SDG 7.

An integrative approach to neighbourhood sustainability assessments using publicly available traffic data

Travel time data and variability of travel times can be used to quantify the network performance as an important aspect of sustainable development in modern cities. We introduce a data collection and analysis framework to be employed as an integral part of a neighbourhood sustainability assessment (NSA). The proposed framework performs a targeted selection and evaluation of indicators based on publicly available traffic data, specifically addressing short-term observations of travel times and traffic counts. Entailing a hybrid and modular approach, we demonstrate the applicability of travel time reliability measures by combining different datasets, various variability estimators, and visualisation methods, to advance the interpretation strength of the assessment. Implemented are regression methods based on cosinor models, which allow us to analyse the rhythmic behaviour of travel time trends. These models can be applied to assess the differential rhythmicity among different factors, like route, weather conditions and type of day (i.e., weekend or workday). Further, the same datasets are used to evaluate two reliability metrics, i.e., a travel time index and a planning time index. We demonstrate the application of the suggested framework on several strategic routes associated with three testbed neighbourhoods in Ljubljana, Slovenia. The results show that average travel times and their variations are affected by route, day of the week, and weather conditions, and correlate with vehicles count data. Analyses of rhythmicity give straightforward insights into travel time trends, appropriate for comparative analysis and thus well-suited for use in NSA and monitoring practices. To the best of our knowledge, this work presents the first study quantifying reliability indicators intended for integration with NSA standards.

EEPR-OLSR: an Energy Efficient and Path Reliability protocol for proactive MANET routing

Routing in Mobile Ad-hoc Networks has received a lot of attention due to the challenges posed by the self-organizing nature of the network, the dynamic topology, and the unreliable wireless medium. One of the most critical issues for MANETs is how to increase network lifetime, since nodes are typically battery powered. In this paper we consider the proactive MANET protocol OLSR to improve the network lifetime, we propose a novel multiple metric routing scheme for MANET, based on energy efficient and path reliability metrics, integrating it to standard OLSR, named Energy Efficient and Path Reliability OLSR (EEPR-OLSR), in which we investigate cross layer parameters that effect the network lifetime and a prediction-based link availability estimation is introduced. Simulation results, by NS3 simulator, show that the proposed EEPR-OLSR provides significant performance gains in both the network lifetime and packet delivery ration (PDR), compared to the standard OLSR.

Pre-Operative Medications as a Predictor for Post-Operative Complications Following Geriatric Hip Fracture Surgery.

BackgroundFragility hip fractures are a common orthopedic injury seen in Emergency Departments, with variable outcomes that can range from average to devastating. Currently, few reliable metrics to predict which patients will suffer post-operative complications exist. The aim of this study was to determine if the number and type of pre-operative medications can help predict post-operative complications.MethodsA prospectively collected database of hip fracture patients was retrospectively reviewed. Patients with isolated greater trochanteric fractures, periprosthetic fractures, or re-fractures were excluded. Pre-operative baseline characteristics as well as number and type of post-operative complications were reviewed. Any complication within 6 months of surgery and complications that could be directly attributable to the surgical procedure within 2 years of surgery were examined. Major complications (return to the operating room, deep infection, pulmonary, cardiac, and hematologic) and minor medical complications were assessed. A multivariate regression model was performed to identify independent risk factors.ResultsThree-hundred ninety-one patients were included. A majority were aged 80–90 and female, and lived at home prior to presentation. Overall, 33.7% of patients suffered a complication within a 2-year follow-up period. Mortality rates were 5.4%, 10.0%, and 14.9% over 30 days, 1 year, and 2 years, respectively. After assessing this relationship while controlling for age, sex, injury type, pre-operative residence, ambulatory status, ASA score, and CCI score, the relationship remained significant for both an increased number of complications (P = .048) and a higher likelihood of having a complication (P = .008). Cardiovascular (P = .003), pulmonary (P = .001), gout (P = .002), or diabetes (P = .042) medications were associated with a higher likelihood for experiencing a complication.ConclusionsOur study suggests that there is a strong and linear relationship between the number and type of pre-operative medications taken and risk of post-operative complications. This exists for up to 8 medications, at which point further increase does not contribute to an increased risk of complication. This relationship exists even after controlling for confounding variables and can be used by surgeons to better counsel patients and families regarding their specific risk for suffering perioperative complications.

Diagnostic accuracy and reliability of Triangular Fibrocartilage Complex (TFCC) provocation tests: a systematic review

Provocation tests are routinely used in the diagnosis of Triangular Fibrocartilage Complex (TFCC) injuries by medical practitioners including physiotherapists and hand therapists. The accuracy and reliability of these tests are essential to ensure a correct diagnosis of injury. Despite their extensive clinical use, the current evidence on these tests is limited. The purpose of this review was to evaluate studies that examined accuracy and reliability of provocation tests for TFCC injuries. A systematic search was undertaken across six databases. Studies were evaluated using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool, and the Quality Appraisal of Reliability Studies (QAREL) scale. Reliability and diagnostic accuracy metrics including reliability, sensitivity specificity, likelihood ratios and predictive values were extracted. A total of 13 eligible studies were retrieved. Of this, five studies investigated reliability; this included three different provocation tests assessing both inter-rater and intra-rater reliability. Eight studies investigated diagnostic accuracy, which included five different TFCC provocation tests. The majority of the diagnostic accuracy studies showed a lower risk of bias, and the reliability studies were of moderate to high quality. This review indicated that a small number of studies showed high reliability, which indicated consistency of the test both between different raters, and between the same rater at different times. However, reported estimates of accuracy indicated limited diagnostic utility. Only a few studies reached acceptable threshold values to determine its clinical usefulness, and there were large inconsistencies of findings between studies that investigated the same test. In this presentation, I will discuss the findings of this systematic review including the usefulness of individual provocation tests for the TFCC, the recommendations and application for clinical practise, and direction needed for future research.

Statistical Deep Parsing for Spanish: Abridged Version

This document presents the development of a statistical HPSG parser for Spanish. HPSG is a deep linguistic formalism that combines syntactic and semantic information in the same representation, and is capable of elegantly modeling many linguistic phenomena. We describe the HPSG grammar adapted to Spanish we designed and the construction of our corpus. Then we present the different parsing algorithms we implemented for our corpus and grammar: a bottom-up strategy, a CKY with supertagger approach, and a LSTM top-down approach. We then show the experimental results obtained by our parsers compared among themselves and also to other external Spanish parsers for some global metrics and for some particular phenomena we wanted to test. The LSTM top-down approach was the strategy that obtained the best results on most of the metrics (for our parsers and external parsers as well), including constituency metrics (87.57 unlabeled F1, 82.06 labeled F1), dependency metrics (91.32 UAS, 88.96 LAS), and SRL (87.68 unlabeled, 80.66 labeled), and most of the particular phenomenon metrics such as clitics reduplication, relative referents detection and coordination chain identification.

IDF21-0386 DFU Track: Building a novel, Artificial Intelligence-based method to monitor wound healing

Background: While Diabetic Foot Ulcer (DFU) prevention and treatment have improved dramatically, assessment of DFU healing remains outdated, presenting a problem for research and clinical practice. Recent new mobile applications for standardizing DFU images have been reported, however, they are not automated and require large manual efforts. Aim: This study aims to define a reliable and sensitive metric by using machine learning to quantify wound severity, taking into account its size, shape, ischaemic and infection severity using a standard 2D photograph. This single metric could be used to identify and triage non-healing DFUs, as well as track wound healing in response to treatment. Method: Serial photographs of DFUs in 28 patients (90% T2DM, mean age 63, 83% male) with varying severity of DFU (72% UT 1A-C), were collected and analysed. Images from sequential clinic visits were automatically aligned. Wounds were automatically localized and segmented. Results: We have characterized DFUs and quantified changes in DFU characteristics across visits. 91% of baseline images were correctly identified and localized by the artificial intelligence (AI)-based wound localisation method. Discussion: If successful, this measure will facilitate targeting of appropriate therapies to individual patients, improving healing outcomes and therefore resulting in fewer amputations.

Performance Evaluation of a Smart Bed Technology against Polysomnography

The Sleep Number smart bed uses embedded ballistocardiography, together with network connectivity, signal processing, and machine learning, to detect heart rate (HR), breathing rate (BR), and sleep vs. wake states. This study evaluated the performance of the smart bed relative to polysomnography (PSG) in estimating epoch-by-epoch HR, BR, sleep vs. wake, mean overnight HR and BR, and summary sleep variables. Forty-five participants (aged 22–64 years; 55% women) slept one night on the smart bed with standard PSG. Smart bed data were compared to PSG by Bland–Altman analysis and Pearson correlation for epoch-by-epoch HR and epoch-by-epoch BR. Agreement in sleep vs. wake classification was quantified using Cohen’s kappa, ROC analysis, sensitivity, specificity, accuracy, and precision. Epoch-by-epoch HR and BR were highly correlated with PSG (HR: r = 0.81, |bias| = 0.23 beats/min; BR: r = 0.71, |bias| = 0.08 breaths/min), as were estimations of mean overnight HR and BR (HR: r = 0.94, |bias| = 0.15 beats/min; BR: r = 0.96, |bias| = 0.09 breaths/min). Calculated agreement for sleep vs. wake detection included kappa (prevalence and bias-adjusted) = 0.74 ± 0.11, AUC = 0.86, sensitivity = 0.94 ± 0.05, specificity = 0.48 ± 0.18, accuracy = 0.86 ± 0.11, and precision = 0.90 ± 0.06. For all-night summary variables, agreement was moderate to strong. Overall, the findings suggest that the Sleep Number smart bed may provide reliable metrics to unobtrusively characterize human sleep under real life-conditions.

Determining Sport Concussion Assessment Tool fifth Edition (SCAT5) reliable change in male professional hockey players

ObjectivesThe purpose of this paper is to provide reliable change metrics and assess the test–retest psychometrics of the Sport Concussion Assessment Tool fifth Edition (SCAT5) in a multilingual sample of...

Adjusting the aim of capacity mechanisms: Future-proof reliability metrics and firm supply calculations

Capacity mechanisms are now deemed a regulatory mainstay in liberalised power system decarbonisation. These instruments aim to ensure sufficient resource adequacy with a mix able to meet the reliability target set by the regulator. Remuneration in capacity mechanisms depends on so-called firm supply (calculated from de-rating factors or capacity credits), taken as a proxy for each resource’s expected long-term contribution to system adequacy. Most adequacy assessment and de-rating methods used to calculate security of supply were developed for power systems very different from today’s and tomorrow’s, in which renewables account for a higher share of the mix and demand is more elastic. Regulators the world over are already revising these methods, although that seldom involves an overall rethink of their general approach. Drawing from theoretical considerations and international best practice, this article defines an updated theoretical framework for the resource adequacy problem against the backdrop of the challenges ahead for the power sector. The conclusions include recommendations for resilient reliability metrics and de-rating calculation methods.

Dry Eye Symptom Severity and Visual Field Reliability Metrics.

Tracking failure frequency (TFF) increases with dry eye symptom severity and in the left eye. Symptoms of dry eye disease are commonly encountered in glaucoma patients and can be exacerbated by topical glaucoma medications. Dry eye disease may influence the reliability of visual field (VF) tests and impact the accurate interpretation of the results. Patients at the Veterans Administration Medical Center San Diego completed the 5-item Dry Eye Questionnaire before VF testing between December 2018 and February 2019. VF reliability metrics were recorded for each patient. Standard reliability metrics included fixation losses, false positive, and false negative rates. Gaze tracking (GT) metrics included percent of stimuli with gaze deviations between 1 and 2 degrees, 3 and 5 degrees, 6 degrees or greater, and percent of stimuli with tracking failure (TFF). The use of glaucoma medications and artificial tears was also recorded. A total of 494 patients completed the 5-item Dry Eye Questionnaire and VF testing. There was no association between dry eye symptom severity and standard reliability metrics or most GT metrics. However, TFF increased as dry eye symptom severity increased (P=0.015). TFF was also greater in the left eye, which was tested second (P=0.012); no other reliability metrics were related to laterality. Patients were more likely to use artificial tears with increased dry eye symptom severity (P<0.001), but there was no relationship between symptom severity and glaucoma medication use. Dry eye symptom severity may influence the acceptable range or threshold of TFF when using GT metrics to determine VF reliability. Likewise, the acceptable range or threshold for TFF may be different between eyes.

A large-scaled corpus for assessing text readability

This paper introduces the CommonLit Ease of Readability (CLEAR) corpus, which provides unique readability scores for ~ 5000 text excerpts along with information about the excerpt’s year of publishing, genre, and other metadata. The CLEAR corpus will provide researchers interested in discourse processing and reading with a resource from which to develop and test readability metrics and to model text readability. The CLEAR corpus includes a number of improvements in comparison to previous readability corpora including size, breadth of the excerpts available, which cover over 250 years of writing in two different genres, and unique readability criterion provided for each text based on teachers’ ratings of text difficulty for student readers. This paper discusses the development of the corpus and presents reliability metrics for the human ratings of readability.

Pitfalls and Recommended Strategies and Metrics for Suppressing Motion Artifacts in Functional MRI.

In resting-state functional magnetic resonance imaging (rs-fMRI), artefactual signals arising from subject motion can dwarf and obfuscate the neuronal activity signal. Typical motion correction approaches involve the generation of nuisance regressors, which are timeseries of non-brain signals regressed out of the fMRI timeseries to yield putatively artifact-free data. Recent work suggests that concatenating all regressors into a single regression model is more effective than the sequential application of individual regressors, which may reintroduce previously removed artifacts. This work compares 18 motion correction pipelines consisting of head motion, independent components analysis, and non-neuronal physiological signal regressors in sequential or concatenated combinations. The pipelines are evaluated on a dataset of cognitively normal individuals with repeat imaging and on datasets of studies of Autism Spectrum Disorder, Major Depressive Disorder, and Parkinson's Disease. Extensive metrics of motion artifact removal are measured, including resting state network recovery, Quality Control-Functional Connectivity (QC-FC) correlation, distance-dependent artifact, network modularity, and test-retest reliability of multiple rs-fMRI analyses. The results reveal limitations in previously proposed metrics, including the QC-FC correlation and modularity quality, and identify more robust artifact removal metrics. The results also reveal limitations in the concatenated regression approach, which is outperformed by the sequential regression approach in the test-retest reliability metrics. Finally, pipelines are recommended that perform well based on quantitative and qualitative comparisons across multiple datasets and robust metrics. These new insights and recommendations help address the need for effective motion artifact correction to reduce noise and confounds in rs-fMRI.