Classification Accuracy Metrics Research Articles

Deep learning‐based aggregate analysis to identify cut‐off points for decision‐making in pancreatic cancer detection

AbstractThis study addresses the problem of detecting pancreatic cancer by classifying computed tomography (CT) images into cancerous and non‐cancerous classes using the proposed deep learning‐based aggregate analysis framework. The application of deep learning, as a branch of machine learning and artificial intelligence, to specific medical challenges can lead to the early detection of diseases, thus accelerating the process towards timely and effective intervention. The concept of classification is to reasonably select an optimal cut‐off point, which is used as a threshold for evaluating the model results. The choice of this point is key to ensure efficient evaluation of the classification results, which directly affects the diagnostic accuracy. A significant aspect of this research is the incorporation of private CT images from Vilnius University Hospital Santaros Klinikos, combined with publicly available data sets. To investigate the capabilities of the deep learning‐based framework and to maximize pancreatic cancer diagnostic performance, experimental studies were carried out combining data from different sources. Classification accuracy metrics such as the Youden index, (0, 1)‐criterion, Matthew's correlation coefficient, the F1 score, LR+, LR−, balanced accuracy, and g‐mean were used to find the optimal cut‐off point in order to balance sensitivity and specificity. By carefully analyzing and comparing the obtained results, we aim to develop a reliable system that will not only improve the accuracy of pancreatic cancer detection but also have wider application in the early diagnosis of other malignancies.

Fully automated explainable abdominal CT contrast media phase classification using organ segmentation and machine learning.

Contrast-enhanced computed tomography (CECT) provides much more information compared to non-enhanced CT images, especially for the differentiation of malignancies, such as liver carcinomas. Contrast media injection phase information is usually missing on public datasets and not standardized in the clinic even in the same region and language. This is a barrier to effective use of available CECT images in clinical research. The aim of this study is to detect contrast media injection phase from CT images by means of organ segmentation and machine learning algorithms. A total number of 2509 CT images split into four subsets of non-contrast (class #0), arterial (class #1), venous (class #2), and delayed (class #3) after contrast media injection were collected from two CT scanners. Seven organs including the liver, spleen, heart, kidneys, lungs, urinary bladder, and aorta along with body contour masks were generated by pre-trained deep learning algorithms. Subsequently, five first-order statistical features including average, standard deviation, 10, 50, and 90 percentiles extracted from the above-mentioned masks were fed to machine learning models after feature selection and reduction to classify the CT images in one of four above mentioned classes. A 10-fold data split strategy was followed. The performance of our methodology was evaluated in terms of classification accuracy metrics. The best performance was achieved by Boruta feature selection and RF model with average area under the curve of more than 0.999 and accuracy of 0.9936 averaged over four classes and 10 folds. Boruta feature selection selected all predictor features. The lowest classification was observed for class #2 (0.9888), which is already an excellent result. In the 10-fold strategy, only 33 cases from 2509 cases (∼1.4%) were misclassified. The performance over all folds was consistent. We developed a fast, accurate, reliable, and explainable methodology to classify contrast media phases which may be useful in data curation and annotation in big online datasets or local datasets with non-standard or no series description. Our model containing two steps of deep learning and machine learning may help to exploit available datasets more effectively.

Appraisal of EnMAP hyperspectral imagery use in LULC mapping when combined with machine learning pixel-based classifiers

The recent availability of satellite hyperspectral imaging combined with the developments in the classification techniques have paved the way towards improving our ability to obtain information on the spatiotemporal distribution of land use/land cover (LULC) at improved accuracy. In this context, the present study aims at evaluating the combined use of the recently launched Environmental Mapping and Analysis Program (EnMAP) hyperspectral satellite mission with two powerful machine learning (ML) classifiers. In particular, the Support Vector Machines (SVM) and Random Forest (RF) are used synergistically with EnMAP imagery in performing LULC mapping at a typical Mediterranean setting located in northern Greece. Evaluation of the derived LULC maps is based on the computation of a series classification accuracy metrics. The McNemar's chi-square statistical significance testing was also computed to confirm the statistical significance of the differences between the classifiers. In overall, results showed that SVM slightly outperformed RF, exhibiting a higher overall accuracy, of 92.6% and 88.1%, respectively, whereas the statistical significance of the findings was also attested by the McNemar's statistical test results. To our knowledge, this study is one of the first published so far focusing on exploring the capabilities of ENMAP imagery when combined with different ML pixel-based classifiers in the context of LULC mapping. Our results, indeed provided useful insights on the potential of EnMAP datasets in deriving information on the spatiotemporal distribution of LULC at a highly fragmented Mediterranean landscape, evidencing the EnMAP promising potential in this field.

Selection of crop varieties and yield prediction based on phenotype applying deep learning

<div align="center"><span lang="EN-US">In India, agriculture plays an important role in the nation’s gross domestic product (GDP) and is also a part of civilization. Countries’ economies are also influenced by the amount of crop production. All business trading involves farming as a major factor. In order to increase crop production, different technological advancements are developed to acquire the information required for crop production. The proposed work is mainly focused on suitable crop selection across districts in Tamil Nadu, considering phenotype factors such as soil type, climatic factors, cropping season, and crop region. The key objective is to predict the suitable crop for the farmers based on their locations, soil types, and environmental factors. This results in less financial loss and a shorter crop production timeframe. Combined feature selection (CFS)-based machine regression helps increase crop production rates. A brief comparative analysis was also made between various machine learning (ML) regression algorithms, which majorly contributed to the process of crop selection considering phenotype factors. Stacked long short-term memory (LSTM) classifiers outperformed other decision tree (DT), k-nearest neighbor (KNN), and logistic regression (LR) with a prediction accuracy of 93% with the lowest classification accuracy metrics. The proposed method can help us select the perfect crop for maximum yield.</span></div>

Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient.

The accuracy of a classification is fundamental to its interpretation, use and ultimately decision making. Unfortunately, the apparent accuracy assessed can differ greatly from the true accuracy. Mis-estimation of classification accuracy metrics and associated mis-interpretations are often due to variations in prevalence and the use of an imperfect reference standard. The fundamental issues underlying the problems associated with variations in prevalence and reference standard quality are revisited here for binary classifications with particular attention focused on the use of the Matthews correlation coefficient (MCC). A key attribute claimed of the MCC is that a high value can only be attained when the classification performed well on both classes in a binary classification. However, it is shown here that the apparent magnitude of a set of popular accuracy metrics used in fields such as computer science medicine and environmental science (Recall, Precision, Specificity, Negative Predictive Value, J, F1, likelihood ratios and MCC) and one key attribute (prevalence) were all influenced greatly by variations in prevalence and use of an imperfect reference standard. Simulations using realistic values for data quality in applications such as remote sensing showed each metric varied over the range of possible prevalence and at differing levels of reference standard quality. The direction and magnitude of accuracy metric mis-estimation were a function of prevalence and the size and nature of the imperfections in the reference standard. It was evident that the apparent MCC could be substantially under- or over-estimated. Additionally, a high apparent MCC arose from an unquestionably poor classification. As with some other metrics of accuracy, the utility of the MCC may be overstated and apparent values need to be interpreted with caution. Apparent accuracy and prevalence values can be mis-leading and calls for the issues to be recognised and addressed should be heeded.

Improving Collaborative Filter Using BERT

With the increasing number of books published and the difficulty of obtaining appropriate research attention, the recommendation systems can increase the affordability and availability of these books. In this work, we expand our work to enhance the accuracy of book collaborative filtering by applying semantic similarity to book summaries, in addition to that addressing major problems of the current work by applying effective techniques to handle the scalability and sparsity problems. The proposed approach consists of three stages: preprocessing, building the system, and evaluation. The technologies used in the pre-processing stage included reduction and normalization. The construction system is divided into two phases: semantic similarity and recommendation. The semantic similarity is done by using BERT for sentence embedding and cosine similarity to calculate the similarity between sentences. During the recommendation phase by using CF based on KNN. In the evaluation stage, classification accuracy metrics had been used. The proposed approach improved the accuracy of the book recommendation system and increased the accuracy to 0.89 compared to previous works on a dataset of 271,000 book summaries. The proposed approach yielded better results due to avoiding problems in previous work, such as scalability and sparsity, by using BERT with CF based KNN. Filtering the data using BERT and the KNN algorithm in the CF added strength to the recommendation, which led to an increase in the accuracy rate.

Cadence (steps/min) as an indicator of the walk-to-run transition

BackgroundHumans naturally transition from walking to running at a point known as the walk-to-run transition (WRT). The WRT commonly occurs at a speed of ∼2.1 m/s (m/s) or a Froude number (dimensionless value considering leg length) of 0.5. Emerging evidence suggests the WRT can also be classified using a cadence of 140 steps/min. An accurate cadence-based WRT metric would aid in classifying wearable technology minute-level step metrics as walking vs. running. PurposeTo evaluate performance of 1) WRT predictors directly identified from a treadmill-based dataset of sequentially faster bouts, and 2) accepted WRT predictors compiled from previous literature. MethodsTwenty-eight adults (71.4% men; age = 36.6 ± 12.8 years, BMI = 26.2 ± 4.7 kg/m2) completed a series of five-minute treadmill walking bouts increasing in 0.2 m/s increments until they freely chose to run. Optimal WRT values for speed, Froude number, and cadence were identified using receiver operating characteristic (ROC) curve analyses. WRT value performance was evaluated via classification accuracy metrics. ResultsOverall accuracies (metric, percent) according to WRT predictors from previous literature were: speed (2.1 m/s, 55.0%), Froude number (0.5, 76.8%), and cadence (140 steps/min, 91.1%), and those from the dataset herein were: speed (1.9 and 2.0 m/s, 78.6%), Froude number (0.68, 77.3%), and cadence (134, 139, and 141 steps/min, 92.9%). The three equally accurate cadence values support a heuristic range of cadence-based WRT values in young and middle-aged adults: 135–140 steps/min. SignificanceA tight range of cadence values performed better as WRT predictors compared to either previously reported or directly identified speed or Froude number values. These findings have important implications for gait classification, especially considering cadence is a simple metric which can be readily assessed across settings using direct observation or wearable technologies.

Integrating and optimizing genomic, weather, and secondary trait data for multiclass classification.

Modern plant breeding programs collect several data types such as weather, images, and secondary or associated traits besides the main trait (e.g., grain yield). Genomic data is high-dimensional and often over-crowds smaller data types when naively combined to explain the response variable. There is a need to develop methods able to effectively combine different data types of differing sizes to improve predictions. Additionally, in the face of changing climate conditions, there is a need to develop methods able to effectively combine weather information with genotype data to predict the performance of lines better. In this work, we develop a novel three-stage classifier to predict multi-class traits by combining three data types-genomic, weather, and secondary trait. The method addressed various challenges in this problem, such as confounding, differing sizes of data types, and threshold optimization. The method was examined in different settings, including binary and multi-class responses, various penalization schemes, and class balances. Then, our method was compared to standard machine learning methods such as random forests and support vector machines using various classification accuracy metrics and using model size to evaluate the sparsity of the model. The results showed that our method performed similarly to or better than machine learning methods across various settings. More importantly, the classifiers obtained were highly sparse, allowing for a straightforward interpretation of relationships between the response and the selected predictors.

Detecting and reducing heterogeneity of error in acoustic classification

Abstract Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F‐score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.

Classification of moderate-intensity overground walking speed in 21- to 85-year-old adults

ABSTRACT The Compendium of Physical Activities reports that walking at 2.5 mph associates with absolutely-defined moderate intensity (i.e., ≥3 metabolic equivalents [METs]). However, it is unclear whether this speed threshold is accurate during overground walking and/or across the adult age-span. This study aimed to identify optimal and heuristic speed thresholds associated with 3 METs during overground walking across age groups. Healthy adults (n = 248, 21–85 years old, 49% women) performed a 5-minute self-paced overground walking trial. Speed was measured using an electronic gait mat, and oxygen uptake was measured using indirect calorimetry and converted to METs. Optimal and heuristic thresholds and classification accuracy metrics were determined and compared using ROC curve analyses. Speed thresholds (95% CIs) associated with 3 METs for the whole sample, young (21–40 years), middle-aged (41–60 years) and older-aged (61–85 years) groups were 1.29 (1.25, 1.33), 1.30 (1,26, 1,35), and 1.25 (1.21, 1.29) m/s, respectively. Overall, 3 mph and 5 km/h performed better than 2.5 mph and 4.5 km/h in balancing both sensitivity and specificity (higher Youden’s Indices). Overground walking speeds associated with 3 METs were similar across age groups. A heuristic threshold of 3 mph or 5 km/h may better identify absolutely-defined moderate intensity overground walking.

A Lightweight Attention-Based Convolutional Neural Networks for Tomato Leaf Disease Classification

Plant diseases pose a significant challenge for food production and safety. Therefore, it is indispensable to correctly identify plant diseases for timely intervention to protect crops from massive losses. The application of computer vision technology in phytopathology has increased exponentially due to automatic and accurate disease detection capability. However, a deep convolutional neural network (CNN) requires high computational resources, limiting its portability. In this study, a lightweight convolutional neural network was designed by incorporating different attention modules to improve the performance of the models. The models were trained, validated, and tested using tomato leaf disease datasets split into an 8:1:1 ratio. The efficacy of the various attention modules in plant disease classification was compared in terms of the performance and computational complexity of the models. The performance of the models was evaluated using the standard classification accuracy metrics (precision, recall, and F1 score). The results showed that CNN with attention mechanism improved the interclass precision and recall, thus increasing the overall accuracy (>1.1%). Moreover, the lightweight model significantly reduced network parameters (~16 times) and complexity (~23 times) compared to the standard ResNet50 model. However, amongst the proposed lightweight models, the model with attention mechanism nominally increased the network complexity and parameters compared to the model without attention modules, thereby producing better detection accuracy. Although all the attention modules enhanced the performance of CNN, the convolutional block attention module (CBAM) was the best (average accuracy 99.69%), followed by the self-attention (SA) mechanism (average accuracy 99.34%).

Evaluating Temporal Bias in Time Series Event Detection Methods

The detection of events in time series is an important task in several areas of knowledge where operations monitoring is essential. Experts often have to deal with choosing the most appropriate event detection method for a time series, which can be a complex task. There is a demand for benchmarking different methods in order to guide this choice. For this, standard classification accuracy metrics are usually adopted. However, they are insufficient for a qualitative analysis of the tendency of a method to precede or delay event detections. Such analysis is interesting for applications in which tolerance for "close" detections is important rather than focusing only on accurate ones. In this context, this paper proposes a more comprehensive event detection benchmark process, including an analysis of temporal bias of detection methods. For that, metrics based on the time distance between event detections and identified events (detection delay) are adopted. Computational experiments were conducted using real-world and synthetic datasets from Yahoo Labs and resources from the Harbinger framework for event detection. Adopting the proposed detection delay-based metrics helped obtain a complete overview of the performance and general behavior of detection methods.

An Enhanced Classification Model for Likelihood of Zero-Day Attack Detection and Estimation

The growing threat to sensitive information stored in computer systems and devices is becoming alarming. This is as a result of the proliferation of different malware created on a daily basis to cause zero-day attacks. Most of the malware whose signatures are known can easily be detected and blocked, however, the unknown malwares are the most dangerous. In this paper a zero-day vulnerability model based on deep-reinforcement learning is presented. The technique employs a Monte Carlo Based Pareto Rule (Deep-RL-MCB-PR) approach that exploits a reward learning and training feature with sparse feature generation and adaptive multi-layered recurrent prediction for the detection and subsequent mitigation of zero-day threats. The new model has been applied to the Kyoto benchmark datasets for intrusion detection systems, and compared to an existing system, that uses a multi-layer protection and a rule-based ranking (RBK) approach to detect a zero-day attack likelihood. Experiments were performed using the dataset, and simulation results show that the Deep-RL-MCB-PR technique when measured with the classification accuracy metrics, produced about 67.77%. The dataset was further magnified, and the result of classification accuracy showed about 75.84%. These results account for a better error response when compared to the RBK technique.

Evaluación de parámetros de segmentación en OBIA para la clasificación de coberturas del suelo a partir de imágenes VANT

<p>Unmanned Aerial Vehicles (UAVs) have given a new boost to remote sensing and image classification techniques due to the high level of detail among other factors. Object-based image analysis (OBIA) could improve classification accuracy unlike to pixel-based, especially in high-resolution images. OBIA application for image classification consists of three stages i.e., segmentation, class definition and training polygons, and classification. However, defining the parameters: spatial radius (SR), range radius (RR) and minimum region size (MR) is necessary during the segmentation stage. Despite their relevance, they are usually visually adjusted, which leads to a subjective interpretation. Therefore, it is of utmost importance to generate knowledge focused on evaluating combinations of these parameters. This study describes the use of the mean-shift segmentation algorithm followed by <em>Random Forest </em>classifier using Orfeo Toolbox software. It was considered a multispectral orthomosaic derived from UAV to generate a suburban map land cover in town of El Pueblito, Durango, Mexico. The main aim was to evaluate efficiency and segmentation quality of nine parameter combinations previously reported in scientific studies.This in terms of number generated polygons, processing time, discrepancy measures for segmentation and classification accuracy metrics. Results evidenced the importance of calibrating the input parameters in the segmentation algorithms. Best combination was RE=5, RR=7 and TMR=250, with a Kappa index of 0.90 and shortest processing time. On the other hand, RR showed a strong and inversely proportional degree of association regarding the classification accuracy metrics.</p>

A generalizable sentiment analysis method for creating a hotel dictionary: using big data on TripAdvisor hotel reviews

论可推广性的情感分析法以创建酒店字典：以TripAdvisor酒店评论为样本的大数据分析摘要研究目的对于在线游客评论的研究在过去的几年中与日俱增, 但是仍缺乏有效方法能在有限的时间喝预算内提供终端用户价值。本论文开发并测试了一套情感分析的新方法, 创建两套酒店相关的词库, 此方法超越了标准词典式分析法。研究设计/方法/途径研究样本为TripAdvisor酒店客户评论的大数据, 通过开发崭新的有配重的词库法, 来开展两极式情感分析。这个崭新的具有配重的词库法能够呈现透明化和可复制的程序, 准备、创建、并检验情感分析的词条。这个方法用到了两种词典（有配重的词典L1和手动选择的词典L2）, 本论文通过对TripAdvisor大数据进行使用词类划分精准度, 来检测和验证这两种词典。本论文采用两种热门方法（公共词典法和复杂机器学习算法）来对比词典的准确度。研究结果精确度对比结果证实了本论文的方法, 相较于机器学习算法, 显著地超越了以字典为基础的方法。研究结果还表明, 本论文的方法可以就预测用户情感趋势进行推广。研究实际启示本论文开发并验证了一项方法, 这种方法通过创建可信的词典进行大数据分析, 以判定用户情感。本论文创建的L2酒店词库对分析客人反馈是可靠有用的工具, 这个词库还能帮助酒店经理了解、预测、以及积极相应客人的态度和改变。本论文还提出了一项可以了解每个用户情感的简易方法, 这项方法可以通过对比的方式来检测和了解客人不同时间的情感变化, 以及根据其不同背景和经历的不同用户之间的变化。研究原创性/价值本论文提出并检测了一项新方法, 这项情感分析方法可以解决之前方法的局限并立脚于旅游行业。基于文献综述, 本论文是首篇研究, 使用词库法来进行情感分析和创建特别领域词典的方式。

Novel online Recommendation algorithm for Massive Open Online Courses (NoR-MOOCs).

Massive Open Online Courses (MOOCs) have gained in popularity over the last few years. The space of online learning resources has been increasing exponentially and has created a problem of information overload. To overcome this problem, recommender systems that can recommend learning resources to users according to their interests have been proposed. MOOCs contain a huge amount of data with the quantity of data increasing as new learners register. Traditional recommendation techniques suffer from scalability, sparsity and cold start problems resulting in poor quality recommendations. Furthermore, they cannot accommodate the incremental update of the model with the arrival of new data making them unsuitable for MOOCs dynamic environment. From this line of research, we propose a novel online recommender system, namely NoR-MOOCs, that is accurate, scales well with the data and moreover overcomes previously recorded problems with recommender systems. Through extensive experiments conducted over the COCO data-set, we have shown empirically that NoR-MOOCs significantly outperforms traditional KMeans and Collaborative Filtering algorithms in terms of predictive and classification accuracy metrics.

Application of Sentinel-1 data in mapping land-use and land cover in a complex seasonal landscape: a case study in coastal area of Vietnamese Mekong Delta

The advent of Sentinel-1 SAR data with high temporal and medium spatial resolutions along with its being unaffected by presence of cloud provided opportunities for using remote sensing in mapping LULCs with high temporal detail. The objective of this study is to explore the possibility of applying Sentinel-1 data together with OBIA and machine learning in classifying LULC in a complex agricultural landscape in the coastal area of Vietnamese Mekong Delta (VMD). Three approaches including Sentinel-1 alone, integrative Sentinel-1 and Sentinel-2 and lastly, only Sentinel-2 were deployed to examine chance of improvement of classification accuracy metrics on using Sentinel-1 data. The result showed that SAR data could capture seasonal patterns of different LULC classes. Also, supplementing Sentinel-1 can improve classification overall accuracy especially on integrating with optical data. However, Sentinel-1-alone approach performed poorly in most of LULC cases and bias of optical data can compromise classification accuracy of integrative approach.

Investigation of Combining Logitboost(M5P) under Active Learning Classification Tasks

Active learning is the category of partially supervised algorithms that is differentiated by its strategy to combine both the predictive ability of a base learner and the human knowledge so as to exploit adequately the existence of unlabeled data. Its ambition is to compose powerful learning algorithms which otherwise would be based only on insufficient labelled samples. Since the latter kind of information could raise important monetization costs and time obstacles, the human contribution should be seriously restricted compared with the former. For this reason, we investigate the use of the Logitboost wrapper classifier, a popular variant of ensemble algorithms which adopts the technique of boosting along with a regression base learner based on Model trees into 3 different active learning query strategies. We study its efficiency against 10 separate learners under a well-described active learning framework over 91 datasets which have been split to binary and multi-class problems. We also included one typical Logitboost variant with a separate internal regressor for discriminating the benefits of adopting a more accurate regression tree than one-node trees, while we examined the efficacy of one hyperparameter of the proposed algorithm. Since the application of the boosting technique may provide overall less biased predictions, we assume that the proposed algorithm, named as Logitboost(M5P), could provide both accurate and robust decisions under active learning scenarios that would be beneficial on real-life weakly supervised classification tasks. Its smoother weighting stage over the misclassified cases during training as well as the accurate behavior of M5P are the main factors that lead towards this performance. Proper statistical comparisons over the metric of classification accuracy verify our assumptions, while adoption of M5P instead of weak decision trees was proven to be more competitive for the majority of the examined problems. We present our results through appropriate summarization approaches and explanatory visualizations, commenting our results per case.

Random forest machine learning algorithm predicts virologic outcomes among HIV infected adults in Lausanne, Switzerland using electronically monitored combined antiretroviral treatment adherence

ABSTRACT Machine Learning (ML) can improve the analysis of complex and interrelated factors that place adherent people at risk of viral rebound. Our aim was to build ML model to predict RNA viral rebound from medication adherence and clinical data. Patients were followed up at the Swiss interprofessional medication adherence program (IMAP). Sociodemographic and clinical variables were retrieved from the Swiss HIV Cohort Study (SHCS). Daily electronic medication adherence between 2008–2016 were analyzed retrospectively. Predictor variables included: RNA viral load (VL), CD4 count, duration of ART, and adherence. Random Forest, was used with 10 fold cross validation to predict the RNA class for each data observation. Classification accuracy metrics were calculated for each of the 10-fold cross validation holdout datasets. The values for each range from 0 to 1 (better accuracy). 383 HIV+ patients, 56% male, 52% white, median (Q1, Q3): age 43 (36, 50), duration of electronic monitoring of adherence 564 (200, 1333) days, CD4 count 406 (209, 533) cells/mm3, time since HIV diagnosis was 8.4 (4, 13.5) years, were included. Average model classification accuracy metrics (AUC and F1) for RNA VL were 0.6465 and 0.7772, respectively. In conclusion, combining adherence with other clinical predictors improve predictions of RNA.

Pragmatic Language and Social Communication Tests for Students Aged 8–18 Years: A Review of Test Accuracy

Purpose The accuracy of commercially available pragmatic language and social communication tests was reviewed. Method Online sources were searched to identify tests providing stand-alone pragmatic or social communication scores. Test manuals were examined to extract information about clinical and typically developing group comparisons, including sample size, inclusionary/exclusionary criteria, and mean group differences. Classification accuracy metrics, including sensitivity, specificity, optimal cut scores, and likelihood ratios, were reviewed. Results Six domain-specific pragmatic language and social communication tests and four omnibus language tests were identified. Test content varied and included video and audio recordings and question-and-answer formats to address hypothetical social vignettes, retell and comprehend narratives, interpret nonliteral language and multiple-meaning words, identify and display emotions, and infer another's perspective. Sample size and inclusionary/exclusionary criteria for comparisons of students with typical development (TD), developmental language disorder (DLD), autism spectrum disorder (ASD), and social (pragmatic) communication disorder (SPCD) varied widely. Mean group differences between TD and ASD or SPCD groups were generally larger (i.e., > 0.5 SD to > 2.0 SD s) than TD and DLD comparisons. Test authors interpreted these results as support for the accuracy of tests to identify deficits associated with ASD and SPCD and not DLD. Classification accuracy was reported for five domain-specific tests. Optimal cut scores ranged from 85 to 92, with adequate sensitivity and specificity (i.e., ≥ 80%); however, positive likelihood ratios and associated confidence intervals revealed limited confidence in reported levels. Conclusions Inadequate sample description for comparison and classification studies limit a clinician's ability to compare a student's profile with those in the samples. Although none of the tests meet all standards for test accuracy reviewed in this report, a checklist to guide test selection is provided to assist in identifying the best available test for a student's profile, if a test is needed.