Learning Metrics Research Articles

Predictive mapping of aquatic ecosystems by means of support vector machines and random forests

In a context of rapid change, mapping aquatic ecosystems is crucial for their protection, particularly in regions of the world where a comprehensive inventory of water bodies is yet to be developed. This paper presents a machine learning methodology for predictive aquatic ecosystem mapping. Two state-of-the-art classifiers were used, namely support vector machines (SVM) and random forests (RFC). These were implemented in a GIS environment and trained on a sample of 350 points of a western Colombian catchment, where the presence or absence or water was known beforehand. Sixty percent of the dataset was used to train the algorithms and the remainder was used to calibrate predictive accuracy. Both classifiers rendered a realistic picture of the catchment’s water bodies. RFC obtained a mean test score in excess of 0.93 and an area under the receiver operating characteristic curve of 0.95. SVM scored 0.89 and 0.89, respectively, after optimization. Performance was better overall in flat regions. SVM identified correctly over 87% of the surface area of lentic and 82% of lotic water bodies, while RFC scored over 80% in both cases. In mountain regions precision dropped to 69% and 62% for SVM and 65% and 76% for RFC. This leads to three conclusions: (a) the method provides satisfactory results at the regional scale and is versatile enough to export to other settings; (b) local-scale detail cannot be captured in the absence of an accurate high-resolution digital elevation model, particularly in mountain areas; (c) complementing standard machine learning metrics with ad hoc indicators is likely needed in cases where the input dataset consists solely of unambiguous examples.

Passports to the Global South, UN flags, favourite experts: understanding the interplay between UNESCO and the OECD within the SDG4 context

ABSTRACT As the OECD and UNESCO increasingly work in the same geographical and thematic areas, this paper explores the interplay between the organizations in relation to the making of global learning metrics and the global monitoring of the Sustainable Development Goal 4. Although the organizations seek to cooperate, this paper shows how this is limited by differences within and between the organizations – differences which relate to their global leadership, their understandings of education, the expertise they draw on, and their approaches to Member States. The paper demonstrates the continuation of an established struggle and interdependency between the organizations as they seek to assert their legitimacy in the context of SDG4.

Out-of-Distribution Detection for Deep Neural Networks With Isolation Forest and Local Outlier Factor

Deep Neural Networks (DNNs) are extensively deployed in today’s safety-critical autonomous systems thanks to their excellent performance. However, they are known to make mistakes unpredictably, e.g., a DNN may misclassify an object if it is used for perception, or issue unsafe control commands if it is used for planning and control. One common cause for such unpredictable mistakes is Out-of-Distribution (OOD) input samples, i.e., samples that fall outside of the distribution of the training dataset. We present a framework for OOD detection based on outlier detection in one or more hidden layers of a DNN with a runtime monitor based on either Isolation Forest (IF) or Local Outlier Factor (LOF). Performance evaluation indicates that LOF is a promising method in terms of both the Machine Learning metrics of precision, recall, F1 score and accuracy, as well as computational efficiency during testing.

НАСТРОЙКА, ОБУЧЕНИЕ И ТЕСТИРОВАНИЕ СВЕРТОЧНОЙ НЕЙРОННОЙ СЕТИ В ЗАДАЧЕ ТЕМАТИЧЕСКОЙ ОБРАБОТКИ СПУТНИКОВЫХ ИЗОБРАЖЕНИЙ

We describe an algorithm based on a convolutional neural network that detects cloud and snow covers in satellite images. Algorithm accuracy was evaluated using machine learning metrics. The proposed algorithm is fully automatic

Machine learning-based prediction of adverse events following an acute coronary syndrome (PRAISE): a modelling study of pooled datasets

The accuracy of current prediction tools for ischaemic and bleeding events after an acute coronary syndrome (ACS) remains insufficient for individualised patient management strategies. We developed a machine learning-based risk stratification model to predict all-cause death, recurrent acute myocardial infarction, and major bleeding after ACS. Different machine learning models for the prediction of 1-year post-discharge all-cause death, myocardial infarction, and major bleeding (defined as Bleeding Academic Research Consortium type 3 or 5) were trained on a cohort of 19 826 adult patients with ACS (split into a training cohort [80%] and internal validation cohort [20%]) from the BleeMACS and RENAMI registries, which included patients across several continents. 25 clinical features routinely assessed at discharge were used to inform the models. The best-performing model for each study outcome (the PRAISE score) was tested in an external validation cohort of 3444 patients with ACS pooled from a randomised controlled trial and three prospective registries. Model performance was assessed according to a range of learning metrics including area under the receiver operating characteristic curve (AUC). The PRAISE score showed an AUC of 0·82 (95% CI 0·78-0·85) in the internal validation cohort and 0·92 (0·90-0·93) in the external validation cohort for 1-year all-cause death; an AUC of 0·74 (0·70-0·78) in the internal validation cohort and 0·81 (0·76-0·85) in the external validation cohort for 1-year myocardial infarction; and an AUC of 0·70 (0·66-0·75) in the internal validation cohort and 0·86 (0·82-0·89) in the external validation cohort for 1-year major bleeding. A machine learning-based approach for the identification of predictors of events after an ACS is feasible and effective. The PRAISE score showed accurate discriminative capabilities for the prediction of all-cause death, myocardial infarction, and major bleeding, and might be useful to guide clinical decision making. None.

Deep learning based static hand gesture recognition

Hand gesture recognition becomes a popular topic of deep learning and provides many application fields for bridging the human–computer barrier and has a positive impact on our daily life. The primary idea of our project is a static gesture acquisition from depth camera and to process the input images to train the deep convolutional neural network pre-trained on ImageNet dataset. Proposed system consists of gesture capture device (Intel® RealSense™ depth camera D435), pre-processing and image segmentation algorithms, feature extraction algorithm and object classification. For pre-processing and image segmentation algorithms computer vision methods from the OpenCV and Intel Real Sense libraries are used. The subsystem for features extracting and gestures classification is based on the modified VGG-16 by using the TensorFlow&Keras deep learning framework. Performance of the static gestures recognition system is evaluated using maching learning metrics. Experimental results show that the proposed model, trained on a database of 2000 images, provides high recognition accuracy both at the training and testing stages.

Classifying defective software projects based on machine learning and complexity metrics

Classifying defective software projects based on machine learning and complexity metrics

Investigating the Impact of Activity Class Number in Fall Detection Systems

In this study, the contribution of reducing the number of classes in the machine learning based classifiers to the accuracy level of the fall detection systems developed for older individuals is investigated. Within the scope of the study, a public dataset is used and the appropriate data source is determined, the features frequently used in determining the activity and posture are extracted and the analysis of the most successful classifier is made according to the machine learning metrics using all the classifiers in the MATLAB Machine Learning Framework. The number of classes is gradually decreased and its effect on success is examined. The classification carried out with the Cubic Support Vector Machine (Cubic SVM) algorithm, which is determined as the most successful classifier in the classification without reducing the number of classes at the initial stage, is 96.4%. In accordance with the nature of the problem, contrary to the studies in the literature, when the number of classes is reduced to two, the falls are correctly determined by 99.3% by using k nearest neighbor algorithm.

Health Monitoring System for Autonomous Vehicles using Dynamic Bayesian Networks for Diagnosis and Prognosis

Autonomous Vehicles have the potential to change the urban transport scenario. However, to be able to safely navigate autonomously they need to deal with faults that its components are subject to. Therefore, Health Monitoring System is a essential component of the autonomous system, since allows Fault Detection and Diagnosis. In addition, Prognosis System is also important, since it allows predictive maintenance and safer decisions during vehicle navigation. This paper presents a Hierarchical Component-based Health Monitoring System with Fault Detection, Diagnosis and Prognosis using Dynamic Bayesian Network (DBN) with residue generation, a combination of knowledge-based and model-based detection, diagnosis and prognosis approaches. We evaluate the proposed Dynamic Bayesian Network using different machine learning metrics and a dataset with sensor readings gathered using the CaRINA II autonomous vehicle platform, and the CARLA simulator. Both simulated and experimental results demonstrated a positive performance of the DBNs even with high rate of missing data for some of the model’s variables.

Multiset Feature Learning for Highly Imbalanced Data Classification.

With the expansion of data, increasing imbalanced data has emerged. When the imbalance ratio (IR) of data is high, most existing imbalanced learning methods decline seriously in classification performance. In this paper, we systematically investigate the highly imbalanced data classification problem, and propose an uncorrelated cost-sensitive multiset learning (UCML) approach for it. Specifically, UCML first constructs multiple balanced subsets through random partition, and then employs the multiset feature learning (MFL) to learn discriminant features from the constructed multiset. To enhance the usability of each subset and deal with the non-linearity issue existed in each subset, we further propose a deep metric based UCML (DM-UCML) approach. DM-UCML introduces the generative adversarial network technique into the multiset constructing process, such that each subset can own similar distribution with the original dataset. To cope with the non-linearity issue, DM-UCML integrates deep metric learning with MFL, such that more favorable performance can be achieved. In addition, DM-UCML designs a new discriminant term to enhance the discriminability of learned metrics. Experiments on eight traditional highly class-imbalanced datasets and two large-scale datasets indicate that: the proposed approaches outperform state-of-the-art highly imbalanced learning methods and are more robust to high IR.

Capacity-building partnerships for surgical post-graduate training in low- and middle-income countries: a scoping review of the literature with exploratory thematic synthesis.

In recent years, international surgical programmes have moved away from vertical service delivery and towards collaborative, capacity-building partnerships. The aim of this review was to provide a map of the current literature on international surgical training partnerships together with an exploration of factors influencing their implementation. Three bibliographic databases were searched for peer-reviewed reports of surgical training partnerships between organizations in high- and low or middle-income countries to July 2018. Reports were sorted in an iterative fashion into groups of similar programmes, and data were extracted to record the intervention strategies, context, financing, reported results and themes around implementation. Eighty-six reports were grouped into five types of programme: full residency training, bi-institutional twinning partnerships, diagonal/sub-specialist programmes, focused interventions or courses and programmes using remote support. Few articles were written from the perspective of the low-middle income partner. Full residency programmes and some diagonal/sub-specialist programmes report numbers trained while twinning partnerships and focused interventions tend to focus on process, partners' reactions to the programme and learning metrics. Two thematic networks emerged from the thematic synthesis. The first made explicit the mechanisms by which partnerships are expected to contribute to improved access to surgical care and a second identified the importance of in-country leadership in determining programme results. Training partnerships are assumed to improve access to surgical care by a number of routes. A candidate programme theory is proposed together with some more focused theories that could inform future research. Supporting the development of the surgical leadership in low- and middle-income countries is key.

E-Recruitment recommender systems: a systematic review

Recommender Systems (RS) are a subclass of information filtering systems that seek to predict the rating or preference a user would give to an item. e-Recruitment is one of the domains in which RS can contribute due to presenting a list of interesting jobs to a candidate or a list of candidates to a recruiter. This study presents an up-to-date systematic review of recommender systems applied to e-Recruitment considering only papers published from 2012 up to 2020. We searched three databases for published journal articles, conference papers and book chapters. We then evaluated these works in terms of which kinds of RS were applied for e-Recruitment, what kind of information was used in the e-Recruitment RS, and how they were assessed. A total of 896 papers were collected, out of which sixty three research works were included in the survey based on the inclusion and exclusion criteria adopted. We divided the recommender types into five categories (Content-Based Recommendation 26.98%; Collaborative Filtering 6.35%; Knowledge-Based Recommendation 12.7%; Hybrid approaches 20.63%; and Other Types 33.33%); the types of information used were divided into four categories (Social Network 38.1%; Resumes and Job Posts 42.85%; Behavior or Feedback 12.7%; and Others 6.35%), and the assessment types were categorized into four types (Expert Validation 20.83%; Machine Learning Metrics 41.67%; Challenge-specific Metrics 22.92%; and Utility measures 14.58%). Although in many cases a paper may belong to more than one category for each evaluation axis, we chose the most predominant one for our categorization. In addition, there is a clear trend for hybrid and non-traditional techniques to overcome the challenges of e-Recruitment domain.

A Field-Based Adaptation of the Classic Morris Water Maze to Assess Learning and Memory in a Free-Living Animal

Free-living animals rely on cognitive functions, such as learning and memory, for numerous aspects of their survival. However, research involving these mechanisms is often limited to the laboratory where animals are far removed from natural influences. Here, we adapted the Morris Water Maze (MWM) for use in the field to test learning and memory of free-living white-footed mice, Peromyscus leucopus. The MWM consists of a pool filled with opaque water and a platform that is hidden beneath the surface of the water. Mice were tasked with finding an escape platform using proximate-cue based spatial learning skills across trials within and between two tests. The two tests occurred 2 hrs apart and each consisted of 5 trials with a maximum search time of 60 s, separated by 15-s breaks. We found mice demonstrated rapid learning within the first 5 trials, as indicated by a faster time and reduced distanced swam to find the platform in trial 5 as compared to trial 1 in Test 1. We also found mice remembered the task, as indicated by equivalent performance (in time and distance) after the two-hour delay period (i.e., performance in Test 2 trial 1 was similar to that in Test 1 trial 5). Lastly, we found that there was no increase in performance as mice completed an additional 5 trials (i.e., no difference in Test 1 trial 5 vs. Test 2 trial 5). By assessing several variables that serve as standard metrics for learning and memory within the MWM, we were able to illustrate the ability of wild mice to successfully navigate within a modified version of an established learning/memory paradigm. This study is important for promoting more accessible forms of cognitive testing in free-living animals and use of such a method could provide insights into how animals cope with environmental stressors in their natural environment such as anthropogenic disturbance, invasive species, and habitat degradation. A better understanding of cognitive behavior in wild animals may help diminish the disparity of knowledge that is often present between laboratory and field research.

On novelty detection for multi-class classification using non-linear metric learning

Novelty detection is a binary task aimed at identifying whether a test sample is novel or unusual compared to a previously observed training set. A typical approach is to consider distance as a criterion to detect such novelties. However, most previous work does not focus on finding an optimum distance for each particular problem. In this paper, we propose to detect novelties by exploiting non-linear distances learned from multi-class training data. For this purpose, we adopt a kernelization technique jointly with the Large Margin Nearest Neighbor (LMNN) metric learning algorithm. The optimum distance tries to keep each known class’ instances together while pushing instances from different known classes to remain reasonably distant. We propose a variant of the K-Nearest Neighbors (KNN) classifier that employs the learned distance to detect novelties. Besides, we use the learned distance to perform multi-class classification. We show quantitative and qualitative experiments conducted on synthetic and real data sets, revealing that the learned metrics are effective in improving novelty detection compared to other metrics. Our method also outperforms previous work regularly used for novelty detection.

Ground Metric Learning on Graphs

Optimal transport (OT) distances between probability distributions are parameterized by the ground metric they use between observations. Their relevance for real-life applications strongly hinges on whether that ground metric parameter is suitably chosen. Selecting it adaptively and algorithmically from prior knowledge, the so-called ground metric learning GML) problem, has therefore appeared in various settings. We consider it in this paper when the learned metric is constrained to be a geodesic distance on a graph that supports the measures of interest. This imposes a rich structure for candidate metrics, but also enables far more efficient learning procedures when compared to a direct optimization over the space of all metric matrices. We use this setting to tackle an inverse problem stemming from the observation of a density evolving with time: we seek a graph ground metric such that the OT interpolation between the starting and ending densities that result from that ground metric agrees with the observed evolution. This OT dynamic framework is relevant to model natural phenomena exhibiting displacements of mass, such as for instance the evolution of the color palette induced by the modification of lighting and materials.

Predicting Opioid Overdose Risk of Patients with Opioid Prescriptions Using Electronic Health Records Based on Temporal Deep Learning

Background: The US is experiencing an opioid epidemic, such that opioid overdose is causing more than 100 deaths per day. Identifying patients at high risk of Opioid Overdose (OD) can help to make early clinical interventions to reduce the risk of OD. We aim to build a deep learning model that can predict the patients at high risk for opioid overdose and identify most relevant features. Method: The study included the information of 5,231,614 patients from the Health Facts database with at least one opioid prescription between January 1, 2008 and December 31, 2017. Potential predictors (n = 1185) were extracted to build a feature matrix for prediction. A Long Short-Term Memory (LSTM) based model was built to predict overdose risk in the next hospital visit. Prediction performance was compared with other machine learning methods assessed using machine learning metrics. Findings: Our temporal deep learning model built upon LSTM outperformed the other methods on opioid overdose prediction. It achieved the highest F-1 score (F-1 score: 0.8065, AUCROC: 0.8885). The model is also able to reveal top ranked predictive features, including medications and vital signs. Interpretation: This study demonstrates that a temporal deep learning based predictive model can achieve promising results on identifying risk of opioid overdose of patients using the history of electronic health records. It provides an alternative informatics-based approach to improving clinical decision support for possible early detection and intervention to reduce opioid overdose. Funding: This work was funded partially by the Stony Brook University OVPR Seed Grant 1158484-63845-6. Declaration of Interests: The authors do not have any conflicts of interest to disclose. Ethics Approval Statement: All the data used in the study are de-identified, and the study is approved by Stony Brook University.

Semisupervised Laplace-Regularized Multimodality Metric Learning.

Distance metric learning, which aims at learning an appropriate metric from data automatically, plays a crucial role in the fields of pattern recognition and information retrieval. A tremendous amount of work has been devoted to metric learning in recent years, but much of the work is basically designed for training a linear and global metric with labeled samples. When data are represented with multimodal and high-dimensional features and only limited supervision information is available, these approaches are inevitably confronted with a series of critical problems: 1) naive concatenation of feature vectors can cause the curse of dimensionality in learning metrics and 2) ignorance of utilizing massive unlabeled data may lead to overfitting. To mitigate this deficiency, we develop a semisupervised Laplace-regularized multimodal metric-learning method in this work, which explores a joint formulation of multiple metrics as well as weights for learning appropriate distances: 1) it learns a global optimal distance metric on each feature space and 2) it searches the optimal combination weights of multiple features. Experimental results demonstrate both the effectiveness and efficiency of our method on retrieval and classification tasks.

Developing a Conceptual Model of Learning Analytics in Serious Games for STEM Education

Utilizing serious games as teaching aid can stimulate students’ interest in learning and enhancing students’ understanding of STEM learning. The use of serious game on learning analytics for STEM learning provides assessment data to measure students’ performances and achievements of predefined learning outcomes. While numerous researches on learning analytic in serious games have been conducted, studies on their association with STEM learning are scarce. Past studies also indicate that teachers use serious games in teaching but yet to utilize serious games as assessment tools. This situation leads to the need of developing a conceptual model of learning analytics in serious games for STEM education (APPS-STEM). The conceptual model was initially developed through the focus group approach and literature review to examine the learning metrics of serious games, which is appropriate for STEM learning before being verified by experts. Based on the result from expert review, the conceptual model of APPS-STEM contains nine themes, namely effectiveness, problem-solving thinking and creativity, flexibility, key stakeholders, emotions, serious game design, curriculum profiles, learning profiles, and target user norms. This conceptual model is expected to serve as a guide for stakeholders to implement learning analytics in serious games design for STEM learning. Further study will be the development of serious game prototypes to determine the effectiveness of the APPS-STEM model in the STEM learning paradigm.

Vision models fine-tuned by cinema professionals for High Dynamic Range imaging in movies

Many challenges that deal with processing of HDR material remain very much open for the film industry, whose extremely demanding quality standards are not met by existing automatic methods. Therefore, when dealing with HDR content, substantial work by very skilled technicians has to be carried out at every step of the movie production chain. Based on recent findings and models from vision science, we propose in this work effective tone mapping and inverse tone mapping algorithms for production, post-production and exhibition. These methods are automatic and real-time, and they have been both fine-tuned and validated by cinema professionals, with psychophysical tests demonstrating that the proposed algorithms outperform both the academic and industrial state-of-the-art. We believe these methods bring the field closer to having fully automated solutions for important challenges for the cinema industry that are currently solved manually or sub-optimally. Another contribution of our research is to highlight the limitations of existing image quality metrics when applied to the tone mapping problem, as none of them, including two state-of-the-art deep learning metrics for image perception, are able to predict the preferences of the observers.

Snow and cloud detection using a convolutional neural network and low-resolution data from the Electro-L No. 2 Satellite

We describe an algorithm based on a convolutional neural network that detects cloud formations and snow cover in satellite images using textures. Herein, multispectral satellite images, received from a multizone scanning instrument used for hydrometeorological support and installed on the Russian satellite Electro-L No. 2, are used as input data. The problem of snow and cloud classification in the absence of a spectral channel in the range of 1.4 to 1.8 μm, which is necessary for their accurate separation, is considered. The developed algorithm can produce cloud and snow cover masks for an area limited by the values of the solar zenith angle in the range of 0 deg to 80 deg for daytime. Algorithm accuracy was evaluated using machine learning metrics and comparing its results with ground truth masks segmented manually by an experienced interpreter. In addition, we compared the resulting masks with a similar cloud mask product from the European Organisation for the Exploitation of Meteorological Satellites based on the data of the Spinning Enhanced Visible and Infrared Imager (SEVIRI) instrument installed on the Meteosat-8 satellite. According to the results of this comparison, we conclude that the cloud masks produced by the proposed convolutional neural network-based algorithm have a lower probability of false detection than products based on the SEVIRI data. The proposed algorithm is fully automatic, and it works in any season of the year during the daytime.