Accuracy Of Machine Learning Models Research Articles

Editorial of the special issue: adaptive and intelligent systems (AIS) for learning, control and optimization in dynamic environments

An increasing number of real world applications, such as network traffic, network monitoring, power systems etc., generates non-stationary behaviors. Therefore, the model built in order to predict their dynamics, optimize their performance or control their behavior must be adapted over time. During the last decade, the modeling based on learning in non-stationary environments has received more and more attention in the machine learning and data mining communities. This is because the requirements from practical (industrial) side induced a permanent paradigm shift from classical batch off-line learning procedures (as conducted throughout the 80ties and 90ties to build up accurate and high-performance machine learning models) to incremental, adaptive algorithms which are able to evolve the model structures, architectures and parameters fully autonomously and on the fly, typical in a single-pass manner. It is especially the case within the scope of dynamic data streaming context, changing environmental conditions or as part of large-scale problems, e.g. web mining, multi-sensor networks, sequential video analysis, Big Data or predictive maintenance in factories for the future (FoF) as part of the Industry 4.0 activities (even an important standpoint of the Horizon 2020 objective programme see https://ec.europa.eu/programmes/horizon2020/en/h2020-sections-projects). The necessity of such algorithms is also underlined by the increasing size of the databases and storages, which induces that conventional batch learning systems cannot be applied within a reasonable time frame and sufficient accuracy. Thus, a block-wise loading of the data has to be carried out, which again requires the usage of incremental and evolving learning algorithms. Thereby the learning algorithms are either guided by a permanent update of model parameters and structures, leading to a so-called anytime up-to-date experience, or by occasional update cycles

Machine learning molecular dynamics for the simulation of infrared spectra.

Machine learning has emerged as an invaluable tool in many research areas. In the present work, we harness this power to predict highly accurate molecular infrared spectra with unprecedented computational efficiency. To account for vibrational anharmonic and dynamical effects - typically neglected by conventional quantum chemistry approaches - we base our machine learning strategy on ab initio molecular dynamics simulations. While these simulations are usually extremely time consuming even for small molecules, we overcome these limitations by leveraging the power of a variety of machine learning techniques, not only accelerating simulations by several orders of magnitude, but also greatly extending the size of systems that can be treated. To this end, we develop a molecular dipole moment model based on environment dependent neural network charges and combine it with the neural network potential approach of Behler and Parrinello. Contrary to the prevalent big data philosophy, we are able to obtain very accurate machine learning models for the prediction of infrared spectra based on only a few hundreds of electronic structure reference points. This is made possible through the use of molecular forces during neural network potential training and the introduction of a fully automated sampling scheme. We demonstrate the power of our machine learning approach by applying it to model the infrared spectra of a methanol molecule, n-alkanes containing up to 200 atoms and the protonated alanine tripeptide, which at the same time represents the first application of machine learning techniques to simulate the dynamics of a peptide. In all of these case studies we find an excellent agreement between the infrared spectra predicted via machine learning models and the respective theoretical and experimental spectra.

Data-driven estimation of blood pressure using photoplethysmographic signals.

Noninvasive measurement of blood pressure by optical methods receives considerable interest, but the complexity of the measurement and the difficulty of adjusting parameters restrict applications. We develop a method for estimating the systolic and diastolic blood pressure using a single-point optical recording of a photoplethysmographic (PPG) signal. The estimation is data-driven, we use automated machine learning algorithms instead of mathematical models. Combining supervised learning with a discrete wavelet transform, the method is insensitive to minor irregularities in the PPG waveform, hence both pulse oximeters and smartphone cameras can record the signal. We evaluate the accuracy of the estimation on 78 samples from 65 subjects (40 male, 25 female, age 29±7) with no history of cardiovascular disease. The estimate for systolic blood pressure has a mean error 4.9±4.9 mm Hg, and 4.3±3.7 mm Hg for diastolic blood pressure when using the oximeter-obtained PPG. The same values are 5.1±4.3 mm Hg and 4.6±4.3 mm Hg when using the phone-obtained PPG, comparing with A&D UA-767PBT result as gold standard. The simplicity of the method encourages ambulatory measurement, and given the ease of sharing the measured data, we expect a shift to data-oriented approaches deriving insight from ubiquitous mobile devices that will yield more accurate machine learning models in monitoring blood pressure.

Machine Learning Techniques for Prediction of Early Childhood Obesity

This paper aims to predict childhood obesity after age two, using only data collected prior to the second birthday by a clinical decision support system called CHICA. Analyses of six different machine learning methods: RandomTree, RandomForest, J48, ID3, Naïve Bayes, and Bayes trained on CHICA data show that an accurate, sensitive model can be created. Of the methods analyzed, the ID3 model trained on the CHICA dataset proved the best overall performance with accuracy of 85% and sensitivity of 89%. Additionally, the ID3 model had a positive predictive value of 84% and a negative predictive value of 88%. The structure of the tree also gives insight into the strongest predictors of future obesity in children. Many of the strongest predictors seen in the ID3 modeling of the CHICA dataset have been independently validated in the literature as correlated with obesity, thereby supporting the validity of the model. This study demonstrated that data from a production clinical decision support system can be used to build an accurate machine learning model to predict obesity in children after age two.

Accuracy of machine learning models versus “hand crafted” expert systems – A credit scoring case study

Relatively few publications compare machine learning models with expert systems when applied to the same problem domain. Most publications emphasize those cases where the former beat the latter. Is it a realistic picture of the state of the art? Some other findings are presented here. The accuracy of a real world “mind crafted” credit scoring expert system is compared with dozens of machine learning models. The results show that while some machine learning models can surpass the expert system’s accuracy with statistical significance, most models do not. More interestingly, this happened only when the problem was treated as regression. In contrast, no machine learning model showed any statistically significant advantage over the expert system’s accuracy when the same problem was treated as classification. Since the true nature of the class data was ordinal, the latter is the more appropriate setting. It is also shown that the answer to the question is highly dependent on the meter that is being used to define accuracy.