Machine Learning Library Research Articles

Quantification of Deep Neural Network Prediction Uncertainties for VVUQ of Machine Learning Models

Recent performance breakthroughs in artificial intelligence (AI) and machine learning (ML), especially advances in deep learning, the availability of powerful and easy-to-use ML libraries (e.g., scikit-learn, TensorFlow, PyTorch), and increasing computational power, have led to unprecedented interest in AI/ML among nuclear engineers. For physics-based computational models, verification, validation, and uncertainty quantification (VVUQ) processes have been very widely investigated, and many methodologies have been developed. However, VVUQ of ML models has been relatively less studied, especially in nuclear engineering. This work focuses on uncertainty quantification (UQ) of ML models as a preliminary step of ML VVUQ, more specifically Deep Neural Networks (DNNs) because they are the most widely used supervised ML algorithm for both regression and classification tasks. This work aims at quantifying the prediction or approximation uncertainties of DNNs when they are used as surrogate models for expensive physical models. Three techniques for UQ of DNNs are compared, namely, Monte Carlo Dropout (MCD), Deep Ensembles (DE), and Bayesian Neural Networks (BNNs). Two nuclear engineering examples are used to benchmark these methods: (1) time-dependent fission gas release data using the Bison code and (2) void fraction simulation based on the Boiling Water Reactor Full-size Fine-Mesh Bundle Tests (BFBT) benchmark using the TRACE code. It is found that the three methods typically require different DNN architectures and hyperparameters to optimize their performance. The UQ results also depend on the amount of training data available and the nature of the data. Overall, all three methods can provide reasonable estimations of the approximation uncertainties. The uncertainties are generally smaller when the mean predictions are close to the test data while the BNN methods usually produce larger uncertainties than MCD and DE.

Priority Evasion Attack: An Adversarial Example That Considers the Priority of Attack on Each Classifier

Deep neural networks (DNNs) provide excellent services in machine learning tasks such as image recognition, speech recognition, pattern recognition, and intrusion detection. However, an adversarial example created by adding a little noise to the original data can result in misclassification by the DNN and the human eye cannot tell the difference from the original data. For example, if an attacker creates a modified right-turn traffic sign that is incorrectly categorized by a DNN, an autonomous vehicle with the DNN will incorrectly classify the modified right-turn traffic sign as a U-Turn sign, while a human will correctly classify that changed sign as right turn sign. Such an adversarial example is a serious threat to a DNN. Recently, an adversarial example with multiple targets was introduced that causes misclassification by multiple models within each target class using a single modified image. However, it has the weakness that as the number of target models increases, the overall attack success rate decreases. Therefore, if there are multiple models that the attacker wishes to attack, the attacker must control the attack success rate for each model by considering the attack priority for each model. In this paper, we propose a priority adversarial example that considers the attack priority for each model in cases targeting multiple models. The proposed method controls the attack success rate for each model by adjusting the weight of the attack function in the generation process while maintaining minimal distortion. We used MNIST and CIFAR10 as data sets and Tensorflow as machine learning library. Experimental results show that the proposed method can control the attack success rate for each model by considering each model's attack priority while maintaining minimal distortion (average 3.95 and 2.45 with MNIST for targeted and untargeted attacks, respectively, and average 51.95 and 44.45 with CIFAR10 for targeted and untargeted attacks, respectively).

The study of the quality of multi-step time series forecasting

The work is devoted to the study of the quality of multistep forecasting of time series using the electricity consumption data for forecasting. Five models of multistep forecasting have been implemented, with their subsequent training and evaluation of the results obtained. The dataset is an upgraded minute-by-minute measurement of four years of electricity consumption. The dataset has been divided into training, validation, and test samples for training and testing models. The implementation is simplified by using the TensorFlow machine learning library, which allows us to conveniently process and present data; build and train neural networks. The TensorFlow functionality also provides standard metrics used to assess the accuracy of time series forecasting, which made it possible to evaluate the obtained models for forecasting the time series of electricity consumption and highlight the best ofthose considered according to the given indicators. The models are built in such a way that they can be used in studies of the quality of time series forecasting in various areas of human life. The problem of multistep forecasting for twenty fourhours ahead, considered in the paper, has not yet been solved for estimating electricity consumption. Theobtainedforecasting accuracy is comparable to recently published methods for estimating electricity consumption used in other conditions.At the same time, the forecasting accuracy of the constructed models has been improved in comparison with other methods.

MULTIMODAL SPEECH RECOGNITION BASED ON AUDIO AND TEXT DATA

Systems of machine translation of texts from one language to another simulate the work of a human translator. Their performance depends on the ability to understand the grammar rules of the language. In translation, the basic units are not individual words, but word combinations or phraseological units that express different concepts. Only by using them, more complex ideas can be expressed through the translated text. The main feature of machine translation is different length for input and output. The ability to work with different lengths of input and output provides us with the approach of recurrent neural networks. A recurrent neural network (RNN) is a class of artificial neural network that has connections between nodes. In this case, a connection refers to a connection from a more distant node to a less distant node. The presence of connections allows the RNN to remember and reproduce the entire sequence of reactions to one stimulus. From the point of view of programming, such networks are analogous to cyclic execution, and from the point of view of the system, such networks are equivalent to a state machine. RNNs are commonly used to process word sequences in natural language processing. Usually, a hidden Markov model (HMM) and an N-program language model are used to process a sequence of words. Deep learning has completely changed the approach to machine translation. Researchers in the deep learning field has created simple solutions based on machine learning that outperform the best expert systems. In this paper was reviewed the main features of machine translation based on recurrent neural networks. The advantages of systems based on RNN using the sequence-to-sequence model against statistical translation systems are also highlighted in the article. Two machine translation systems based on the sequence-to-sequence model were constructed using Keras and PyTorch machine learning libraries. Based on the obtained results, libraries analysis was done, and their performance comparison.

Large scale K-means clustering using GPUs

The k-means algorithm is widely used for clustering, compressing, and summarizing vector data. We present a fast and memory-efficient GPU-based algorithm for exact k-means, Asynchronous Selective Batched K-means (ASB K-means). Unlike most GPU-based k-means algorithms that require loading the whole dataset onto the GPU for clustering, the amount of GPU memory required to run our algorithm can be chosen to be much smaller than the size of the whole dataset. Thus, our algorithm can cluster datasets whose size exceeds the available GPU memory. The algorithm works in a batched fashion and applies the triangle inequality in each k-means iteration to omit a data point if its membership assignment, i.e., the cluster it belongs to, remains unchanged, thus significantly reducing the number of data points that need to be transferred between the CPU’s RAM and the GPU’s global memory and enabling the algorithm to very efficiently process large datasets. Our algorithm can be substantially faster than a GPU-based implementation of standard k-means even in situations when application of the standard algorithm is feasible because the whole dataset fits into GPU memory. Experiments show that ASB K-means can run up to 15x times faster than a standard GPU-based implementation of k-means, and it also outperforms the GPU-based k-means implementation in NVIDIA’s open-source RAPIDS machine learning library on all the datasets used in our experiments.

Evaluating and testing neural-network algorithm capabilities for automating image data analysis for remote sensing of the Earth

Subject of study. Results of comparative analysis and testing of the application capabilities of several neural network detection algorithms, programming interfaces, and machine learning libraries for real-time analysis of graphical data from scanning thermal imaging surveying systems are presented. Method. The availability of programming interfaces for integration and adaptation of algorithms into the developed software, data processing rate, and object detection accuracy were selected as the main criteria for assessing the detection algorithms. These criteria were evaluated using practical experiments involving training and running neural network algorithms on test software using computers with different configurations. Main results. Modern neural network algorithms were demonstrated to enable image data processing with detection accuracy for specified object classes, which is sufficient for the automation of image recognition aimed at real-time processing of images from scanning thermal imaging surveying systems. Practical significance. The results of the investigation and tests presented in this study can be advantageous and reduce the time required for developers to find base neural network algorithms suitable for a practical implementation of automation aimed at image processing. Implementation of the considered algorithms in the developed software enables image analysis and processing in real time during surveillance owing to a reduction in the amount of data processed by the operator, thus enabling the removal of the post-processing stage from the technological sequence of the surveillance.

Impact of data quality on supervised machine learning: Case study on drilling vibrations

Training complex machine learning and deep learning models has become straightforward with the advent of highly efficient, open-source machine learning libraries. Supervised classification techniques such as logistic regression, random forests, and neural networks have also gained popularity in the drilling industry on the back of promising results. As a result, these techniques have been increasingly researched, especially in the domain of drilling vibrations. However, much of this research interest has been limited to finding the best classification model for estimating severity of downhole vibrations. While the choice of classification model is important, we argue that the successful implementation and adoption of machine learning technologies is equally dependent on correctly studying, cleaning, pre-processing the vibration drilling data before applying machine learning techniques. We show that, in certain cases, correctly pre-processing the data guarantees competitive classification performance regardless of the choice of classification model. Specifically, we empirically investigate how factors such as data sampling frequency, data labeling technique, feature extraction technique, and class imbalance impact the performance of different popular classifiers, when dealing with drilling data. We make recommendations specific to vibration classification and highlight pitfalls of certain techniques in that context. Finally, we also develop a step-by-step workflow which enables users to select the correct parameters and techniques at every step, from data collection to model training.

DeePKS-kit: A package for developing machine learning-based chemically accurate energy and density functional models

We introduce DeePKS-kit, an open-source software package for developing machine learning based energy and density functional models. DeePKS-kit is interfaced with PyTorch, an open-source machine learning library, and PySCF, an ab initio computational chemistry program that provides simple and customized tools for developing quantum chemistry codes. It supports the DeePHF and DeePKS methods. In addition to explaining the details in the methodology and the software, we also provide an example of developing a chemically accurate model for water clusters. Program summaryProgram Title: DeePKS-kitCPC Library link to program files:https://doi.org/10.17632/x54bnz5vxk.1Developer's repository link:https://github.com/deepmodeling/deepks-kitLicensing provisions: LGPLProgramming language: PythonNature of problem: Modeling the energy and density functional in electronic structure problems with high accuracy by neural network models. Solving electronic ground state energy and charge density using the learned model.Solution method: DeePHS and DeePKS methods are implemented, interfaced with PyTorch and PySCF for neural network training and self-consistent field calculations. An iterative learning procedure is included to train the model self-consistently.

Supervised Machine Learning for Predicting Carbohydrate Malabsorptions Using Hydrogen Breath Tests

Abstract Introduction: Carbohydrate malabsorptions symptoms include intestinal fluid retention, causing diarrhea and abdominal distention. The aim of this work is to create a machine learning model that predicts carbohydrate malabsorption using H2 measurements from lactose and fructose tolerance tests. Methods: We compare the predictive ability of popular classifiers with classifiers that are specifically designed for time series data. Our approach was implemented using sklearn and sktime Python machine learning libraries. Results: The highest predictive ability for the fructose dataset was achieved using a Random Forest Classifier (balanced accuracy = 0.91). In contrast, the highest predictive ability (balanced accuracy = 0.81) for the lactose dataset was obtained using an IndividualTDE time classifier. Conclusion: Our results indicate a high predictive ability for distinguishing between carbohydrate malabsorptions. However, the detection of SIBO is challenging but adapted time classifier models could reach higher performances compared to standard methods. Our results could establish the basis of an expert system for diagnosing carbohydrate malabsorptions and SIBO, respectively.

Machine Learning in Pansharpening: A benchmark, from shallow to deep networks

Machine learning (ML) is influencing the literature in several research fields, often through state-of-the-art approaches. In the past several years, ML has been explored for pansharpening, i.e., an image fusion technique based on the combination of a multispectral (MS) image, which is characterized by its medium/low spatial resolution, and higher-spatial-resolution panchromatic (PAN) data. Thus, ML for pansharpening represents an emerging research line that deserves further investigation. In this article, we go through some powerful and widely used ML-based approaches for pansharpening that have been recently proposed in the related literature. Eight approaches are extensively compared. Implementations of these eight methods, exploiting a common software platform and ML library, are developed for comparison purposes. The ML framework for pansharpening will be freely distributed to the scientific community. Experimental results using data acquired by five commonly used sensors for pansharpening and well-established protocols for performance assessment (both at reduced resolution and at full resolution) are shown. The ML-based approaches are compared with a benchmark consisting of classical and variational optimization (VO)-based methods. The pros and cons of each pansharpening technique, based on the training-by-examples philosophy, are reported together with a broad computational analysis. The toolbox is provided in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/liangjiandeng/DLPan-Toolbox</uri> .

Webcam Based Gesture Detection and Recognition to Navigate in the Application

— Most e-commerce websites do not provide their users an engaging purchasing experience. The buyer can only watch images or videos of the product that he or she wishes to buy; there is no method for the buyer to virtually interact with the product. Secondly, e-commerce websites can be difficult to use for the disabled and elderly, who are less familiar with latest technology. Thus, the purpose of this research was to provide fast and time efficient code-based solution that will allow people who are unfamiliar with new technologies to use websites and apps using dynamic hand gestures. It is a live project that will be utilized for shopping on the firms' e-commerce websites. We used Pytorch, a machine learning library, to create the gesture detection and recognition system which is capable of recognizing and detecting various human gesture, such as swiping up hand, swiping down hand, turning fist clockwise, etc. All of these motions are pre- linked to various sorts of tasks. The images of these gestures were utilized to train and develop a machine learning model, which, when applied, recognized and executed the pre-linked functions for the taught human gestures.

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem.

Artificial intelligence, specifically machine learning, has been applied in a variety of methods by the research group to transform several data sources into valuable facts and understanding, allowing for superior pattern identification skills. Machine learning algorithms on huge and complicated data sets, computationally expensive on the other hand, processing requires hardware and logical resources, such as space, CPU, and memory. As the amount of data created daily reaches quintillion bytes, A complex big data infrastructure becomes more and more relevant. Apache Spark Machine learning library (ML-lib) is a famous platform used for big data analysis, it includes several useful features for machine learning applications, involving regression, classification, and dimension reduction, as well as clustering and features extraction. In this contribution, we consider Apache Spark ML-lib as a computationally independent machine learning library, which is open-source, distributed, scalable, and platform. We have evaluated and compared several ML algorithms to analyze the platform’s qualities, compared Apache Spark ML-lib against Rapid Miner and Sklearn, which are two additional Big data and machine learning processing platforms. Logistic Classifier (LC), Decision Tree Classifier (DTc), Random Forest Classifier (RFC), and Gradient Boosted Tree Classifier (GBTC) are four machine learning algorithms that are compared across platforms. In addition, we have tested general regression methods such as Linear Regressor (LR), Decision Tree Regressor (DTR), Random Forest Regressor (RFR), and Gradient Boosted Tree Regressor (GBTR) on SUSY and Higgs datasets. Moreover, We have evaluated the unsupervised learning methods like K-means and Gaussian Mixer Models on the data set SUSY and Hepmass to determine the robustness of PySpark, in comparison with the classification and regression models. We used ”SUSY,” ”HIGGS,” ”BANK,” and ”HEPMASS” dataset from the UCI data repository. We also talk about recent developments in the research into Big Data machines and provide future research directions.

Toward Backdoor Attacks for Image Captioning Model in Deep Neural Networks

Deep neural networks perform well in image recognition, speech recognition, and text recognition fields. The image caption model provides captions for images by generating text after image recognition. After extracting features from the original image, this model generates a representation vector and provides captions for the image by generating text through a recursive neural network. However, this image caption model has weaknesses in the backdoor sample. In this paper, we propose a method for generating backdoor samples for image caption models. By adding a specific trigger to the original sample, this proposed method creates a backdoor sample that is misrecognized as a target class by the target model. The MS-COCO dataset was used as the experimental dataset, and Tensorflow was used as the machine learning library. When the trigger size of the backdoor sample is 4%, experimental results show that the average attack success rate of the backdoor sample is 96.67%, and the average error rate of the original sample is 9.65%.

Consideration of the possibilities of applying machine learning methods for data analysis when promoting services to bank's clients

The object of the research is modern online services and machine learning libraries for predicting the probability of the bank client's consent to the provision of the proposed services. One of the most problematic areas is the high unpredictability of the result in the field of banking marketing using the most common technique of introducing new services for clients – the so-called cold calling. Therefore, the question of assessing the probability and predicting the behavior of a potential client when promoting new banking services and services using cold calling is particularly relevant. In the course of the study, libraries of machine learning methods and data analysis of the Python programming language were used. A program was developed to build a model for predicting the behavior of bank customers using data processing methods using gradient boosting, regularization of gradient boosting, random forest algorithm and recurrent neural networks. Analogous models were built using cloud machine learning services Azure ML, BigML and the Auto-sklearn library. Data analysis and prediction models built using Python language libraries have a fairly high quality – an average of 94.5 %. Using the Azure ML cloud service, a predictive model with an accuracy of 88.6 % was built. The BigML machine learning service made it possible to build a model with an accuracy of 88.8 %. Machine learning methods from the Auto-sklearn library made it possible to obtain a model with a higher quality – 94.9 %. This is due to the fact that the proposed libraries of the Python programming language allow better customization of data processing methods and machine learning to obtain more accurate models than free cloud services that do not provide such capabilities. Thanks to this, it is possible to obtain a predictive model of the behavior of bank customers with a fairly high degree of accuracy. It is worth noting that in order to make a prediction (forecast), it is necessary to study the context of the task, process the data, build various machine learning algorithms, evaluate the quality of the models and choose the best of them.

New Drug Development and Clinical Trial Design by Applying Genomic Information Management.

Depending on the patients’ genotype, the same drug may have different efficacies or side effects. With the cost of genomic analysis decreasing and reliability of analysis methods improving, vast amount of genomic information has been made available. Several studies in pharmacology have been based on genomic information to select the optimal drug, determine the dose, predict efficacy, and prevent side effects. This paper reviews the tissue specificity and genomic information of cancer. If the tissue specificity of cancer is low, cancer is induced in various organs based on a single gene mutation. Basket trials can be performed for carcinomas with low tissue specificity, confirming the efficacy of one drug for a single gene mutation in various carcinomas. Conversely, if the tissue specificity of cancer is high, cancer is induced in only one organ based on a single gene mutation. An umbrella trial can be performed for carcinomas with a high tissue specificity. Some drugs are effective for patients with a specific genotype. A companion diagnostic strategy that prescribes a specific drug for patients selected with a specific genotype is also reviewed. Genomic information is used in pharmacometrics to identify the relationship among pharmacokinetics, pharmacodynamics, and biomarkers of disease treatment effects. Utilizing genomic information, sophisticated clinical trials can be designed that will be better suited to the patients of specific genotypes. Genomic information also provides prospects for innovative drug development. Through proper genomic information management, factors relating to drug response and effects can be determined by selecting the appropriate data for analysis and by understanding the structure of the data. Selecting pre-processing and appropriate machine-learning libraries for use as machine-learning input features is also necessary. Professional curation of the output result is also required. Personalized medicine can be realized using a genome-based customized clinical trial design.

Intelligent ensembling of auto-ML system outputs for solving classification problems

Automatic Machine Learning (Auto-ML) tools enable the automatic solution of real-world problems through machine learning techniques. These tools tend to be more time consuming than standard machine learning libraries, therefore, exploiting all the available resources to the full is a valuable feature. This paper presents a two-phase optimization system for solving classification problems. The system is designed to produce more robust classifiers by exploiting the different architectures that are generated while solving classification problems with Auto-ML tools, particularly AutoGOAL. In the first phase, the system follows a probabilistic strategy to find the best combination of algorithms and hyperparameters to generate a collection of base models according to certain diversity criteria; and in the second, it follows similar Auto-ML strategies to ensemble those models. The HAHA 2019 challenge corpus and the Adult dataset were used to evaluate the system. The experimental results show that: i) a better solution can be built by ensembling a subset of the already tested models; ii) the performance of ensemble methods depends on the collection of base models used; and, iii) ensuring diversity using the double-fault measure produces better results than the disagreement measure. The source code is available online for the research community.

Scalable and Fast Characteristic Mode Analysis using GPUs

Characteristic mode analysis (CMA) is used in the design and analysis of a wide range of electromagnetic devices such as antennas and nanostructures. The implementation of CMA involves the evaluation of a large method of moments (MoM) complex impedance matrix at every frequency. In this work, we use different open-source software for the GPU acceleration of the CMA. This open-source software comprises a wide range of computer science numerical and machine learning libraries not typically used for electromagnetic applications. Specifically, this paper shows how these different Python-based libraries can optimize the computational time of the matrix operations that compose the CMA algorithm. Based on our computational experiments and optimizations, we propose an approach using a GPU platform that is able to achieve up to 16×× and 26×× speedup for the CMA processing of a single 15k ×× 15k MoM matrix of a perfect electric conductor scatterer and a single 30k ×× 30k MoM matrix of a dielectric scatterer, respectively. In addition to improving the processing speed of CMA, our approach provided the same accuracy as independent CMA simulations. The speedup, efficiency, and accuracy of our CMA implementation will enable the analysis of electromagnetic systems much larger than what was previously possible at a fraction of the computational time.

Restricted-Area Adversarial Example Attack for Image Captioning Model

Deep neural networks provide good performance in the fields of image recognition, speech recognition, and text recognition. For example, recurrent neural networks are used by image captioning models to generate text after an image recognition step, thereby providing captions for the images. The image captioning model first extracts features from the image and generates a representation vector; it then generates the text for the image captions by using the recursive neural network. This model has a weakness, however: it is vulnerable to adversarial examples. In this paper, we propose a method for generating restricted adversarial examples that target image captioning models. By adding a minimal amount of noise just to a specific area of an original sample image, the proposed method creates an adversarial example that remains correctly recognizable to humans yet is misinterpreted by the target model. We evaluated the method’s performance through experiments with the MS COCO dataset and using TensorFlow as the machine learning library. The results show that the proposed method generates a restricted adversarial example that is misinterpreted by the target model while minimizing its distortion from the original sample.

Vehicle Detection and Count in the Captured Stream Video Using Opencv in Machine Learning

The technology of detection within the captured video has implementation within the sort of fields. This emerging technology when implemented over the real-time video feeds could even be beneficial. The supreme good thing about vehicle detection within the real-time streaming video feed is to trace vehicles in busy roads or Bridges like Padma or Jamuna Bridge. An accidents occurred anywhere which may rather be detected. Vehicle detection also called computer vision beholding, basically the scientific methods and ways of how machines see instead of human eyes. This chapter aims to explore the prevailing challenging issue within the planet of unsupervised surveillance and security, Helps traffic police, Maintaining records and Traffic surveillance control. The detection of vehicles is implemented with enhanced algorithms and machine learning libraries like OpenCV, TensorFlow, and others. The varied approaches are accustomed identify and track the particular object through the trained model from the captured video.

Prediction of COVID-19 Using Some Machine Learning Models and Its Comparison with a Deep Learning Model

Coronavirus (COVID-19) started from Wuhan, China in December 2019. Since then, this virus has affected millions of people around the world and has caused deaths in millions. As of right now, there is no cure or permanent treatment for this disease. It is well known that machine learning plays an important role in the health care system. In this research, we are going to use some machine learning models such as Decision Tree (DT), logistic regression (LR), and Random Forest (RF) for the forecasting of corona virus. These models are implemented using different machine learning libraries available in Python. This work not only serves the purpose of COVID-19 predictions using machine learning, but also attempts to find out suitable model with the best features to save time and resources. Furthermore, we also compare some of the features of different machine learning models with a deep learning model (CNN). Since healthcare environment, computational resources have to be optimized, prediction models which use less computational resources are always preferred. We believe that the outcomes of this study can help understand the performance of various predictions models in the prediction of COVID-19.