Complex Machine Learning Tasks Research Articles

PyCaret for Predicting Type 2 Diabetes: A Phenotype- and Gender-Based Approach with the "Nurses' Health Study" and the "Health Professionals' Follow-Up Study" Datasets.

Predicting type 2 diabetes mellitus (T2DM) by using phenotypic data with machine learning (ML) techniques has received significant attention in recent years. PyCaret, a low-code automated ML tool that enables the simultaneous application of 16 different algorithms, was used to predict T2DM by using phenotypic variables from the "Nurses' Health Study" and "Health Professionals' Follow-up Study" datasets. Ridge Classifier, Linear Discriminant Analysis, and Logistic Regression (LR) were the best-performing models for the male-only data subset. For the female-only data subset, LR, Gradient Boosting Classifier, and CatBoost Classifier were the strongest models. The AUC, accuracy, and precision were approximately 0.77, 0.70, and 0.70 for males and 0.79, 0.70, and 0.71 for females, respectively. The feature importance plot showed that family history of diabetes (famdb), never having smoked, and high blood pressure (hbp) were the most influential features in females, while famdb, hbp, and currently being a smoker were the major variables in males. In conclusion, PyCaret was used successfully for the prediction of T2DM by simplifying complex ML tasks. Gender differences are important to consider for T2DM prediction. Despite this comprehensive ML tool, phenotypic variables alone may not be sufficient for early T2DM prediction; genotypic variables could also be used in combination for future studies.

Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS

Machine Learning (ML) has become one of the important tools for High Energy Physics analysis. As the size of the dataset increases at the Large Hadron Collider (LHC), and at the same time the search spaces become bigger and bigger in order to exploit the physics potentials, more and more computing resources are required for processing these ML tasks. In addition, complex advanced ML workflows are developed in which one task may depend on the results of previous tasks. How to make use of vast distributed CPUs/GPUs in WLCG for these big complex ML tasks has become a popular research area. In this paper, we present our efforts enabling the execution of distributed ML workflows on the Production and Distributed Analysis (PanDA) system and intelligent Data Delivery Service (iDDS). First, we describe how PanDA and iDDS deal with large-scale ML workflows, including the implementation to process workloads on diverse and geographically distributed computing resources. Next, we report real-world use cases, such as HyperParameter Optimization, Monte Carlo Toy confidence limits calculation, and Active Learning. Finally, we conclude with future plans.

Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

Metrics optimized in complex machine learning tasks are often selected in an ad-hoc manner. It is unknown how they align with human expert perception. We explore the correlations between established quantitative segmentation quality metrics and qualitative evaluations by professionally trained human raters. Therefore, we conduct psychophysical experiments for two complex biomedical semantic segmentation problems. We discover that current standard metrics and loss functions correlate only moderately with the segmentation quality assessment of experts. Importantly, this effect is particularly pronounced for clinically relevant structures, such as the enhancing tumor compartment of glioma in brain magnetic resonance and grey matter in ultrasound imaging. It is often unclear how to optimize abstract metrics, such as human expert perception, in convolutional neural network (CNN) training. To cope with this challenge, we propose a novel strategy employing techniques of classical statistics to create complementary compound loss functions to better approximate human expert perception. Across all rating experiments, human experts consistently scored computer-generated segmentations better than the human-curated reference labels. Our results, therefore, strongly question many current practices in medical image segmentation and provide meaningful cues for future research.

Non-interactive and privacy-preserving neural network learning using functional encryption

Fast and reliable modeling of distributed data with sensitive information is a major goal of privacy-preserving machine learning (PPML). Artificial Neural Networks (ANNs) are powerful and scalable, making them suitable to settle large-scale and highly complex machine learning tasks. Since a large number of neurons and various non-linear functions are embedded in ANNs, it is still a huge challenge to train an ANNs model in the encrypted-domain. Majority of existing approaches for PPML require multiple communications among data owners and cloud servers, leading to a substantial overhead of computation and data transmission costs, and are restricted to approximation models by polynomials or piecewise linear functions. Here, to facilitate the integration of any training data from any data owner without violating privacy constraints, we introduce a non-interactive and lossless ANNs training approach that unites mask matrix, function encryption for inner-product and coordination while maintaining confidentiality with the help of Internet Service Providers (ISPs), thereby going beyond those schemes with interactive fashion and using approximation substitutions. To illustrate the feasibility and efficiency of using our method to develop ANNs classifiers using distributed data, we choose two well-known MNIST datasets with a large amount of image data with high dimensionality in PPML field. We show the protocol is feasible to use in practice while achieving high accuracy. Furthermore, the evaluation reveals that the computational efficiency is improved at least 25 times compared with the state-of-the-art privacy-preserving ANNs approach.

ReaLPrune: ReRAM Crossbar-Aware Lottery Ticket Pruning for CNNs

Training machine learning (ML) models at the edge (on-chip training on end user devices) can address many pressing challenges including data privacy/security, increase the accessibility of ML applications to different parts of the world by reducing the dependence on the communication fabric and the cloud infrastructure, and meet the real-time requirements of AR/VR applications. However, existing edge platforms do not have sufficient computing capabilities to support complex ML tasks such as training large CNNs. ReRAM-based architectures offer high-performance yet energy efficient computing platforms for on-chip CNN training/inferencing. However, ReRAM-based architectures are not scalable with the size of the CNN. Larger CNNs have more weights, which requires more ReRAM cells that cannot be integrated in a single chip. Moreover, training larger CNNs on-chip will require higher power, which cannot be afforded by these smaller devices. Pruning is an effective way to solve this problem. However, existing pruning techniques are either targeted for inferencing only, or they are not crossbar-aware. This leads to sub-optimal hardware savings and performance benefits for CNN training on ReRAM-based architectures. In this paper, we address this problem by proposing a novel crossbar-aware pruning strategy, referred as ReaLPrune, which can prune more than 90% of CNN weights. The pruned model can be trained from scratch without any accuracy loss. Experimental results indicate that ReaLPrune reduces hardware requirements by 77.2% and accelerates CNN training by ∼20× compared to unpruned CNNs. ReaLPrune also outperforms other crossbar-aware pruning techniques in terms of both performance and hardware savings. In addition, ReaLPrune is equally effective for diverse datasets and more complex CNNs.

Human Verification over Activity Analysis via Deep Data Mining

Human verification and activity analysis (HVAA) are primarily employed to observe, track, and monitor human motion patterns using red-green-blue (RGB) images and videos. Interpreting human interaction using RGB images is one of the most complex machine learning tasks in recent times. Numerous models rely on various parameters, such as the detection rate, position, and direction of human body components in RGB images. This paper presents robust human activity analysis for event recognition via the extraction of contextual intelligence-based features. To use human interaction image sequences as input data, we first perform a few denoising steps. Then, human-to-human analyses are employed to deliver more precise results. This phase follows feature engineering techniques, including diverse feature selection. Next, we used the graph mining method for feature optimization and AdaBoost for classification. We tested our proposed HVAA model on two benchmark datasets. The testing of the proposed HVAA system exhibited a mean accuracy of 92.15% for the Sport Videos in the Wild (SVW) dataset. The second benchmark dataset, UT-interaction, had a mean accuracy of 92.83%. Therefore, these results demonstrated a better recognition rate and outperformed other novel techniques in body part tracking and event detection. The proposed HVAA system can be utilized in numerous real-world applications including, healthcare, surveillance, task monitoring, atomic actions, gesture and posture analysis.

Solitonic Neural Network: A novel approach of Photonic Artificial Intelligence based on photorefractive solitonic waveguides

Neuromorphic models are proving capable of performing complex machine learning tasks, overcoming the structural limitations imposed by software systems and electronic neuromorphic models. Unlike computers, the brain uses a unified geometry whereby memory and computation occur in the same physical location. The neuromorphic approach tries to reproduce the functional blocks of biological neural networks. In the photonics field, one possible and efficient way is to use integrated circuits based on soliton waveguides, ie channels self-written by light. Thanks to the nonlinearity of some crystals, propagating light can write waveguides and then can modulate them according to the information it carries. Thus, the created structures are not static but they can self-modify by varying the input information pattern. These hardware systems show a neuroplasticity which is very close to the one which characterize the brain functioning. The solitonic neuromorphic paradigm this work introduces is based on X-junction solitonic neurons as the fundamental elements for complex neural networks. These solitonic units are able to learn information both in supervised and unsupervised ways by unbalancing the X-junction. The storage of information coincides with the evolution of structure that changes plastically. Thus, complex solitonic networks can store information as propagation trajectories and use them for reasoning.

Controlling chaotic itinerancy in laser dynamics for reinforcement learning.

Photonic artificial intelligence has attracted considerable interest in accelerating machine learning; however, the unique optical properties have not been fully used for achieving higher-order functionalities. Chaotic itinerancy, with its spontaneous transient dynamics among multiple quasi-attractors, can be used to realize brain-like functionalities. In this study, we numerically and experimentally investigate a method for controlling the chaotic itinerancy in a multimode semiconductor laser to solve a machine learning task, namely, the multiarmed bandit problem, which is fundamental to reinforcement learning. The proposed method uses chaotic itinerant motion in mode competition dynamics controlled via optical injection. We found that the exploration mechanism is completely different from a conventional searching algorithm and is highly scalable, outperforming the conventional approaches for large-scale bandit problems. This study paves the way to use chaotic itinerancy for effectively solving complex machine learning tasks as photonic hardware accelerators.

CoRoL: A Reliable Framework for Computation Offloading in Collaborative Robots

Collaborative robots (cobots) are becoming more prominent in the manufacturing industry due to their ability to operate outside safety zones and work in tandem with humans to perform precise and repetitive tasks, such as visual inspection, product categorization, quality control, etc. Cobots generally have limited computational resources that limit their ability to perform complex machine learning (ML) tasks and as such, various cloud- and fog-based computational task offloading mechanisms have been proposed. However, a growing reluctance to share manufacturing data on the cloud, cybersecurity concerns, and demand of agile decision making is encouraging researchers to design resource-sharing frameworks for on-floor cobots where they can share the execution of complex ML tasks. However, agile on-floor environments and the potential presence of malicious elements make reliable task offloading a significant challenge. This article investigates reliability issues and their effects on executing complex ML task executed by participating on-floor cobots. Specifically, this article aims to answer <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">whom to offload?</i> with the objective of ensuring reliability, security, and data protection when offloading computation tasks. To this end, a reputation-based collaborative robotic learning (CoRoL) framework is proposed with the ability to isolate and/or minimize the impact of malicious or poor-performing cobots on computation task execution. In addition, CoRoL is supported by split learning for privacy-preserving task offloading with minimum data exchange. Simulation results and comparative analysis will demonstrate CoRoL’s efficiency in terms of percentage of completed tasks, achieved accuracy, and impact on energy consumption.

Episodic Memory and Information Recognition Using Solitonic Neural Networks Based on Photorefractive Plasticity

Neuromorphic models are proving capable of performing complex machine learning tasks, overcoming the structural limitations imposed by software algorithms and electronic architectures. Recently, both supervised and unsupervised learnings were obtained in photonic neurons by means of spatial-soliton-waveguide X-junctions. This paper investigates the behavior of networks based on these solitonic neurons, which are capable of performing complex tasks such as bit-to-bit information memorization and recognition. By exploiting photorefractive nonlinearity as if it were a biological neuroplasticity, the network modifies and adapts to the incoming signals, memorizing and recognizing them (photorefractive plasticity). The information processing and storage result in a plastic modification of the network interconnections. Theoretical description and numerical simulation of solitonic networks are reported and applied to the processing of 4-bit information.

Privacy-preserving federated learning for scalable and high data quality computational-intelligence-as-a-service in Society 5.0.

Training supervised machine learning models like deep learning requires high-quality labelled datasets that contain enough samples from various categories and specific cases. The Data as a Service (DaaS) can provide this high-quality data for training efficient machine learning models. However, the issue of privacy can minimize the participation of the data owners in DaaS provision. In this paper, a blockchain-based decentralized federated learning framework for secure, scalable, and privacy-preserving computational intelligence, called Decentralized Computational Intelligence as a Service (DCIaaS), is proposed. The proposed framework is able to improve data quality, computational intelligence quality, data equality, and computational intelligence equality for complex machine learning tasks. The proposed framework uses the blockchain network for secure decentralized transfer and sharing of data and machine learning models on the cloud. As a case study for multimedia applications, the performance of DCIaaS framework for biomedical image classification and hazardous litter management is analysed. Experimental results show an increase in the accuracy of the models trained using the proposed framework compared to decentralized training. The proposed framework addresses the issue of privacy-preserving in DaaS using the distributed ledger technology and acts as a platform for crowdsourcing the training process of machine learning models.

LAAT: Locally Aligned Ant Technique for discovering multiple faint low dimensional structures of varying density

Dimensionality reduction and clustering are often used as preliminary steps for many complex machine learning tasks. The presence of noise and outliers can deteriorate the performance of such preprocessing and therefore impair the subsequent analysis tremendously. In manifold learning, several studies indicate solutions for removing background noise or noise close to the structure when the density is substantially higher than that exhibited by the noise. However, in many applications, including astronomical datasets, the density varies alongside manifolds that are buried in a noisy background. We propose a novel method to extract manifolds in the presence of noise based on the idea of Ant colony optimization. In contrast to the existing random walk solutions, our technique captures points that are locally aligned with major directions of the manifold. Moreover, we empirically show that the biologically inspired formulation of ant pheromone reinforces this behavior enabling it to recover multiple manifolds embedded in extremely noisy data clouds. The algorithm performance in comparison to state-of-the-art approaches for noise reduction in manifold detection and clustering is demonstrated, on several synthetic and real datasets, including an N-body simulation of a cosmological volume.

IChain: Peer-To-Peer Machine Learning Powered by Blockchain Technology

iChain is an application which was created to help meet the growing demand of machine learning. It allows users to pay those with powerful machines to run machine learning tasks for them, bypassing the need for a significant investment in a powerful computer to run it themselves. This is similar to services like a render farm. Our application functions using the Ethereum blockchain which ensures security and decentralization, as well as providing a platform for payment transactions. This article will discuss the background on machine learning and blockchain, the application, how it works, how the data moves through it, and how to use it. We hope our application will enable many without the funds to build or buy a powerful computer to experiment with and utilize complex machine learning tasks.

Verifiable Privacy-Preserving Neural Network on Encrypted Data

The widespread acceptance of machine learning, particularly of neural networks leads to great success in many areas, such as recommender systems, medical predictions, and recognition. It is becoming possible for any individual with a personal electronic device and Internet access to complete complex machine learning tasks using cloud servers. However, it must be taken into consideration that the data from clients may be exposed to cloud servers. Recent work to preserve data confidentiality has allowed for the outsourcing of services using homomorphic encryption schemes. But these architectures are based on honest but curious cloud servers, which are unable to tell whether cloud servers have completed the computation delegated to the cloud server. This paper proposes a verifiable neural network framework which focuses on solving the problem of data confidentiality and training integrity in machine learning. Specifically, we first leverage homomorphic encryption and extended diagonal packing method to realize a privacy-preserving neural network model efficiently, it enables the user training over encrypted data, thereby protecting the user’s private data. Then, considering the problem that malicious cloud servers are likely to return a wrong result for saving cost, we also integrate a training validation modular Proof-of-Learning, a strategy for verifying the correctness of computations performed during training. Moreover, we introduce practical byzantine fault tolerance to complete the verification progress without a verifiable center. Finally, we conduct a series of experiments to evaluate the performance of the proposed framework, the results show that our construction supports the verifiable training of PPNN based on HE without introducing much computational cost.

A semi-supervised deep learning image caption model based on Pseudo Label and N-gram

Image caption is an important application field of artificial intelligence technique. When a machine can describe a picture reasonably like a human, it represents that the machine has higher intelligence to understand the picture. However, for complex machine learning tasks such as image caption, data annotation is time-consuming and laborious. Usually in a new application scenario, data annotation rarely results in poor model performance. A large number of easily available unlabeled image data make it possible to semi-supervised learning of image caption methods. Based on the existing end-to-end deep learning paradigm, a semi-supervised deep learning method is proposed in this paper, called N-gram + Pseudo Label NIC method. The method combines the current mainstream deep neural network method, e.g. the NIC (Neural Image Caption) model, and the semi-supervised deep learning idea of pseudo labels, and N-gram. This method generates pseudo labels by N-gram Search algorithm, and improves the effect of the model by utilizing the prior knowledge of the N-gram table and people's descriptive habits. This method has achieved better results than the original NIC model on different sub-data sets of Flickr 8K data set and MSCOCO data set of 0.5k, 1k, 2k and 3k under BLEU-1 evaluation criteria.

Local Differential Privacy for data collection and analysis

Local Differential Privacy (LDP) can provide each user with strong privacy guarantees under untrusted data curators while ensuring accurate statistics derived from privatized data. Due to its powerfulness, LDP has been widely adopted to protect privacy in various tasks (e.g., heavy hitters discovery, probability estimation) and systems (e.g., Google Chrome, Apple iOS). In particular, (∊,δ)-LDP has been studied in related statistical tasks like private learning and hypothesis testing, but is mainly achieved by using Gaussian mechanism, leading to the limited data utility. In this paper, we investigate several novel mechanisms that achieve (∊,δ)-LDP with higher data utility in collecting and analyzing users’ data. Specifically, we first design two (∊,δ)-LDP algorithms for mean estimations on multi-dimensional numeric data, which can ensure higher accuracy than the optimal Gaussian mechanism. Then, we investigate different local protocols for frequency estimations on categorical attributes under (∊,δ)-LDP. Based on the proposed mechanisms, we further study on (∊,δ)-LDP-compliant stochastic gradient descent algorithms for machine learning models. Besides, the theoretical analysis of the error bound and the variance of the proposed algorithms are also presented in the paper. We have conducted extensive experiments on both real-world and synthetic datasets and demonstrated the high data utility of our proposed algorithms in the perspectives of simple data statistics tasks and complex machine learning tasks. The experimental results have shown that our proposed algorithms can effectively improve the data utility in different tasks while alleviating the privacy concerns of each individual.

DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

In the past decade deep neural networks (DNNs) have shown state-of-the-art performance on a wide range of complex machine learning tasks. Many of these results have been achieved while growing the size of DNNs, creating a demand for efficient compression and transmission of them. In this work we present DeepCABAC, a universal compression algorithm for DNNs that is based on applying Context-based Adaptive Binary Arithmetic Coder (CABAC) to the DNN parameters. CABAC was originally designed for the H.264/AVC video coding standard and became the state-of-the-art for the lossless compression part of video compression. DeepCABAC applies a novel quantization scheme that minimizes a rate-distortion function while simultaneously taking the impact of quantization to the DNN performance into account. Experimental results show that DeepCABAC consistently attains higher compression rates than previously proposed coding techniques for DNN compression. For instance, it is able to compress the VGG16 ImageNet model by x63.6 with no loss of accuracy, thus being able to represent the entire network with merely 9 MB. The source code for encoding and decoding can be found at https://github.com/fraunhoferhhi/DeepCABAC.

Robust image classification against adversarial attacks using elastic similarity measures between edge count sequences

Due to their unprecedented capacity to learn patterns from raw data, deep neural networks have become the de facto modeling choice to address complex machine learning tasks. However, recent works have emphasized the vulnerability of deep neural networks when being fed with intelligently manipulated adversarial data instances tailored to confuse the model. In order to overcome this issue, a major effort has been made to find methods capable of making deep learning models robust against adversarial inputs. This work presents a new perspective for improving the robustness of deep neural networks in image classification. In computer vision scenarios, adversarial images are crafted by manipulating legitimate inputs so that the target classifier is eventually fooled, but the manipulation is not visually distinguishable by an external observer. The reason for the imperceptibility of the attack is that the human visual system fails to detect minor variations in color space, but excels at detecting anomalies in geometric shapes. We capitalize on this fact by extracting color gradient features from input images at multiple sensitivity levels to detect possible manipulations. We resort to a deep neural classifier to predict the category of unseen images, whereas a discrimination model analyzes the extracted color gradient features with time series techniques to determine the legitimacy of input images. The performance of our method is assessed over experiments comprising state-of-the-art techniques for crafting adversarial attacks. Results corroborate the increased robustness of the classifier when using our discrimination module, yielding drastically reduced success rates of adversarial attacks that operate on the whole image rather than on localized regions or around the existing shapes of the image. Future research is outlined towards improving the detection accuracy of the proposed method for more general attack strategies.

Evaluating the Visualization of What a Deep Neural Network Has Learned.

Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the "importance" of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.Deep neural networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multilayer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision, given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the "importance" of individual pixels with respect to the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper, we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012, and MIT Places data sets. Our main result is that the recently proposed layer-wise relevance propagation algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of the neural network performance.

Exploring ant-based algorithms for gene expression data analysis

Recently, much research has been proposed using nature inspired algorithms to perform complex machine learning tasks. Ant colony optimization (ACO) is one such algorithm based on swarm intelligence and is derived from a model inspired by the collective foraging behavior of ants. Taking advantage of the ACO in traits such as self-organization and robustness, this paper investigates ant-based algorithms for gene expression data clustering and associative classification. An ant-based clustering (Ant-C) and an ant-based association rule mining (Ant-ARM) algorithms are proposed for gene expression data analysis. The proposed algorithms make use of the natural behavior of ants such as cooperation and adaptation to allow for a flexible robust search for a good candidate solution. Ant-C has been tested on the three datasets selected from the Stanford Genomic Resource Database and achieved relatively high accuracy compared to other classical clustering methods. Ant-ARM has been tested on the acute lymphoblastic leukemia (ALL)/acute myeloid leukemia (AML) dataset and generated about 30 classification rules with high accuracy. Ant-C can generate optimal number of clusters without incorporating any other algorithms such as K-means or agglomerative hierarchical clustering. For associative classification, while a few of the well-known algorithms such as Apriori, FP-growth and Magnum Opus are unable to mine any association rules from the ALL/AML dataset within a reasonable period of time, Ant-ARM is able to extract associative classification rules.