Machine Learning Practitioners Research Articles

Return on Investment in Machine Learning: Crossing the Chasm between Academia and Business

Abstract Academia remains the central place of machine learning education. While academic culture is the predominant factor influencing the way we teach machine learning to students, many practitioners question this culture, claiming the lack of alignment between academic and business environments. Drawing on professional experiences from both sides of the chasm, we describe the main points of contention, in the hope that it will help better align academic syllabi with the expectations towards future machine learning practitioners. We also provide recommendations for teaching of the applied aspects of machine learning.

Democratizing AI: non-expert design of prediction tasks.

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

Submodularity in Action: From Machine Learning to Signal Processing Applications

Submodularity is a discrete domain functional property that can be interpreted as mimicking the role of well-known convexity/concavity properties in the continuous domain. Submodular functions exhibit strong structure that lead to efficient optimization algorithms with provable near-optimality guarantees. These characteristics, namely, efficiency and provable performance bounds, are of particular interest for signal processing (SP) and machine learning (ML) practitioners, as a variety of discrete optimization problems are encountered in a wide range of applications. Conventionally, two general approaches exist to solve discrete problems: 1) relaxation into the continuous domain to obtain an approximate solution or 2) the development of a tailored algorithm that applies directly in the discrete domain. In both approaches, worst-case performance guarantees are often hard to establish. Furthermore, they are often complex and thus not practical for large-scale problems. In this article, we show how certain scenarios lend themselves to exploiting submodularity for constructing scalable solutions with provable worst-case performance guarantees. We introduce a variety of submodular-friendly applications and elucidate the relation of submodularity to convexity and concavity, which enables efficient optimization. With a mixture of theory and practice, we present different flavors of submodularity accompanying illustrative real-world case studies from modern SP and ML. In all of the cases, optimization algorithms are presented along with hints on how optimality guarantees can be established.

The Number of Confirmed Cases of Covid-19 by using Machine Learning: Methods and Challenges.

Covid-19 is one of the biggest health challenges that the world has ever faced. Public health policy makers need the reliable prediction of the confirmed cases in future to plan medical facilities. Machine learning methods learn from the historical data and make predictions about the events. Machine learning methods have been used to predict the number of confirmed cases of Covid-19. In this paper, we present a detailed review of these research papers. We present a taxonomy that groups them in four categories. We further present the challenges in this field. We provide suggestions to the machine learning practitioners to improve the performance of machine learning methods for the prediction of confirmed cases of Covid-19.

Prudence when assuming normality: An advice for machine learning practitioners

In a binary classification problem the feature vector (predictor) is the input to a scoring function that produces a decision value (score), which is compared to a particular chosen threshold to provide a final class prediction (output). Although the normal assumption of the scoring function is important in many applications, sometimes it is severely violated even under the simple multinormal assumption of the feature vector. This article proves this result mathematically with a counterexample to provide an advice for practitioners to avoid blind assumptions of normality. On the other hand, the article provides a set of experiments that illustrate some of the expected and well-behaved results of the Area Under the ROC curve (AUC) under the multinormal assumption of the feature vector. Therefore, the message of the article is not to avoid the normal assumption of either the input feature vector or the output scoring function; however, a prudence is needed when adopting either of both.

Linear Algebra and Learning From Data [Bookshelf

This book contains the key linear algebra and optimization techniques at the forefront of active data-science and machine learning practice today. This is an appropriate choice of content because while state-of-the-art machine learning applications can change each month (as in reinforcement learning, language translation, game playing, or image classification), the underlying mathematical concepts and algorithms do not. The text is an offspring of a current Massachusetts Institute of Technology (MIT) mathematics course, Matrix Methods in Data Analysis and Signal Processing, which, in turn, was greatly influenced by a current University of Michigan electrical engineering and computer science course, Matrix Methods for Signal Processing, Data Analysis, and Machine Learning. Both courses are aimed at advanced undergraduate and graduate students in science and engineering. The book is broken down into seven parts. Part 1 presents highlights of linear algebra and covers a variety of basic matrix concepts, starting with matrix multiplication and moving up to singular values, principal components, low-rank approximations, and matrix/tensor factorizations. Part 2 discusses computation with large matrices, focusing on matrix factorizations, iterative methods, and recent powerful algorithms for approximating matrix problem solutions using randomization and projection. Due to the writing style, this is one of those books that you can open to almost any page and find a topic you didn’t know about or understand. After reading a few pages, it is quite likely that you will learn something new and useful. To me, such a resource has great value above and beyond its use solely as a textbook. I believe this textbook would be valuable for a matrix methods class oriented toward signal processing and machine learning. It would also be useful as a reference for someone seeking to be self-taught or review material learned years ago (but not used recently). Given that there is no discussion of system identification or recurrent networks for learning systems or automata, the book may be slightly less appealing to control theorists. Overall, a book of this kind is a very ambitious project. It covers many topics, several of which are typically reserved for dedicated graduate courses. Consequently, some depth must be sacrificed when such breadth is attempted. To help readers delve deeper, a more coherent approach to presenting references would be useful. As it stands, some references are embedded in the main text, while others are listed at the end of a section or chapter, but with inconsistent cross references. Some sections simply do not reference anything for further reading. Although one might argue that this is less of a problem (because Google Scholar and other web searches can help find original articles quite efficiently) it would still be helpful to get pointers about where to start. As this textbook becomes more widely used and the material continues to evolve, I believe a second edition that combines Jupyter Notebook with Matlab, Python, or TensorFlow examples would be a very welcome addition, consistent with the spirit of how machine learning practitioners actually work. In summary, this text is a valuable foundational asset for teaching, learning, and reference and is surely worth considering for your classroom and/or personal library.

Feature representations using the reflected rectified linear unit (RReLU) activation

Deep Neural Networks (DNNs) have become the tool of choice for machine learning practitioners today. One important aspect of designing a neural network is the choice of the activation function to be used at the neurons of the different layers. In this work, we introduce a four-output activation function called the Reflected Rectified Linear Unit (RReLU) activation which considers both a feature and its negation during computation. Our activation function is “sparse”, in that only two of the four possible outputs are active at a given time. We test our activation function on the standard MNIST and CIFAR-10 datasets, which are classification problems, as well as on a novel Computational Fluid Dynamics (CFD) dataset which is posed as a regression problem. On the baseline network for the MNIST dataset, having two hidden layers, our activation function improves the validation accuracy from 0.09 to 0.97 compared to the well-known ReLU activation. For the CIFAR-10 dataset, we use a deep baseline network that achieves 0.78 validation accuracy with 20 epochs but overfits the data. Using the RReLU activation, we can achieve the same accuracy without overfitting the data. For the CFD dataset, we show that the RReLU activation can reduce the number of epochs from 100 (using ReLU) to 10 while obtaining the same levels of performance.

Identification of topical subpopulations on social media

We tackle a major challenge of information filtering on social media (SM): rather than address the general question “what are people talking about on SM?”, we consider a finer question, “what are ... talking about on SM?”, where ... stands for some subpopulation of SM users of interest. We take a set expansion approach, where a seed of example members of the target subpopulation is initially defined, and additional SM users who belong to that subpopulation are identified, thus enabling the effective tracking of relevant information that pertains to that subpopulation on SM. Specifically, the Personalized PageRank (PPR) random walk measure is iteratively applied to detect additional members of the subpopulation based on their structural similarity to the seed set within the social media graph. There are several main contributions of this work. We outline Splash PPR, an efficient distributed computation of PPR adapted for potentially large seed sets and very large SM graphs. Using Splash PPR, we examine and tune graph representations towards the retrieval of two subpopulations from Twitter, namely human rights Activists, and Machine Learning practitioners. We believe this work is first to introduce and evaluate a generic framework for subpopulation identification at scale.

A Stochastic Derivative-Free Optimization Method with Importance Sampling: Theory and Learning to Control

We consider the problem of unconstrained minimization of a smooth objective function in ℝn in a setting where only function evaluations are possible. While importance sampling is one of the most popular techniques used by machine learning practitioners to accelerate the convergence of their models when applicable, there is not much existing theory for this acceleration in the derivative-free setting. In this paper, we propose the first derivative free optimization method with importance sampling and derive new improved complexity results on non-convex, convex and strongly convex functions. We conduct extensive experiments on various synthetic and real LIBSVM datasets confirming our theoretical results. We test our method on a collection of continuous control tasks on MuJoCo environments with varying difficulty. Experiments show that our algorithm is practical for high dimensional continuous control problems where importance sampling results in a significant sample complexity improvement.

Optimization frameworks for machine learning: Examples and case study

Abstract Mathematical optimization is at the algorithmic core of machine learning. Almost any known algorithm for solving mathematical optimization problems has been applied in machine learning and the machine learning community itself is actively designing and implementing new algorithms for specific problems. These implementations have to be made available to machine learning practitioners which is mostly accomplished by distributing them as standalone software. Successful well-engineered implementations are collected in machine learning toolboxes that provide a more uniform access to the different solvers. A disadvantage of the toolbox approach is a lack of flexibility as toolboxes only provide access to a fixed set of machine learning models that cannot be modified. This can be a problem for the typical machine learning workflow that iterates the process of modeling, solving and validating. If a model does not perform well on validation data, it needs to be modified. In most cases these modifications require a new solver for the entailed optimization problems. Optimization frameworks that combine a modeling language for specifying optimization problems with a solver are better suited to the iterative workflow since they allow to address large problem classes. Here, we provide examples of the use of optimization frameworks in machine learning. We also illustrate the use of one such framework in a case study that follows the typical machine learning workflow.

Nonlinear retinal response modeling for future neuromorphic instrumentation

The development of a bio-inspired image sensor that can match the functionality of the vertebrate retina in terms of image resolution, power dissipation and dynamic resolution demands an extremely challenging standard in terms of efficiency and performance [1]–[4]. At present, functional biological image processing possesses extremely high temporal resolution by asynchronously optimizing the sampling rate of a scene but suffers from low spatial resolution, noise, and an inability to process low frequency content [5]. Regardless, the use of dynamic vision sensors that emulate biological retina behaviors are important in performing classification tasks using high-speed motion detection at low-power requirements [6]–[9]. The difficulty in pushing image sensor performance to meet the specifications of the retina lies in the biological complexities of the underlying neural networks. Therefore, a prerequisite for hardware mapping of biological vision systems is understanding the bioprocessing between various retina cells. Here, we present a biologically plausible cellular network retina simulator, where light stimuli are accepted at the front-end array of photoreceptor cells, and pass through a system of nonlinear integral equations derived from experimental voltage- and current-clamp data used to calculate the response of each sequential retina cell. It was shown in [10], [11] that integrating the system improves performance when compared with conventional numerical solvers. We expect that by placing our simulator in the hands of interdisciplinary researchers, especially computational neuroscientists, mathematical scientists and machine learning practitioners, it will foster the development of more efficient representations of visual inputs in deep learning applications.

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of “unrolling” an RNN is routinely presented without justification throughout the literature. The goal of this tutorial is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in Signal Processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the “Vanilla LSTM”11The nickname “Vanilla LSTM” symbolizes this model’s flexibility and generality (Greff et al., 2015). network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this treatise valuable as well.

A Cloud-Based Framework for Machine Learning Workloads and Applications

In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models.

A Human Factors Engineering Education Perspective on Data Science, Machine Learning and Automation

The explosion of data science (DS) in all areas of technology coupled with the rapid growth of machine learning (ML) techniques in the last decade create novel applications in automation. Many working with DS techniques rely on the concept of “black boxes” to explain how ML works, noting that algorithms find patterns in the data that humans might not. While the mathematics are still being developed, the implications for the application of ML, specifically to questions of automation, also are being studied, but still remain poorly understood. The decisions made by ML practitioners with respect to data selection, model training and testing, data visualization, and model applications remain relatively unconstrained and have the potential to yield unexpected results at the systems level. Unfortunately, human factors engineers concerned with automation often have limited training and awareness of DS and ML applications and are unable to provide the meaningful guidance that is needed to ensure the future safety of these newly emerging automated systems. Moreover, undergraduate and graduate programs in human factors engineering (HFE) have not kept pace with these developments and future HFEs may continue to find themselves unable to contribute meaningfully to the development of automated systems based on algorithms derived from ML. In this paper, human factors engineers and educators explore some of the challenges to our understanding of automation posed by specific ML techniques and contrast this with an outline of some of the historical work in HFE that has contributed to our understanding of safe and effective automation. Examples are provided from more conventional applications using both supervised and unsupervised learning techniques, that are explored with respect to implications for algorithm performance, use in system automation, and the potential for unintended results. Implications for human factors engineering education are discussed.

DeepCompare: Visual and Interactive Comparison of Deep Learning Model Performance

Deep learning models have become the state-of-the-art for many tasks, from text sentiment analysis to facial image recognition. However, understanding why certain models perform better than others or how one model learns differently than another is often difficult yet critical for increasing their effectiveness, improving prediction accuracy, and enabling fairness. Traditional methods for comparing models' efficacy, such as accuracy, precision, and recall provide a quantitative view of performance; however, the qualitative intricacies of why one model performs better than another are hidden. In this paper, we interview machine learning practitioners to understand their evaluation and comparison workflow. From there, we iteratively design a visual analytic approach, DeepCompare, to systematically compare the results of deep learning models, in order to provide insight into the model behavior and interactively assess tradeoffs between two such models. The tool allows users to evaluate model results, identify and compare activation patterns for misclassifications, and link the test results back to specific neurons. We conduct a preliminary evaluation through two real-world case studies to show that experts can make more informed decisions about the effectiveness of different types of models, understand in more detail the strengths and weaknesses of the models, and holistically evaluate the behavior of the models.

Review of second-order optimization techniques in artificial neural networks backpropagation

Second-order optimization technique is the advances of first-order optimization in neural networks. It provides an addition curvature information of an objective function that adaptively estimate the step-length of optimization trajectory in training phase of neural network. With the additional information, it reduces training iteration and achieves fast convergence with less tuning of hyper-parameter. The current improved memory allocation and computing power further motivates machine learning practitioners to revisit the benefits of second-order optimization techniques. This paper covers the review on second-order optimization techniques that involve Hessian calculation for neural network training. It reviews the basic theory of Newton method, quasi-Newton, Gauss-Newton, Levenberg-Marquardt, Approximate Greatest Descent and Hessian-Free optimization. This paper summarizes the feasibility and performance of optimization techniques in deep neural network training. Comments and suggestions are highlighted for second-order optimization techniques in artificial neural network training in term of advantages and limitations.

Machine Learning Classification Models for Banking Domain

The prime aim of machine learning is to develop techniques or methods which automatically detect patterns in the given data, and later to make use the discovered pattern to predict future or other results of interest. Machine learning practitioners around the globe are trying to model the huge variety of data of varying levels of complexity using their diverse tools and methods. The models thus built are representations of the observed data which will identify some the regularities or some interesting patterns. In this paper, models constructed are based on classification which is a function that maps or classifies a data instance into one of several predefined class labels. A separate testing set is used to test the classifying ability of the learned model or function which uses the features such as Term frequency- Inverse Document Frequency (TF-IDF). In recent years Artificial Intelligence has impacted the Banking industry to a great extent. For organizations working in the banking industry, it has become increasingly crucial to keep up with competition, and increase their standing as an innovative company. In this work, the machine learning classification models are generated for the Banking domain to improve the system by considering and involving the feedback provided by customers, by collecting the text data related to banking from the web. Web crawlers are used on target specific sites to collect raw data.

Analyses of Classifier’s Performance Measures Used in Software Fault Prediction Studies

Assessing the quality of the software is both important and difficult. For this purpose, software fault prediction (SFP) models have been extensively used. However, selecting the right model and declaring the best out of multiple models are dependent on the performance measures. We analyze 14 frequently used, non-graphic classifier’s performance measures used in SFP studies. These analyses would help machine learning practitioners and researchers in SFP to select the most appropriate performance measure for the models’ evaluation. We analyze the performance measures for resilience against producing invalid values through our proposed plausibility criterion. After that, consistency and discriminancy analyses are performed to find the best out of the 14 performance measures. Finally, we draw the order of the selected performance measures from better to worse in both balance and imbalance datasets. Our analyses conclude that the F-measure and the G-mean1 are equally the best candidates to evaluate the SFP models with careful analysis of the result, as there is a risk of invalid values in certain scenarios.

MulNet: A Flexible CNN Processor With Higher Resource Utilization Efficiency for Constrained Devices

Leveraging deep convolutional neural networks (DCNNs) for various application areas has become a recent inclination of many machine learning practitioners due to their impressive performance. Research trends show that the state-of-the-art networks are getting deeper and deeper and such networks have shown significant performance increase. Deeper and larger neural networks imply the increase in computational intensity and memory footprint. This is particularly a problem for inference-based applications on resource constrained computing platforms. On the other hand, field-programmable gate arrays (FPGAs) are becoming a promising choice in giving hardware solutions for most deep learning implementations due to their high-performance and low-power features. With the rapid formation of various state-of-the-art CNN architectures, a flexible CNN hardware processor that can handle different CNN architectures and yet customize itself to achieve higher resource efficiency and optimum performance is critically important. In this paper, a novel and highly flexible DCNN processor, MulNet, is proposed. MulNet can be used to process most regular state-of-the-art CNN variants aiming at maximizing resource utilization of a target device. A processing core with multiplier and without multiplier is employed to achieve that. We formulated optimum fixed-point quantization format for MulNet by analyzing layer-by-layer quantization error. We also created a power-of-2 quantization for multiplier-free (MF) processing core of MulNet. Both quantizations significantly reduced the memory space needed and the logic consumption in the target device. We utilized Xilinx Zynq SoCs to leverage the one die hybrid (CPU and FPGA) architecture. We devised a scheme that utilizes Zynq processing system (PS) for memory intensive layers and the Zynq programmable logic (PL) for computationally intensive layers. We implemented modified LeNet, CIFAR-10 full, ConvNet processor (CNP), MPCNN, and AlexNet to evaluate MulNet. Our architecture with MF processing cores shows the promising result, by saving 36%-72% on-chip memory and 10%-44% DSP48 IPs, compared to the architecture with cores implemented using the multiplier. Comparison with the state of the art showed a very promising 25-$40\times $ DSP48 and 25-$29\times $ on-chip memory reduction with up to 136.9-GOP/s performance and 88.49-GOP/s/W power efficiency. Hence, our results demonstrate that the proposed architecture can be very expedient for resource constrained devices.

A Study of the Impact of Base Traditional Learners on Transfer Learning Algorithms

A transfer learning environment is characterized by not having sufficient labeled training data from the domain of interest (target domain) to build a high-performing machine learner. Transfer learning algorithms use labeled data from an alternate domain (source domain), that is similar to the target domain, to build high-performing learners. The design of a transfer learning algorithm is typically comprised of a domain adaptation step following by a learning step. The domain adaptation step attempts to align the distribution differences between the source domain and the target domain. Then, the aligned data from the domain adaptation step is used in the learning step, which is typically implemented with a traditional machine learning algorithm. Our research studies the impact of the learning step on the performance of various transfer learning algorithms. In our experiment, we use five unique domain adaptation methods coupled with seven different traditional machine learning methods to create 35 different transfer learning algorithms. We perform comparative performance analyses of the 35 transfer learning algorithms, along with the seven stand-alone traditional machine learning methods. This research will aid machine learning practitioners in the algorithm selection process for a transfer learning environment in the absence of reliable validation techniques.