Large Historical Dataset Research Articles

Improving the Accuracy and Transparency of Underwriting with Artificial Intelligence to Transform the Life‐Insurance Industry

Life insurance provides trillions of dollars of financial security for hundreds of millions of individuals and families worldwide. To simultaneously offer affordable products while managing this financial ecosystem, life‐insurance companies use an underwriting process to assess the mortality risk posed by individual applicants. Traditional underwriting is largely based on examining an applicant's health and behavioral profile. This manual process is incompatible with expectations of a rapid customer experience through digital capabilities. Fortunately, the availability of large historical data sets and the emergence of new data sources provide an unprecedented opportunity for artificial intelligence to transform underwriting in the life‐insurance industry with standard measures of mortality risk. We combined one of the largest application data sets in the industry with a responsible artificial intelligence framework to develop a mortality model and life score. We describe how the life score serves as the primary risk‐driving engine of deployed algorithmic underwriting systems and demonstrate its high level of accuracy, yielding a nine‐percent reduction in claims within the healthiest pool of applicants. Additionally, we argue that, by embracing transparency, the industry can build consumer trust and respond to a dynamic regulatory environment focused on algorithmic decision‐making. We present a consumer‐facing tool that uses a state‐of‐the‐art method for interpretable machine learning to offer transparency into the life score.

EHRtemporalVariability: delineating temporal data-set shifts in electronic health records.

BackgroundTemporal variability in health-care processes or protocols is intrinsic to medicine. Such variability can potentially introduce dataset shifts, a data quality issue when reusing electronic health records (EHRs) for secondary purposes. Temporal data-set shifts can present as trends, as well as abrupt or seasonal changes in the statistical distributions of data over time. The latter are particularly complicated to address in multimodal and highly coded data. These changes, if not delineated, can harm population and data-driven research, such as machine learning. Given that biomedical research repositories are increasingly being populated with large sets of historical data from EHRs, there is a need for specific software methods to help delineate temporal data-set shifts to ensure reliable data reuse.ResultsEHRtemporalVariability is an open-source R package and Shiny app designed to explore and identify temporal data-set shifts. EHRtemporalVariability estimates the statistical distributions of coded and numerical data over time; projects their temporal evolution through non-parametric information geometric temporal plots; and enables the exploration of changes in variables through data temporal heat maps. We demonstrate the capability of EHRtemporalVariability to delineate data-set shifts in three impact case studies, one of which is available for reproducibility.ConclusionsEHRtemporalVariability enables the exploration and identification of data-set shifts, contributing to the broad examination and repurposing of large, longitudinal data sets. Our goal is to help ensure reliable data reuse for a wide range of biomedical data users. EHRtemporalVariability is designed for technical users who are programmatically utilizing the R package, as well as users who are not familiar with programming via the Shiny user interface.Availability: https://github.com/hms-dbmi/EHRtemporalVariability/Reproducible vignette: https://cran.r-project.org/web/packages/EHRtemporalVariability/vignettes/EHRtemporalVariability.htmlOnline demo: http://ehrtemporalvariability.upv.es/

Robust control charts for the mean of a locally linear time series

We study residual control charts for the detection of sudden changes in time series with an underlying time-varying trend. Our control charts compare the most recent observations to their direct predecessors by applying two-sample tests in a moving time window to one-step-ahead prediction errors obtained from a local linear regression. Global model assumptions, a fixed target value, or large sets of historical in-control data are unnecessary therefore. Disturbing in-control structures, like short outlier patches or minor fluctuations, are resisted by using robust methods. Our simulation results indicate that a control chart based on the two-sample Hodges-Lehmann estimator in combination with repeated-median regression possesses an approximately distribution-free in-control average run length, good robustness against outliers, and a desirable detection quality.

Private Portfolio Attribution Analysis

There are many methods for conducting performance attribution with portfolios containing only liquid assets. Efforts to conduct similar types of attribution analyses for portfolios of private equity funds (and other illiquid investments) have been thwarted by a lack of periodic asset return data and a clear definition of what constitutes an appropriate market benchmark. We propose a method for decomposing private fund portfolio performance into effects from timing, strategy selection, geographic focus, sizing of fund allocation, and fund selection attributes. We test the method with a simulation study and derive approximate confidence intervals for assessing attribute selection skill using a large historical dataset of buyout and venture capital funds.

The two types of society: Computationally revealing recurrent social formations and their evolutionary trajectories.

Comparative social science has a long history of attempts to classify societies and cultures in terms of shared characteristics. However, only recently has it become feasible to conduct quantitative analysis of large historical datasets to mathematically approach the study of social complexity and classify shared societal characteristics. Such methods have the potential to identify recurrent social formations in human societies and contribute to social evolutionary theory. However, in order to achieve this potential, repeated studies are needed to assess the robustness of results to changing methods and data sets. Using an improved derivative of the Seshat: Global History Databank, we perform a clustering analysis of 271 past societies from sampling points across the globe to study plausible categorizations inherent in the data. Analysis indicates that the best fit to Seshat data is five subclusters existing as part of two clearly delineated superclusters (that is, two broad "types" of society in terms of social-ecological configuration). Our results add weight to the idea that human societies form recurrent social formations by replicating previous studies with different methods and data. Our results also contribute nuance to previously established measures of social complexity, illustrate diverse trajectories of change, and shed further light on the finite bounds of human social diversity.

Mapping the Transformation of Information into Knowledge in Early Modern Florence

The recent use of geographic information systems (GIS) to visualize large historical datasets are particularly useful to History instructors who seek real-world platforms that support an open-ended investigation of past societies. The University of Toronto’s DECIMA Project (Digitally Encoded Census and Information Mapping Archive) presents an open-access platform through which anyone can explore census material collected in early modern Florence. This article explores how senior undergraduate students used the DECIMA Project to construct research-based learning assignments that they developed independently. A rubric assessed the resulting student work in order to evaluate several cognitive abilities that students employed during their method design, data analysis, and the contextualization of conclusions. The assignment also drew on central History disciplinary concepts (articulating a question, constructing an argument, evaluating sources, and discerning challenges to the narrative), while introducing students to 'real Florentines.'

Enhanced data reconciliation of freight rail dispatch data

In order to enable widespread use of data driven analysis for rail operations problems, large volumes of complete and clean data are needed. In this work a data reconciliation problem for rail dispatch data is proposed to automatically clean and complete noisy and incomplete data. The proposed method finds a minimally-perturbed modification of the observed historical data that satisfies operational constraints, such as feasibility of meet and overtake events. The method is demonstrated on a large historical dataset from freight rail territory in Tennessee, US, containing over 3000 train records over six months. The results show that data reconciliation reduces timing error of imputed points by up to 15% and increases the number of meet and overtake events estimated at the correct historical location from less than 40% to approximately 95%. It is also shown that regularizing the data reconciliation problem with historical train performance data further decreases the error of reconstructed points by 15%, and using an L2 normalization can reduce mean squared error by over 50%. These findings indicate that the data reconciliation method is a useful preprocessing step for analysis and modeling of railroad operations that are based on real-world physical dispatching data.

A Decentralized Three-Level Optimization Scheme for Optimal Planning of a Prosumer Nano-Grid

Rapid improvement in efficiency of renewable energy sources (RESs), increased concerns about carbon emission, substantial power-flow losses, and the emergence of smart grid have all contributed to increased interest toward customers’ contribution in electricity generation in RES-based prosumer nano-grids (PNGs). Such contributions within a PNG can be effectively controlled using dynamic pricing tariffs (DPTs) that hold customers accountable for their electric power exchange behaviors. Prosumer customers optimize their internal power grid based on utility-provided DPT and expect to surpass the break-even point within an anticipated period of time. However, their expected payback period may not be met, as the DPT changes over time as more customers join the market. Therefore, the joint design of PNG and DPT is necessary to give the customers the opportunity to revisit their investment plans, and, at the same time, the utility the opportunity to adjust the pricing scheme. We propose an iterative decentralized three-level optimization scheme to achieve both objectives. The proposed scheme can be applied to a large system for planning of optimal PNGs, while providing a reliable payback period on investment for customers that are involved. The effectiveness of the proposed scheme is verified numerically using a large historical data set.

Scalable Algorithms for Maximizing Spatiotemporal Range Sum and Range Sum Change in Spatiotemporal Datasets

In this paper, we introduce the three-dimensional Maximum Range-Sum (3D MaxRS) problem and the Maximum Spatiotemporal Range-Sum Change (MaxStRSC) problem. The 3D MaxRS problem tries to find the 3D range where the sum of weights across all objects inside is maximized, and the MaxStRSC problem tries to find the spatiotemporal range where the sum of weights across all objects inside is maximally increased. The goal of this paper is to provide efficient methods for data analysts to find interesting spatiotemporal regions in a large historical spatiotemporal dataset by addressing two problems. We provide a mathematical explanation for each problem and propose several algorithms for them. Existing methods tried to find the optimal region over two-dimensional datasets or to monitor a burst region over two-dimensional data streams. The majority of them cannot directly solve our problems. Although some existing methods can be used or modified to solve the 3D MaxRS problems, they have limited scalability. In addition, none of them can be used to solve the MaxStRS-RC problem (a type of MaxStRSC problem). Finally, we study the performance of the proposed algorithms experimentally. The experimental results show that the proposed algorithms are scalable and much more efficient than existing methods.

Towards cloud energy metering system with 32 bit FPGA device architecture

Cloud based Advanced Metering Infrastructure (CAMI) is the next digital future for energy management (EM). Various efforts on EM capabilities are mainly skewed towards embedded device architectures that support non-concurrent execution. This paper presents cloud energy metering system (CEMS) using high speed 32 bit field programmable gate array (FPGA) device. The architectural framework for energy tracking and profile measurement in CEMS is presented. This aims at accurate metering with demand side management (DSM). An application context that supports an EM architecture is highlighted. The contextual CEMS features energy monitoring in distributed energy utilities such as solar generators, wind and energy storage sources. Process integration with Cloud based Internet for real-time energy reading is achieved through the FPGA synthesis to provide end-to-end energy analytics. CEM prototype (Xilinx FPGA) running on a wireless open-access research platform supports management of large historical data-sets from the current data up to the last granular interval, hourly, daily, monthly and yearly dataset captures. The system provides the low latency datasets in both tabular and graphical forms for end-user visualization of energy consumption patterns. In the experimental setup, three case scenarios demonstrate how the metering system executes fast edge computing profiling, thereby providing data-visualization services to end-users. The results show the Avg. Latency time for CAMI household 1, 2 and 3 respectively. In case 1, the average latencies for actual and measured (proposed CAMI) are 75% and 25% respectively. In case 2 and case 3, this gave 66.67% and 33.33% respectively. Clearly, the proposed CAMI offers lower latency for all scenarios of energy consumption metering usage.Keywords: Advanced Metering, Energy management, Cyber-physical systems, Cloud Computing, IoT, Fog. 1.0 INTRODUCTION

A general anomaly detection framework for fleet-based condition monitoring of machines

Machine failures decrease up-time and can lead to extra repair costs or even to human casualties and environmental pollution. Recent condition monitoring techniques use artificial intelligence in an effort to avoid time-consuming manual analysis and handcrafted feature extraction. Many of these only analyze a single machine and require a large historical data set. In practice, this can be difficult and expensive to collect. However, some industrial condition monitoring applications involve a fleet of similar operating machines. In most of these applications, it is safe to assume healthy conditions for the majority of machines. Deviating machine behavior is then an indicator for a machine fault.This work proposes an unsupervised, generic, anomaly detection framework for fleet-based condition monitoring. It uses generic building blocks and offers three key advantages. First, a historical data set is not required due to online fleet-based comparisons. Second, it allows incorporating domain expertise by user-defined comparison measures. Finally, contrary to most black-box artificial intelligence techniques, easy interpretability allows a domain expert to validate the predictions made by the framework.Two use-cases on an electrical machine fleet demonstrate the applicability of the framework to detect a voltage unbalance by means of electrical and vibration signatures.

A Message Length Verification of Modern Messaging Systems

Nowadays, there has been a rapid increase in the variety and popularity of messaging systems and social networks. It is imperative to consider the effect and impact of the number of words feature on the verification process for modern messaging systems such as Twitter, Facebook, SMS and Email. Given the volume of text is often a restricted factor (due to the nature of messaging systems), key to this investigation is a better understanding of what length of message is required to improve performance. A large historical dataset containing 50 participants, the four datasets containing a large number of messaging system samples (4539 samples for Facebook, 13,616 for Twitter, 6538 for Email and 106,359 for Text message), the best performance was for Text messages, with an EER of 7.6% if the number of words was more than nine; followed by Email with an EER of 14.9% if the number of words was between 25 to 60; then, Twitter tweets, with an EER of 22.5% if the number of words was less than ten. Finally, the Facebook platform with an EER of 31.9% if the number of words was over 11.

Application of artificial neural networks in the drilling processes: Can equivalent circulation density be estimated prior to drilling?

As the drilling environment became more challenging nowadays, managing equivalent circulating density (ECD) is a key factor to minimize non-productive time (NPT) due to many drilling obstacles such as stuck pipe, formation fracturing, and lost circulation. The goal of this work was to predict ECD prior to drilling by using artificial neural network (ANN). Once ECD is recognized, the crucial drilling variables impact ECD can be modified to control ECD within the acceptable ranges. Data from over 2000 wells collected worldwide were used in this study to create an ANN to predict ECD prior to drilling. Into training, validation, and testing sets, the data were splitted. 70% of the data utilized for training, the other part used for validation and testing to avoid overfitting and create a generalized network that can perform well on new data. Based on the mean square of error (MSE), a decision was made to have one hidden layer with twelve neurons, this scenario was selected since it gave the lowest MSE among other scenarios. Multiple training functions were tested to train the network, Bayesian regularization (BR) algorithm was chosen from the other algorithms since it had the lowest MSE and the highest R-squared. After optimizing the weights and biases, the results revealed that the created network has the ability to estimate ECD with an overall R-squared of 0.982, which is very high. This result gives an indication that the created network can predict ECD prior to drilling globally within a very small margin of error. Due to the availability of large historical data sets in the petroleum industry, the ANN can be used to make better future decisions to minimize NPT and the cost of drilling.

Detecting BHP Flood Attacks in OBS Networks: A Machine Learning Prospective

The Optical Bust Switching (OBS) network has become the most promising switching technology for building the next generation of internet backbone infrastructure. However, an OBS network still faces a number of security and Quality of Service (QoS) challenges, particularly from Burst Header Packet (BHP) flood attacks. If a source node (ingress) becomes compromised by an attacker, overloading the network with malicious BHPs, the network resources will be reserved without proper utilization. This prevents legitimate BHPs from reserving the required resources, and can lead to severe issues, such as burst loss and Denial of Service (DoS) among others. One way to prevent a BHP flood attack is to detect the misbehaving edge nodes overloading the network with malicious BHPs, and taking the proper action to secure and sustain the QoS performance in an OBS network. A powerful and promising approach in identifying misbehaving edge nodes causing BHP flooding attacks is Machine Learning (ML), and in particular, classification techniques. A classification technique learns models by applying them to a large historical data set derived from an edge node’s performance during a simulation run. The data set contains behavior traces from a number of edge nodes, with respect to input data characteristics, sensitivity, efficiency performance, predictive performance, and model content. The learned model can then be utilized to single out (classify) misbehaving edge nodes based on their future performance as accurately as possible, hence disciplining them. In this paper, we investigate the BHP flood attack problem by evaluating a number of ML techniques in classifying edge nodes, and determine the most suitable method to prevent this type of attack. Specifically, we evaluate Decision Tree (C4.5), Bagging, Boosting (AdaBosst), Probabilistic (Naïve Bayes), Rule Induction (RIppleDOwn Rule Learner- RIDOR), Neural Network (NN-MultilayerPerceptron), Logistic Regression, and Support Vector Machine-Sequential Minimal Optimization (SVM-SMO) on a real dataset to identify the method(s) most appropriate to combat the BHP flood attack problem in OBS networks.

Solar Energy Production for a Decarbonization Scenario in Spain

Solar energy is one of the most promising sources of energy that could be used to address distributed supply problems. Global warming and decarbonization are significant global concerns, particularly for countries that are not fossil fuel providers. This paper presents a study focused on Spain, a country with a favorable location with respect to horizontal irradiance. The study addresses the future energy demand forecast and how photovoltaic energy could supply an important part of electricity needs. Our approach focuses on two analyses. First, several traditional statistical techniques are discussed in order to obtain a model that best suits Spanish energy demand forecasts for the future years. Different algorithms are compared in order to determine which is the most appropriate for the considered purpose. Second, the evolution of solar photovoltaic technology in Spain is analyzed. The latitude of Spanish cities makes them suitable for utilizing this kind of technology. In this sense, seasonal and monthly trends are identified with high levels of detail, considering a large historical dataset. The increase of the capacity of electricity generation based on this procedure is evaluated. Finally, a discussion about matching electricity demand forecasts and photovoltaic production is offered. Considering the selected model for the photovoltaic power of Spain, from 5 to 42 GW in 2030, the Spanish production is determined as 81 TWh. The obtained results suggest that a possible energy transition is feasible. However, some challenges have to be considered, such us the design of an effective strategy to store excess energy produced when generation is higher than electricity demand. In this way, the electrical distribution system could be fed by the stored energy when solar energy production is deficient.

Representing antibiotic relationships using measurements of efficacy against clinical isolates

Introduction. Antimicrobial resistance (AMR) is a worrying and confusing problem for both patients and medical professionals. Current antimicrobial nomenclature makes it difficult to understand how different antibiotics are related to one another, and their empirical uses and efficacy. Here, I attempt to address this problem by using experimental data from hundreds of thousands of clinical isolates to infer relationships between antibiotics and represent them with simple diagrams. Methods. The minimum inhibitory concentration (MIC) of a bacterial isolate for a given antibiotic is defined as the lowest concentration that prevents visible growth. Measuring MICs for multiple antibiotics using the same isolate implicitly records the relationships of the antibiotics for a given species. The basic principle is that antibiotics with similar mechanisms of action should give rise to similar mechanisms of resistance, so should have correlated MICs across large numbers of isolates. This information can then be used to calculate distances between antibiotics based on pairwise correlations of their rank-ordered MICs. I apply this approach to a large historical AMR surveillance dataset (the Pfizer ATLAS surveillance dataset, 2004-2017). Results. I demonstrate that clustering antibiotics in this way allows a simple visual comparison of how similar antibiotics are to each other based on their efficacy within a species. The resulting visualizations broadly recapitulate antibiotic classes. They also clearly show the dramatic effects of combining beta-lactam antibiotics with beta-lactamase inhibitors, as well as highlighting antibiotics which have unexpected correlations in MICs that are not predicted from their chemical similarities alone. Conclusion. Large AMR surveillance datasets can be used in a hypothesis-free manner to show relationships between antibiotics based on their real-world efficacy. The resulting 'data-driven' diagrams could be used as simple visual aids to explain the complexity of antibiotic relationships and AMR.

Representing antibiotic relationships using measurements of efficacy against clinical isolates

Introduction. Antimicrobial resistance (AMR) is a worrying and confusing problem for both patients and medical professionals. It is often difficult for non-specialists to understand how different antibiotics are related to one another. Here, I use experimental data from hundreds of thousands of clinical isolates to infer relationships between antibiotics and represent them with simple diagrams. Methods. The minimum inhibitory concentration (MIC) of a bacterial isolate for a given antibiotic is defined as the lowest concentration that prevents visible growth. Measuring MICs for multiple antibiotics using the same isolate implicitly records the relationships of the antibiotics for a given species. The basic principle is that antibiotics with similar mechanisms of action should give rise to similar mechanisms of resistance, so should have correlated MICs across large numbers of isolates. This information can then be used to calculate distances between antibiotics based on pairwise correlations of their rank-ordered MICs. I apply this approach to a large historical AMR surveillance dataset (the Pfizer ATLAS surveillance dataset, 2004-2017). Results. I demonstrate that clustering antibiotics in this way allows a simple visual comparison of how similar antibiotics are to each other based on their efficacy within a species. The resulting visualizations broadly recapitulate antibiotic classes. They also clearly show the dramatic effects of combining beta-lactam antibiotics with beta-lactamase inhibitors, as well as highlighting antibiotics which have unexpected correlations in MICs that are not predicted from their chemical similarities alone. Conclusion. Large AMR surveillance datasets can be used in a hypothesis-free manner to show relationships between antibiotics based on their real-world efficacy.

Climate and society in long-term perspective: Opportunities and pitfalls in the use of historical datasets.

Recent advances in paleoclimatology and the growing digital availability of large historical datasets on human activity have created new opportunities to investigate long‐term interactions between climate and society. However, noncritical use of historical datasets can create pitfalls, resulting in misleading findings that may become entrenched as accepted knowledge. We demonstrate pitfalls in the content, use and interpretation of historical datasets in research into climate and society interaction through a systematic review of recent studies on the link between climate and (a) conflict incidence, (b) plague outbreaks and (c) agricultural productivity changes. We propose three sets of interventions to overcome these pitfalls, which involve a more critical and multidisciplinary collection and construction of historical datasets, increased specificity and transparency about uncertainty or biases, and replacing inductive with deductive approaches to causality. This will improve the validity and robustness of interpretations on the long‐term relationship between climate and society.This article is categorized under: Climate, History, Society, Culture > Disciplinary Perspectives

Transforming Underwriting in the Life Insurance Industry

Life insurance provides trillions of dollars of financial security for hundreds of millions of individuals and families worldwide. Life insurance companies must accurately assess individual-level mortality risk to simultaneously maintain financial strength and price their products competitively. The traditional underwriting process used to assess this risk is based on manually examining an applicant’s health, behavioral, and financial profile. The existence of large historical data sets provides an unprecedented opportunity for artificial intelligence and machine learning to transform underwriting in the life insurance industry. We present an overview of how a rich application data set and survival modeling were combined to develop a life score that has been deployed in an algorithmic underwriting system at MassMutual, an American mutual life insurance company serving millions of clients. Through a novel evaluation framework, we show that the life score outperforms traditional underwriting by 6% on the basis of claims. We describe how engagement with actuaries, medical doctors, underwriters, and reinsurers was paramount to building an algorithmic underwriting system with a predictive model at its core. Finally, we provide details of the deployed system and highlight its value, which includes saving millions of dollars in operational efficiency while driving the decisions behind tens of billions of dollars of benefits.

Representing antibiotic relationships using measurements of efficacy against clinical isolates

Introduction. Antimicrobial resistance (AMR) is a worrying and confusing problem for both patients and medical professionals. It is often difficult for non-specialists to understand how different antibiotics are related to one another. Here, I use experimental data from hundreds of thousands of clinical isolates to infer relationships between antibiotics and represent them with simple diagrams. Methods. The minimum inhibitory concentration (MIC) of a bacterial isolate for a given antibiotic is defined as the lowest concentration that prevents visible growth. Measuring MICs for multiple antibiotics using the same isolate implicitly records the relationships of the antibiotics for a given species. The basic principle is that antibiotics with similar mechanisms of action should give rise to similar mechanisms of resistance, so should have correlated MICs across large numbers of isolates. This information can then be used to calculate distances between antibiotics based on pairwise correlations of their rank-ordered MICs. I apply this approach to a large historical AMR surveillance dataset (the Pfizer ATLAS surveillance dataset, 2004-2017). Results. I demonstrate that clustering antibiotics in this way allows a simple visual comparison of how similar antibiotics are to each other based on their efficacy within a species. The resulting visualizations broadly recapitulate antibiotic classes. They also clearly show the dramatic effects of combining beta-lactam antibiotics with beta-lactamase inhibitors, as well as highlighting antibiotics which have unexpected correlations in MICs that are not predicted from their chemical similarities alone. Conclusion. Large AMR surveillance datasets can be used in a hypothesis-free manner to show relationships between antibiotics based on their real-world efficacy.