Categorical Embedding Research Articles

A Nested GLM Framework with Neural Network Encoding and Spatially Constrained Clustering in Non-Life Insurance Ratemaking

The generalized linear model (GLM) is a popular modeling choice for pricing non-life insurance policies. However, high-cardinality categorical insurance data presents significant challenges for these GLM rate-making models. Additionally, insurance regulators often require rating territories, which are clusters of insurance policies’ geographic locations for setting insurance rates, to meet certain standards. For instance, (1) the credibility standard ensures that the number of policies in a territory is large enough to be credible and representative, (2) the contiguity standard requires the locations in each territory to be geographically adjacent to promote a logical and practical spatial grouping, and (3) the cardinality standard specifies an acceptable range for the number of territories in a geographic area. To address these challenges, this article proposes a nested GLM framework for non-life insurance rate-making applications. In this framework, neural network models with categorical embedding layers are constructed to model the residual deviance from simple GLMs, using high-cardinality categorical variables as input. Low-dimensionly features extracted from the neural network model effectively translate categorical variables into meaningful numerical representations, capturing their effects on the initial model’s residuals. The features corresponding to the location-related variable are further converted into a contiguous territory rating variable via spatially constrained clustering models. By incorporating outcomes from these models, the nested GLM not only satisfies regulatory requirements but also enhances the model’s predictive power, while maintaining the interpretability from the (generalized) linear form. The construction of a nested Poisson GLM is presented in this article. Its performance is demonstrated using a real-life Brazil auto insurance data to model claim frequency.

Explainable unsupervised anomaly detection for healthcare insurance data

BackgroundWaste and fraud are important problems for health insurers to deal with. With the advent of big data, these insurers are looking more and more towards data mining and machine learning methods to help in detecting waste and fraud. However, labeled data is costly and difficult to acquire as it requires expert investigators and known care providers with atypical behavior.MethodsIn this work we show how recent advances in machine learning can be used to set up a workflow that can aid investigators in discovering practitioners or groups of practitioners with unusual resource use in order to more efficiently combat waste and fraud. We combine three different techniques, which have not been used in the context of healthcare insurance anomaly detection: categorical embeddings to deal with high-cardinality categorical variables, state-of-the-art unsupervised anomaly detection techniques to detect anomalies and Shapley additive explanations (SHAP) to explain the model output.ResultsThe method has been evaluated on providers with a known anomalous profile and with the help of experts of the largest health insurance fund in Belgium. The quantitative experiments show that categorical embeddings offer a significant improvement compared to standard methods and that the state-of-the-art unsupervised anomaly detection techniques generally show an improvement over traditional methods. In a practical setting, the proposed workflow with SHAP was able to detect a previously unknown, anomalous trend among general practitioners.ConclusionsThe proposed workflow is able to detect known care providers with atypical behaviour and helps expert investigators in making informed decisions concerning possible fraud or overconsumption in the health insurance field.

Non-Life Insurance Risk Classification Using Categorical Embedding

This article presents several actuarial applications of categorical embedding in the context of non-life insurance risk classification. In non-life insurance, many rating factors are naturally categorical, and often the categorical variables have a large number of levels. The high cardinality of categorical rating variables presents challenges in the implementation of traditional actuarial methods. Categorical embedding that is proposed in the machine learning literature for handling categorical variables has recently received attention in actuarial studies. The method is inspired by the neural network language models for learning text data and maps a categorical variable into a real-valued representation in the Euclidean space. Using a property insurance claims we demonstrate the use of categorical embedding in three applications. The first shows how embeddings are used to construct rating classes and calculate rating relativities for a single insurance risk. The second concerns predictive modeling for multivariate insurance risks and emphasizes the effects of dependence on tail risks. The third focuses on pricing new products where transfer learning is used to gather knowledge from existing products.

A comparison of neural and non-neural machine learning models for food safety risk prediction with European Union RASFF data

The world is becoming more globalized every day and people can buy products from almost every country in the world in their local stores. Given the different food and feed safety laws from country to country, the European Union began to register in 1977 all irregularities related to traded products to ensure cross-border monitoring of information and a quick reaction when risks to public health are detected in the food chain. This information has also an enormous potential as a preventive tool, in order to warn actors involved in food safety and optimize their resources. In this paper, a set of data related to food issues was scraped and analysed with Machine Learning techniques to predict some features of future notifications, so that pre-emptive measures can be taken. The novelty of the work relies on two points: the use of categorical embeddings with Deep Learning models (Multilayer Perceptron and 1-Dimension Convolutional Neural Networks) and its application to solve the problem of predicting food issues in the European Union. The models allow several features to be predicted: product category, hazard category and finally the proper action to be taken. Results show that the system can predict these features with an accuracy ranging from 74.08% to 93.06%.

Implementing Vertical Federated Learning Using Autoencoders: Practical Application, Generalizability, and Utility Study.

BackgroundMachine learning (ML) is now widely deployed in our everyday lives. Building robust ML models requires a massive amount of data for training. Traditional ML algorithms require training data centralization, which raises privacy and data governance issues. Federated learning (FL) is an approach to overcome this issue. We focused on applying FL on vertically partitioned data, in which an individual’s record is scattered among different sites.ObjectiveThe aim of this study was to perform FL on vertically partitioned data to achieve performance comparable to that of centralized models without exposing the raw data.MethodsWe used three different datasets (Adult income, Schwannoma, and eICU datasets) and vertically divided each dataset into different pieces. Following the vertical division of data, overcomplete autoencoder-based model training was performed for each site. Following training, each site’s data were transformed into latent data, which were aggregated for training. A tabular neural network model with categorical embedding was used for training. A centrally based model was used as a baseline model, which was compared to that of FL in terms of accuracy and area under the receiver operating characteristic curve (AUROC).ResultsThe autoencoder-based network successfully transformed the original data into latent representations with no domain knowledge applied. These altered data were different from the original data in terms of the feature space and data distributions, indicating appropriate data security. The loss of performance was minimal when using an overcomplete autoencoder; accuracy loss was 1.2%, 8.89%, and 1.23%, and AUROC loss was 1.1%, 0%, and 1.12% in the Adult income, Schwannoma, and eICU dataset, respectively.ConclusionsWe proposed an autoencoder-based ML model for vertically incomplete data. Since our model is based on unsupervised learning, no domain-specific knowledge is required in individual sites. Under the circumstances where direct data sharing is not available, our approach may be a practical solution enabling both data protection and building a robust model.

Utilizing Text Mining, Data Linkage and Deep Learning in Police and Health Records to Predict Future Offenses in Family and Domestic Violence.

Family and Domestic violence (FDV) is a global problem with significant social, economic, and health consequences for victims including increased health care costs, mental trauma, and social stigmatization. In Australia, the estimated annual cost of FDV is $22 billion, with one woman being murdered by a current or former partner every week. Despite this, tools that can predict future FDV based on the features of the person of interest (POI) and victim are lacking. The New South Wales Police Force attends thousands of FDV events each year and records details as fixed fields (e.g., demographic information for individuals involved in the event) and as text narratives which describe abuse types, victim injuries, threats, including the mental health status for POIs and victims. This information within the narratives is mostly untapped for research and reporting purposes. After applying a text mining methodology to extract information from 492,393 FDV event narratives (abuse types, victim injuries, mental illness mentions), we linked these characteristics with the respective fixed fields and with actual mental health diagnoses obtained from the NSW Ministry of Health for the same cohort to form a comprehensive FDV dataset. These data were input into five deep learning models (MLP, LSTM, Bi-LSTM, Bi-GRU, BERT) to predict three FDV offense types (“hands-on,” “hands-off,” “Apprehended Domestic Violence Order (ADVO) breach”). The transformer model with BERT embeddings returned the best performance (69.00% accuracy; 66.76% ROC) for “ADVO breach” in a multilabel classification setup while the binary classification setup generated similar results. “Hands-off” offenses proved the hardest offense type to predict (60.72% accuracy; 57.86% ROC using BERT) but showed potential to improve with fine-tuning of binary classification setups. “Hands-on” offenses benefitted least from the contextual information gained through BERT embeddings in which MLP with categorical embeddings outperformed it in three out of four metrics (65.95% accuracy; 78.03% F1-score; 70.00% precision). The encouraging results indicate that future FDV offenses can be predicted using deep learning on a large corpus of police and health data. Incorporating additional data sources will likely increase the performance which can assist those working on FDV and law enforcement to improve outcomes and better manage FDV events.

Watts Spaces and Smooth Maps

A new category is described, which generalizes a select variety of categories having smooth mappings as their class of morphisms, those like the C∞ manifolds and C∞ mappings between them. Categorical embeddings are produced to justify this claim of generalization. Theorems concerning the equivalence of smooth and continuous versions of different separation axioms are proved following the categorical discussion. These are followed by a generalization of Whitney's approximation theorem, a smooth version of the Tietze extension theorem, and a sufficient condition to guarantee that the connected components of these spaces are smoothly path-connected.

HCE: Hierarchical Context Embedding for Region-Based Object Detection.

State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The region-wise features, in spite of aligning well with the proposal locations, may still lack the crucial context information which is necessary for filtering out noisy background detections, as well as recognizing objects possessing no distinctive appearances. To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to facilitate the classification ability of a series of region-based detectors by mining contextual cues. Specifically, to advance the recognition of context-dependent object categories, we propose an image-level categorical embedding module which leverages the holistic image-level context to learn object-level concepts. Then, novel RoI features are generated by exploiting hierarchically embedded context information beneath both whole images and interested regions, which are also complementary to conventional RoI features. Moreover, to make full use of our hierarchical contextual RoI features, we propose the early-and-late fusion strategies (i.e., feature fusion and confidence fusion), which can be combined to boost the classification accuracy of region-based detectors. Comprehensive experiments demonstrate that our HCE framework is flexible and generalizable, leading to significant and consistent improvements upon various region-based detectors, including FPN, Cascade R-CNN, Mask R-CNN and PA-FPN. With simple modification, our HCE framework can be conveniently adapted to fit the structure of one-stage detectors, and achieve improved performance for SSD, RetinaNet and EfficientDet.

An Artificial Intelligence Approach to Shadow Rating

We analyse the effectiveness of modern deep learning techniques in predicting credit ratings over a universe of thousands of global corporate entities obligations when compared to most popular, traditional machine-learning approaches such as linear models and tree-based classifiers. Our results show a adequate accuracy over different rating classes when applying categorical embeddings to artificial neural networks (ANN) architectures.

Sum-Product Autoencoding: Encoding and Decoding Representations Using Sum-Product Networks

Sum-Product Networks (SPNs) are a deep probabilistic architecture that up to now has been successfully employed for tractable inference. Here, we extend their scope towards unsupervised representation learning: we encode samples into continuous and categorical embeddings and show that they can also be decoded back into the original input space by leveraging MPE inference. We characterize when this Sum-Product Autoencoding (SPAE) leads to equivalent reconstructions and extend it towards dealing with missing embedding information. Our experimental results on several multi-label classification problems demonstrate that SPAE is competitive with state-of-the-art autoencoder architectures, even if the SPNs were never trained to reconstruct their inputs.

Envelopes of geometric lattices

A categorical embedding theorem is proved for geometric lattices. This states roughly that, if one wants to consider only those embeddings into projective spaces having a suitable universal property, then the existence of such an embedding can be checked by seeing whether corresponding properties hold for many small intervals. Tutte's embedding theorem for binary geometric lattices is a consequence of this result.