Data Setting Research Articles

Machine learning in pharmacometrics: Opportunities and challenges.

The explosive growth in medical devices, imaging and diagnostics, computing, and communication and information technologies in drug development and healthcare has created an ever-expanding data landscape that the pharmacometrics (PMX) research community must now traverse. The tools of machine learning (ML) have emerged as a powerful computational approach in other data-rich disciplines but its effective utilization in the pharmaceutical sciences and PMX modelling is in its infancy. ML-based methods can complement PMX modelling by enabling the information in diverse sources of big data, e.g. population-based public databases and disease-specific clinical registries, to be harnessed because they are capable of efficiently identifying salient variables associated with outcomes and delineating their interdependencies. ML algorithms are computationally efficient, have strong predictive capabilities and can enable learning in the big data setting. ML algorithms can be viewed as providing a computational bridge from big data to complement PMX modelling. This review provides an overview of the strengths and weaknesses of ML approaches vis-à-vis population methods, assesses current research into ML applications in the pharmaceutical sciences and provides perspective for potential opportunities and strategies for the successful integration and utilization of ML in PMX.

Distributed Bayesian Inference in Linear Mixed-Effects Models

Linear mixed-effects models play a fundamental role in statistical methodology. A variety of Markov chain Monte Carlo (MCMC) algorithms exist for fitting these models, but they are inefficient in massive data settings because every iteration of any such MCMC algorithm passes through the full data. Many divide-and-conquer methods have been proposed to solve this problem, but they lack theoretical guarantees, impose restrictive assumptions, or have complex computational algorithms. Our focus is one such method called the Wasserstein Posterior (WASP), which has become popular due to its optimal theoretical properties under general assumptions. Unfortunately, practical implementation of the WASP either requires solving a complex linear program or is limited to one-dimensional parameters. The former method is inefficient and the latter method fails to capture the joint posterior dependence structure of multivariate parameters. We develop a new algorithm for computing the WASP of multivariate parameters that is easy to implement and is useful for computing the WASP in any model where the posterior distribution of parameter belongs to a location-scatter family of probability measures. The algorithm is introduced for linear mixed-effects models with both implementation details and theoretical properties. Our algorithm outperforms the current state-of-the-art method in inference on the functions of the covariance matrix of the random effects across diverse numerical comparisons. Supplemental materials for this article are available online.

Head-to-head comparison of clustering methods for heterogeneous data: a simulation-driven benchmark

The choice of the most appropriate unsupervised machine-learning method for “heterogeneous” or “mixed” data, i.e. with both continuous and categorical variables, can be challenging. Our aim was to examine the performance of various clustering strategies for mixed data using both simulated and real-life data. We conducted a benchmark analysis of “ready-to-use” tools in R comparing 4 model-based (Kamila algorithm, Latent Class Analysis, Latent Class Model [LCM] and Clustering by Mixture Modeling) and 5 distance/dissimilarity-based (Gower distance or Unsupervised Extra Trees dissimilarity followed by hierarchical clustering or Partitioning Around Medoids, K-prototypes) clustering methods. Clustering performances were assessed by Adjusted Rand Index (ARI) on 1000 generated virtual populations consisting of mixed variables using 7 scenarios with varying population sizes, number of clusters, number of continuous and categorical variables, proportions of relevant (non-noisy) variables and degree of variable relevance (low, mild, high). Clustering methods were then applied on the EPHESUS randomized clinical trial data (a heart failure trial evaluating the effect of eplerenone) allowing to illustrate the differences between different clustering techniques. The simulations revealed the dominance of K-prototypes, Kamila and LCM models over all other methods. Overall, methods using dissimilarity matrices in classical algorithms such as Partitioning Around Medoids and Hierarchical Clustering had a lower ARI compared to model-based methods in all scenarios. When applying clustering methods to a real-life clinical dataset, LCM showed promising results with regard to differences in (1) clinical profiles across clusters, (2) prognostic performance (highest C-index) and (3) identification of patient subgroups with substantial treatment benefit. The present findings suggest key differences in clustering performance between the tested algorithms (limited to tools readily available in R). In most of the tested scenarios, model-based methods (in particular the Kamila and LCM packages) and K-prototypes typically performed best in the setting of heterogeneous data.

Demosaicing by Differentiable Deep Restoration

A mosaic of color filter arrays (CFAs) is commonly used in digital cameras as a spectrally selective filter to capture color images. The captured raw image is then processed by a demosaicing algorithm to recover the full-color image. In this paper, we formulate demosaicing as a restoration problem and solve it by minimizing the difference between the input raw image and the sampled full-color result. This under-constrained minimization is then solved with a novel convolutional neural network that estimates a linear subspace for the result at local image patches. In this way, the result in an image patch is determined by a few combination coefficients of the subspace bases, which makes the minimization problem tractable. This approach further allows joint learning of the CFA and demosaicing network. We demonstrate the superior performance of the proposed method by comparing it with state-of-the-art methods in both settings of noise-free and noisy data.

Website Fingerprinting in the Age of QUIC

Abstract With the meteoric rise of the QUIC protocol, the supremacy of TCP as the de facto transport protocol underlying web traffic will soon cease. HTTP/3, the next version of the HTTP protocol, will not support TCP. Current website-fingerprinting literature has ignored the introduction of this new protocol to all modern browsers. In this work, we investigate whether classifiers trained in the TCP setting generalise to QUIC traces, whether QUIC is inherently more difficult to fingerprint than TCP, how feature importance changes between these protocols, and how to jointly classify QUIC and TCP traces. Experiments using four state-of-theart website-fingerprinting classifiers and our combined QUIC-TCP dataset of ~117,000 traces show that while QUIC is not inherently more difficult to fingerprint than TCP, TCP-trained classifiers may fail to detect up to 96% of QUIC visits to monitored URLs. Furthermore, classifiers that take advantage of the common information between QUIC and TCP traces for the same URL may outperform ensembles of protocol-specific classifiers in limited data settings.

Change-Point Detection for Graphical Models in the Presence of Missing Values

We propose estimation methods for change points in high-dimensional covariance structures with an emphasis on challenging scenarios with missing values. We advocate three imputation like methods and investigate their implications on common losses used for change-point detection. We also discuss how model selection methods have to be adapted to the setting of incomplete data. The methods are compared in a simulation study and applied to a time series from an environmental monitoring system. An implementation of our proposals within the R-package hdcd is available via the online supplementary materials.

Tensor Canonical Correlation Analysis With Convergence and Statistical Guarantees

In many applications, such as classification of images or videos, it is of interest to develop a framework for tensor data instead of an ad-hoc way of transforming data to vectors due to the computational and under-sampling issues. In this article, we study convergence and statistical properties of two-dimensional canonical correlation analysis under an assumption that data come from a probabilistic model. We show that carefully initialized the power method converges to the optimum and provide a finite sample bound. Then we extend this framework to tensor-valued data and propose the higher-order power method, which is commonly used in tensor decomposition, to extract the canonical directions. Our method can be used effectively in a large-scale data setting by solving the inner least squares problem with a stochastic gradient descent, and we justify convergence via the theory of Lojasiewicz’s inequalities without any assumption on data generating process and initialization. For practical applications, we further develop (a) an inexact updating scheme which allows us to use the state-of-the-art stochastic gradient descent algorithm, (b) an effective initialization scheme which alleviates the problem of local optimum in nonconvex optimization, and (c) a deflation procedure for extracting several canonical components. Empirical analyses on challenging data including gene expression and air pollution indexes in Taiwan, show the effectiveness and efficiency of the proposed methodology. Our results fill a missing, but crucial, part in the literature on tensor data. Supplementary materials for this article are available online.

False discovery rate for functional data

Since Benjamini and Hochberg introduced false discovery rate (FDR) in their seminal paper, this has become a very popular approach to the multiple comparisons problem. An increasingly popular topic within functional data analysis is local inference, i.e. the continuous statistical testing of a null hypothesis along the domain. The principal issue in this topic is the infinite amount of tested hypotheses, which can be seen as an extreme case of the multiple comparisons problem. In this paper, we define and discuss the notion of FDR in a very general functional data setting. Moreover, a continuous version of the Benjamini–Hochberg procedure is introduced along with a definition of adjusted p value function. Some general conditions are stated, under which the functional Benjamini–Hochberg procedure provides control of the functional FDR. Two different simulation studies are presented; the first study has a one-dimensional domain and a comparison with another state-of-the-art method, and the second study has a planar two-dimensional domain. Finally, the proposed method is applied to satellite measurements of Earth temperature. In detail, we aim at identifying the regions of the planet where temperature has significantly increased in the last decades. After adjustment, large areas are still significant.

Functional analysis of generalized linear models under non-linear constraints with applications to identifying highly-cited papers

This article introduces a versatile functional form for Generalized Linear Models (GLMs) through a simple, yet effective, transformation of the current framework. The models are applied through a new hierarchical bayesian estimation procedure for logistic regression to highly-cited papers in the Management Information Systems (MIS) field. The results are uniformly better, in regards to model fit and inference for in-sample and out-of-sample data, for simulation studies and real-world data applications, requiring very little time to convergence to true population parameters. In simulation studies, I show that the method contains the true parameters nearly three times as often as widely used existing GLMs, and does so while having confidence intervals that are 54.50% smaller, while requiring around two-thirds the number of MCMC iterations as existing bayesian methods. In Scientometric applications the methodology is shown to be highly robust with predictive/classification accuracy, either equaling or exceeding existing methods for identifying highly-cited articles including Artificial Neural Networks (ANN). Thus, the method is shown to be robust to the amount of asymmetry (or symmetry) of the probability of success (or failure) and robust to unbalanced samples and varying Data Generating Processes. Further, the methodology is equivalent to current methods if the data support them and is therefore complementary to existing methods, without loss of interpretability of model parameters. For the MIS field it finds that Popularity Parameter (PP) of an article Keywords can predict whether a paper will be highly-cited (top 25% of highly-cited articles) between two to three years after publication and beyond. Furthermore, given the small number of iterations needed for convergence, the methodology can also be used as a baseline method in Big Data (BD) settings for both Artificial Intelligence (AI) and Machine Learning (ML) contexts as well.

An efficient numerical method for condition number constrained covariance matrix approximation

In the high-dimensional data setting, the sample covariance matrix is singular. In order to get a numerically stable and positive definite modification of the sample covariance matrix in the high-dimensional data setting, in this paper we consider the condition number constrained covariance matrix approximation problem and present its explicit solution with respect to the Frobenius norm. The condition number constraint guarantees the numerical stability and positive definiteness of the approximation form simultaneously. By exploiting the special structure of the data matrix in the high-dimensional data setting, we also propose some new algorithms based on efficient matrix decomposition techniques. Numerical experiments are also given to show the computational efficiency of the proposed algorithms.

Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked uniformly from a given dataset. Our method results in an O(d) improvement in complexity in comparison to LSTD, where d is the dimension of the data. We provide non-asymptotic bounds for our proposed method, both in high probability and in expectation, under the assumption that the matrix underlying the LSTD solution is positive definite. The latter assumption can be easily satisfied for the pathwise LSTD variant proposed by Lazaric (J Mach Learn Res 13:3041–3074, 2012). Moreover, we also establish that using our method in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. These rate results coupled with the low computational complexity of our method make it attractive for implementation in big data settings, where d is large. A similar low-complexity alternative for least squares regression is well-known as the stochastic gradient descent (SGD) algorithm. We provide finite-time bounds for SGD. We demonstrate the practicality of our method as an efficient alternative for pathwise LSTD empirically by combining it with the least squares policy iteration algorithm in a traffic signal control application. We also conduct another set of experiments that combines the SA-based low-complexity variant for least squares regression with the LinUCB algorithm for contextual bandits, using the large scale news recommendation dataset from Yahoo.

Analyzing state government spending: balanced budget rules or forward-looking decisions?

We analyze US state government spending behavior with a general intertemporal model allowing for asymmetry in balanced budget rules in a panel data setting. We find no strong evidence for forward-looking behavior in state spending; balanced budget rules are a significant constraint. States with budget rules imposing lighter restrictions are more likely to exhibit habit formation, while those with stricter rules demonstrate asymmetric responses to revenue changes. Evidence for a precautionary savings motive is limited. Balanced budget requirements trigger substantial pro-cyclical spending, possibly amplifying state economic volatility for states with tight fiscal rules, especially after revenue increases.

Coresets for Regressions with Panel Data

This paper introduces the problem of coresets for regression problems to panel data settings. We first define coresets for several variants of regression problems with panel data and then present efficient algorithms to construct coresets of size that depend polynomially on 1/$\varepsilon$ (where $\varepsilon$ is the error parameter) and the number of regression parameters - independent of the number of individuals in the panel data or the time units each individual is observed for. Our approach is based on the Feldman-Langberg framework in which a key step is to upper bound the "total sensitivity" that is roughly the sum of maximum influences of all individual-time pairs taken over all possible choices of regression parameters. Empirically, we assess our approach with synthetic and real-world datasets; the coreset sizes constructed using our approach are much smaller than the full dataset and coresets indeed accelerate the running time of computing the regression objective.

A Correlated Noise-assisted Decentralized Differentially Private Estimation Protocol, and its application to fMRI Source Separation.

Blind source separation algorithms such as independent component analysis (ICA) are widely used in the analysis of neuroimaging data. To leverage larger sample sizes, different data holders/sites may wish to collaboratively learn feature representations. However, such datasets are often privacy-sensitive, precluding centralized analyses that pool the data at one site. In this work, we propose a differentially private algorithm for performing ICA in a decentralized data setting. Due to the high dimension and small sample size, conventional approaches to decentralized differentially private algorithms suffer in terms of utility. When centralizing the data is not possible, we investigate the benefit of enabling limited collaboration in the form of generating jointly distributed random noise. We show that such (anti) correlated noise improves the privacy-utility trade-off, and can reach the same level of utility as the corresponding non-private algorithm for certain parameter choices. We validate this benefit using synthetic and real neuroimaging datasets. We conclude that it is possible to achieve meaningful utility while preserving privacy, even in complex signal processing systems.

Semi-parametric Estimation of Biomarker Age Trends with Endogenous Medication Use in Longitudinal Data

In cohort studies, non-random medication use can pose barriers to estimation of the natural history trend in a mean biomarker value-namely, the association between a predictor of interest and a biomarker outcome that would be observed in the total absence of biomarker-specific treatment. Common causes of treatment and outcomes are often unmeasured, obscuring our ability to easily account for medication use with assumptions commonly invoked in causal inference such as conditional ignorability. Further, without a high degree of confidence in the availability of a variable satisfying the exclusion restriction, use of instrumental variable approaches may be difficult to justify. Heckman's hybrid model with structural shift (sometimes referred to less specifically as the treatment effects model) can be used to correct endogeneity bias via a homogeneity assumption (i.e., that average treatment effects do not vary across covariates) and parametric specification of a joint model for the outcome and treatment. In recent work, we relaxed the homogeneity assumption by allowing observed covariates to serve as treatment effect modifiers. While this method has been shown to be reasonably robust in settings of cross-sectional data, application of this methodology to settings of longitudinal data remains unexplored. We demonstrate how the assumptions of the treatment effects model can be extended to accommodate clustered data arising from longitudinal studies. Our proposed approach is semi-parametric in nature in that valid inference can be obtained without the need to specify any component of the longitudinal correlation structure. As an illustrative example, we use data from the Multi-Ethnic Study of Atherosclerosis to evaluate trends in low-density lipoprotein by age and gender. Results from a collection of simulation studies, as well as our illustrative example, confirm that our generalization of the treatment effects model can serve as a useful tool to uncover natural history trends in longitudinal data that are obscured by endogenous treatment.

Unsupervised Adaptation for High-Dimensional with Limited-Sample Data Classification Using Variational Autoencoder

High-dimensional with limited-sample size (HDLSS) datasets exhibit two critical problems: (1) Due to the insufficiently small-sample size, there is a lack of enough samples to build classification models. Classification models with a limited-sample may lead to overfitting and produce erroneous or meaningless results. (2) The 'curse of dimensionality' phenomena is often an obstacle to the use of many methods for solving the high-dimensional with limited-sample size problem and reduces classification accuracy. This study proposes an unsupervised framework for high-dimensional limited-sample size data classification using dimension reduction based on variational autoencoder (VAE). First, the deep learning method variational autoencoder is applied to project high-dimensional data onto lower-dimensional space. Then, clustering is applied to the obtained latent-space of VAE to find the data groups and classify input data. The method is validated by comparing the clustering results with actual labels using purity, rand index, and normalized mutual information. Moreover, to evaluate the proposed model strength, we analyzed 14 datasets from the Arizona State University Digital Repository. Also, an empirical comparison of dimensionality reduction techniques shown to conclude their applicability in the high-dimensional with limited-sample size data settings. Experimental results demonstrate that variational autoencoder can achieve more accuracy than traditional dimensionality reduction techniques in high-dimensional with limited-sample-size data analysis.

Hierarchical Bayes Conjoint Choice Models - Model Framework, Bayesian Inference, Model Selection, and Interpretation of Estimation Results

Choice-based conjoint (CBC) is nowadays the most widely used variant of conjoint analysis, a class of methods for measuring consumer preferences. The primary reason for the increasing dominance of the CBC approach over the last 35 years is that it closely mimics real choice behavior of consumers by asking respondents repeatedly to choose their preferred alternative from a set of several offered alternatives (choice sets). Within the framework of CBC analysis, the multinomial logit (MNL) model is the most frequently used discrete choice model due to the existence of closed form solutions for conditional choice probabilities. The popularity of CBC and the MNL model has grown even more since the introduction of hierarchical Bayesian (HB) estimation techniques that accommodate individual consumer heterogeneity in choice data, and which have now become state-of-the-art in marketing theory and practice. Still, researchers and practitioners have to make further decisions under this framework (CBC, MNL, HB estimation), such as how to represent preference heterogeneity. Here, using a normal distribution (and therefore a unimodal distribution) has become the standard approach in the marketing literature. However, the thin tails of the normal distribution suggest that the standard HB-MNL model should not be the “go-to” approach to approximate multimodal preference distributions, because individual preference patterns lying at the tails of the normal distribution (i.e., that do not fit well with the assumption of a unimodal distribution) tend to be shrunk to the population mean. This shrinkage, especially in multimodal data settings, could mask important information (e.g., new or different structures in the data). A mixture of normal distributions avoids this limited flexibility of the most simple continuous approach of assuming a unimodal prior heterogeneity distribution. There are currently two prominent HB-CBC modeling approaches embedding the mixture-of-normals (MoN) approach: the more widespread MoN-HB-MNL model, and the Dirichlet process mixture (DPM)-HB-MNL model. In this article, we review the prominent HB-MNL model (with its normal prior), the MoN-HB-MNL model, and the DPM-HB-MNL model and apply them to an empirical multi-country CBC data set. We compare the statistical performance of the three models in terms of goodness-of-fit and predictive accuracy, show how to include consumer background characteristics in the upper level of these models, and illustrate how to interpret the estimation results (with a special focus on cross-county heterogeneity). In sum, our article serves as a kind of user guide to the estimation and interpretation of Hierarchical Bayes Conjoint Choice Models. For our data, we observed that all three choice models (both with and without consumer background characteristics) resulted in a one-component solution. The DPM-HB-MNL model nevertheless yielded a higher cross-validated hit rate compared to the MoN-HB-MNL and the HB-MNL models due to its even more flexible prior assumptions. The two latter models tended to slightly overfit our empirical data, which was reflected by higher goodness-of-fit statistics but a lower predictive accuracy compared to the DPM-HB-MNL model. We showed that this result could be attributed to the weaker extent of Bayesian shrinkage of these two models. The DPM-HB-MNL model showed a stronger shrinkage effect and seems therefore somewhat more robust against overfitting. Including consumer background characteristics in terms of country of origin information for the respondents did not improve the statistical model performance (especially not the predictive performance). Still, using the country of origin information for respondents in a post-hoc segmentation analysis helped us to explain some differences in brand preferences between the five countries.

Compositional trend filtering

Trend filtering is known as the technique for detecting piecewise linear trends in univariate time series. This technique is extended to the setting of compositional data, which are multivariate data where only the relative information is of importance. According to this, we formulate the problem and present a procedure how to efficiently solve it. To show the usefulness of this method, we consider the number of COVID-19 infections in several European countries in a chosen time period.

A computational workflow to explore material properties of existing settings of point cloud data

This paper explores a planned utilization of direction cloud data in supporting the talk examination of the designed air all through the theoretical style strategy. Frequently, the nature of site information makes draftsmen disregard numerous significant properties that will that will execution examination, especially once adapting to an opulent style case. For example, the current methodologies of 3D site demonstrating need partner degree comprehension of the situating qualities of existing conditions concerning either mathematical or material properties. As a sort of style dynamic help, this investigation at last creates a methodology system for partner degreealysing the designed air from that planners may direct an exhaustive examination of a current setting prior to starting style investigation.

A Simulation Study for Performance Comparison between Generalized Linear Mixed Modeling (GLMM) and Generalized Regression Neural Networks (GRNN) in Joint Modeling of Mixed Responses

Joint modeling of mixed responses has become a popular research area due to its applicability in many disciplines. When there is an association between two responses, a joint model will provide improved results than modeling the responses separately. In this study, the joint modeling of survival and count variables was carried out using Generalized Linear Mixed Modeling (GLMM) and Generalized Regression Neural Network (GRNN) to compare their performances under the setting of clustered data. A joint model of survival and count variables that was developed by joining the Discrete Time Hazard Model (DTHM) and Poisson Regression model was used in this study as the GLMM model. A simulation study was carried out under three different sample sizes; n = 20, 100, and 500, and for three levels of correlations between two responses: low (r = 0.30), moderate low (r = 0.30), and high (r = 0.30). The root mean square error, absolute mean error and correlation coefficient between actual and predicted response data were calculated to compare the performances of GLMM and GRNN models. The results revealed that the GRNN has a better fit in general, but under large sample sizes and high correlations between response variables, GLMM outperformed the GRNN. Application of these methods to a data set from poultry industry further confirmed the suitability of Generalized Regression Neural Network fit for joint modeling of real-world data.