Empirical Performance Research Articles

An EAP-Based Mutual Authentication Protocol for WLAN-Connected IoT Devices

Several symmetric and asymmetric encryption based authentication protocols have been developed for the wireless local area networks (WLANs). However, recent findings reveal that these protocols are either vulnerable to numerous attacks or computationally expensive. Considering the demerits of these protocols and the necessity to provide enhanced security, a lightweight extensible authentication protocol based authentication protocol for WLAN-connected Internet of Things devices is presented. We conduct an informal and formal security analysis to ensure robustness against the attacks. Furthermore, the empirical performance analysis and comparison show that the proposed protocol outperforms its counterparts, reducing computational, communication, storage costs, and energy consumption by up to 99%, 80%, 91.8%, and 98%, respectively. Simulation results of the protocol using the NS3 and its overhead under unknown attacks demonstrate that the proposed protocol performs better in all scenarios. A prototype implementation of the protocol has also been tested to evaluate its feasibility in real-time applications.

New C-indices for assessing importance of longitudinal biomarkers in fitting competing risks survival data inthepresence of partially masked causes.

Competing risks survival data in the presence of partially masked causes are frequently encountered in medical research or clinical trials. When longitudinal biomarkers are also available, it is of great clinical importance to examine associations between the longitudinal biomarkers and the cause-specific survival outcomes. In this article, we propose a cause-specific C-index for joint models of longitudinal and competing risks survival data accounting for masked causes. We also develop a posterior predictive algorithm for computing the out-of-sample cause-specific C-index using Markov chain Monte Carlo samples from the joint posterior of the in-sample longitudinal and competing risks survival data. We further construct the C-index to quantify the strength of association between the longitudinal and cause-specific survival data, or between the out-of-sample longitudinal and survival data. Empirical performance of the proposed assessment criteria is examined through an extensive simulation study. An in-depth analysis of the real data from large cancer prevention trials is carried out to demonstrate the usefulness of the proposed methodology.

Causal Mediation Analysis for Multivariate Longitudinal Data and Survival Outcomes

This study proposes a joint modeling approach to conduct causal mediation analysis that accommodates multivariate longitudinal data, dynamic latent mediator, and survival outcome. First, we introduce a confirmatory factor analysis model to characterize a time-varying latent mediator through multivariate longitudinal observable variables. Then, we establish a growth curve model to describe the linear trajectory of the dynamic latent mediator and simultaneously explore the relationship between the exposure and the mediating process. Finally, we link the mediating process to the survival outcome through a proportional hazards model. In addition, we use the mediation formula approach to assess the natural direct and indirect effects and prove the identifiability of the causal effects under sequential ignorability assumptions. A Bayesian approach incorporating the Markov chain Monte Carlo algorithm is developed to estimate the causal effects efficiently. Simulation studies are conducted to evaluate the empirical performance of the proposed method. An application to the Alzheimer’s Disease Neuroimaging Initiative study further confirms the utility of the proposed method.

On randomized sketching algorithms and the Tracy–Widom law

There is an increasing body of work exploring the integration of random projection into algorithms for numerical linear algebra. The primary motivation is to reduce the overall computational cost of processing large datasets. A suitably chosen random projection can be used to embed the original dataset in a lower-dimensional space such that key properties of the original dataset are retained. These algorithms are often referred to as sketching algorithms, as the projected dataset can be used as a compressed representation of the full dataset. We show that random matrix theory, in particular the Tracy–Widom law, is useful for describing the operating characteristics of sketching algorithms in the tall-data regime when the sample size n is much greater than the number of variables d. Asymptotic large sample results are of particular interest as this is the regime where sketching is most useful for data compression. In particular, we develop asymptotic approximations for the success rate in generating random subspace embeddings and the convergence probability of iterative sketching algorithms. We test a number of sketching algorithms on real large high-dimensional datasets and find that the asymptotic expressions give accurate predictions of the empirical performance.

A conditional higher-moment CAPM

This paper investigates whether dynamic and moment extensions to the traditional CAPM can improve its empirical performance and offer some alternative explanation to the cross-section of average returns on portfolios of stocks double sorted on book-to-market ratios and size. We consider three extensions. First, we introduce time-varying factor loadings obtained from a multivariate GARCH and dynamic conditional correlations. Second, we extend the model to a four-moment CAPM, which incorporates coskewness and cokurtosis. Finally, we allow for time-varying risk premia, based on a Markov-switching process. Our results confirm that the higher-moment CAPM does not perform well in its unconditional version, but its performance is significantly improved when we introduce a conditional version that accounts for both time-varying factor loadings and time-varying risk premia. The four-moment CAPM tests lead to a positive total risk premium estimate of 0.67% per month over the period 1926–2021, with all risk premia (beta, coskewness, and cokurtosis) exhibiting the expected theoretical signs.

Structural risk minimization for quantum linear classifiers

Quantum machine learning (QML) models based on parameterized quantum circuits are often highlighted as candidates for quantum computing's near-term “killer application''. However, the understanding of the empirical and generalization performance of these models is still in its infancy. In this paper we study how to balance between training accuracy and generalization performance (also called structural risk minimization) for two prominent QML models introduced by Havlíček et al. \cite{havlivcek:qsvm}, and Schuld and Killoran \cite{schuld:qsvm}. Firstly, using relationships to well understood classical models, we prove that two model parameters – i.e., the dimension of the sum of the images and the Frobenius norm of the observables used by the model – closely control the models' complexity and therefore its generalization performance. Secondly, using ideas inspired by process tomography, we prove that these model parameters also closely control the models' ability to capture correlations in sets of training examples. In summary, our results give rise to new options for structural risk minimization for QML models.

Practice Briefing Analysis of the emergence and initial performance of REITs in India

PurposeIn 2014, real estate investment trust (REIT) emerged as a new alternative investment option in India. This research aims to give an empirical authentication of the Indian REITs performance from April 2019 to July 2022 across a range of investment variables.Design/methodology/approachUsing monthly total returns in Indian Rupee, risk-adjusted Indian REIT performance and investment portfolio characteristics are examined. Indian REITs' potential in a diversified multi-asset portfolio is analysed using the mean-variance analysis, asset allocation diagram and efficient frontier.FindingsDuring April 2019–July 2022, Indian REITs provided a lower return than stocks but outperformed bonds despite coronavirus disease 2019 (COVID-19) lockdowns, which hurt the traditional working from office concept. The study also examined REIT allocation to an Indian mixed-asset portfolio and the benefits of a diversified portfolio.Practical implicationsIndian REITs provide a liquid, transparent alternative to direct property for investors seeking exposure to Indian real estate markets. Indian REITs gave real estate companies an extra funding source and investors an alternate asset. This paper explores Indian REITs' potential opportunities, given that domestic and foreign investors' demand for transparent property investment in India. The analysis found a positive early performance despite a challenging environment.Originality/valueThis paper offers the first empirical performance validation of Indian REITs as a way to obtain exposure to commercial property in India and the REITs' role in a diversified asset portfolio. The authors' study improves investors' decision-making abilities by providing empirically validated, valuable and practicable property investing insights.

Do the Hamilton and Beveridge–Nelson filters provide the same information about output gaps? An empirical comparison for practitioners

I compare the empirical performances of the recently-developed Hamilton and Beveridge–Nelson filters of nonstationary time series, using quarterly data on real gross state product in U.S. states. There is meaningful overlap between the two filters, with average correlation coefficients ranging between 0.60 and 0.97. The Hamilton filter and its more recent modification produce cycles of greater volatility and amplitude than the Beveridge–Nelson filter and appear to outperform in pseudo-out-of-sample forecasting exercises of future GSP growth and inflation (though the outperformance is not generally statistically significant). The Beveridge–Nelson filter is, however, less sensitive to realizations of new data.

Valid and Reliable Measures of Generalized Trust: Evidence from a Nationally Representative Survey and Behavioral Experiment

Generalized trust has been one of the most frequently examined constructs since researchers first introduced measures of it in the 1940s. Despite its significance, there is a growing consensus that traditional measures of generalized trust are prone to measurement invalidity and nonequivalence, calling into question sociological knowledge about generalized trust. In this article, I advance trust research in sociology by (1) refining two new self-report measures of generalized trust—the Stranger Face Trust scale (SFT) and the Imaginary Stranger Trust scale (IST)—and (2) assessing their empirical performance on a nationally representative probability sample (N = 1,264). I compare the reliability and validity of SFT, IST, and traditional measures of generalized trust across a range of measurement validation tests. Results suggest that SFT provides the most accurate and consistent measure of generalized trust.

Towards Enabling Binary Decomposition for Partial Multi-Label Learning.

Partial multi-label learning (PML) is an emerging weakly supervised learning framework, where each training example is associated with multiple candidate labels which are only partially valid. To learn the multi-label predictive model from PML training examples, most existing approaches work by identifying valid labels within candidate label set via label confidence estimation. In this paper, a novel strategy towards partial multi-label learning is proposed by enabling binary decomposition for handling PML training examples. Specifically, the widely used error-correcting output codes (ECOC) techniques are adapted to transform the PML learning problem into a number of binary learning problems, which refrains from using the error-prone procedure of estimating labeling confidence of individual candidate label. In the encoding phase, a ternary encoding scheme is utilized to balance the definiteness and adequacy of the derived binary training set. In the decoding phase, a loss weighted scheme is applied to consider the empirical performance and predictive margin of derived binary classifiers. Extensive comparative studies against state-of-the-art PML learning approaches clearly show the performance advantage of the proposed binary decomposition strategy for partial multi-label learning.

Protecting Sensitive Attributes by Adversarial Training Through Class-Overlapping Techniques

In recent years, machine learning as a service (MLaaS) has brought considerable convenience to our daily lives. However, these services raise the issue of leaking users’ sensitive attributes, such as race, when provided through the cloud. The present work overcomes this issue by proposing an innovative privacy-preserving approach called privacy-preserving class overlap (PPCO), which incorporates both a Wasserstein generative adversarial network and the idea of class overlapping to obfuscate data for better resilience against the leakage of attribute-inference attacks(i.e., malicious inference on users’ sensitive attributes). Experiments show that the proposed method can be employed to enhance current state-of-the-art works and achieve superior privacy–utility trade-off. Furthermore, the proposed method is shown to be less susceptible to the influence of imbalanced classes in training data. Finally, we provide a theoretical analysis of the performance of our proposed method to give a flavour of the gap between theoretical and empirical performances.

Evaluation of Arterial Signal Coordination with Commercial Connected Vehicle Data: Empirical Traffic Flow Visualization and Performance Measurement

Emerging connected vehicle (CV) data sets have recently become commercially available. This paper presents several tools using CV data to evaluate traffic progression quality along a signalized corridor. These include both performance measures for high-level analysis as well as visualizations to examine details of the coordinated operation. With the use of CV data, it is possible to assess not only the movement of traffic on the corridor but also to consider its origin-destination (OD) path through the corridor. Results for the real-world operation of an eight-intersection signalized arterial are presented. A series of high-level performance measures are used to evaluate overall performance by time of day, with differing results by metric. Next, the details of the operation are examined with the use of two visualization tools: a cyclic time-space diagram (TSD) and an empirical platoon progression diagram (PPD). Comparing flow visualizations developed with different included OD paths reveals several features. In addition, speed heat maps are generated, providing both speed performance along the corridor. The proposed visualization tools portray the corridor performance holistically instead of combining individual signal performance metrics. The techniques exhibited in this study are compelling for identifying locations where engineering solutions are required. The recent progress in infrastructure-free sensing technology has significantly increased the scope of CV data-based traffic management systems. The study demonstrates the utility of CV trajectory data for obtaining high-level details of the corridor performance and drilling down into the minute specifics.

Distance‐weighted discrimination for functional data

The main contribution of the paper is the development of a new margin‐based classifier called distance‐weighted discrimination (DWD) for functional data classification. The proposed classifier employs functional principal component analysis (FPCA) to reduce the dimensionality of the functional data and is free of the restrictive assumptions imposed by Bayes classifiers in terms of mean and covariance functions. Theoretical results show that the proposed classifier is Bayes risk consistent under mild assumptions. Simulation studies and real data examples demonstrate that the DWD classifier outperforms several conventional classifiers in terms of prediction accuracy. Overall, the paper provides a new approach for functional data classification with good empirical performance.

Goodness‐of‐fit test for semiparametric copula models with bivariate interval‐censored data

This paper discusses the goodness‐of‐fit test of semiparametric copula models when one faces bivariate interval‐censored failure time data, which often occur in many areas including epidemiological and medical studies as well as social science experiments. For the problem, three test statistics or procedures are proposed: two based on the pseudo in‐and‐out‐sample approach and one based on the information ratio criterion. The asymptotic properties of the proposed test procedures are established, and in particular, the three methods are shown to be asymptotically equivalent. To assess the empirical performance of the proposed methods, an extensive simulation study is conducted and indicates that they work well in practical situations. An application to a Signal–Tandmobiel study is provided.

Empirical Performance of CART, C5.0 and Random Forest Classification Algorithms for Decision Trees

This study compares the performance of <i>CART</i>, <i>C5.0</i> and Random Forest (<i>RF</i>) algorithms. 25 continuous predictors and 25 factors were simulated using a population size of 10,000. Based on this data, sample data were generated by varying the number of predictors, the proportion of categorical versus continuous predictors and the sample size. The performance of the tree algorithms increases with sample size and the number of variables, but for <i>RF</i>, it is highly greater than the one of <i>CART</i> and <i>C5.0</i>. Irrespective of the algorithms, the performance decreases when there are more categorical variables than continuous variables.

Data-Efficient Learning via Minimizing Hyperspherical Energy.

Deep learning on large-scale data is currently dominant nowadays. The unprecedented scale of data has been arguably one of the most important driving forces behind its success. However, there still exist scenarios where collecting data or labels could be extremely expensive, e.g., medical imaging and robotics. To fill up this gap, this paper considers the problem of data-efficient learning from scratch using a small amount of representative data. First, we characterize this problem by active learning on homeomorphic tubes of spherical manifolds. This naturally generates feasible hypothesis class. With homologous topological properties, we identify an important connection - finding tube manifolds is equivalent to minimizing hyperspherical energy (MHE) in physical geometry. Inspired by this connection, we propose a MHE-based active learning (MHEAL) algorithm, and provide comprehensive theoretical guarantees for MHEAL, covering convergence and generalization analysis. Finally, we demonstrate the empirical performance of MHEAL in a wide range of applications for data-efficient learning, including deep clustering, distribution matching, version space sampling, and deep active learning.

Self-Supervised Point Cloud Representation Learning via Separating Mixed Shapes

The manual annotation for large-scale point clouds costs a lot of time and is usually unavailable in harsh real-world scenarios. Inspired by the great success of the pre-training and fine-tuning paradigm in both vision and language tasks, we argue that pre-training is one potential solution for obtaining a scalable model to 3D point cloud downstream tasks as well. In this paper, we, therefore, explore a new self-supervised learning method, called Mixing and Disentangling ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MD</b> ), for 3D point cloud representation learning. As the name implies, we mix two input shapes and demand the model learning to separate the inputs from the mixed shape. We leverage this reconstruction task as the pretext optimization objective for self-supervised learning. There are two primary advantages: 1) Compared to prevailing image datasets, e.g., ImageNet, point cloud datasets are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de facto</i> small. The mixing process can provide a much larger online training sample pool. 2) On the other hand, the disentangling process motivates the model to mine the geometric prior knowledge, e.g., key points. To verify the effectiveness of the proposed pretext task, we build one baseline network, which is composed of one encoder and one decoder. During pre-training, we mix two original shapes and obtain the geometry-aware embedding from the encoder, then an instance-adaptive decoder is applied to recover the original shapes from the embedding. Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks. The proposed method has improved the empirical performance on both ModelNet-40 and ShapeNet-Part datasets in terms of point cloud classification and segmentation tasks. We further conduct ablation studies to explore the effect of each component and verify the generalization of our proposed strategy by harnessing different backbones.

Statistical inference via conditional Bayesian posteriors in high-dimensional linear regression

We propose a new method under the Bayesian framework to perform valid inference for low dimensional parameters in high dimensional linear models under sparsity constraints. Our approach is to use surrogate Bayesian posteriors based on partial regression models to remove the effect of high dimensional nuisance variables. We name the final distribution we used to conduct inference “conditional Bayesian posterior” as it is a surrogate posterior constructed conditional on quasi posterior distributions of other parameters and does not admit a fully Bayesian interpretation. Unlike existing Bayesian regularization methods, our method can be used to quantify the estimation uncertainty for arbitrarily small signals and therefore does not require variable selection consistency to guarantee its validity. Theoretically, we show that the resulting Bayesian credible intervals achieve desired coverage probabilities in the frequentist sense. Methodologically, our proposed Bayesian framework can easily incorporate popular Bayesian regularization procedures such as those based on spike and slab priors and horseshoe priors to facilitate high accuracy estimation and inference. Numerically, our proposed method rectifies the uncertainty underestimation of Bayesian shrinkage approaches and has a comparable empirical performance with state-of-the-art frequentist methods based on extensive simulation studies and a real data analysis.

CONFIG: Constrained Efficient Global Optimization for Closed-Loop Control System Optimization with Unmodeled Constraints

In this paper, the CONFIG algorithm, a simple and provably efficient constrained global optimization algorithm, is applied to optimize the closed-loop control performance of an unknown system with unmodeled constraints. Existing Gaussian process based closed-loop optimization methods, either can only guarantee local convergence (e.g., SafeOPT), or have no known optimality guarantee (e.g., constrained expected improvement) at all, whereas the recently introduced CONFIG algorithm has been proven to enjoy a theoretical global optimality guarantee. In this study, we demonstrate the effectiveness of CONFIG algorithm in the applications. The algorithm is first applied to an artificial numerical benchmark problem to corroborate its effectiveness. It is then applied to a classical constrained steady-state optimization problem of a continuous stirred-tank reactor. Simulation results show that our CONFIG algorithm can achieve performance competitive with the popular CEI (Constrained Expected Improvement) algorithm, which has no known optimality guarantee. As such, the CONFIG algorithm offers a new tool, with both a provable global optimality guarantee and competitive empirical performance, to optimize the closed-loop control performance for a system with soft unmodeled constraints. Last, but not least, the open-source code is available as a python package to facilitate future applications.

Differentiable Safe Controller Design Through Control Barrier Functions

Learning-based controllers, such as neural network (NN) controllers, can show high empirical performance but lack formal safety guarantees. To address this issue, control barrier functions (CBFs) have been applied as a safety filter to monitor and modify the outputs of learning-based controllers in order to guarantee the safety of the closed-loop system. However, such modification can be myopic with unpredictable long-term effects. In this work, we propose a safe-by-construction NN controller which employs differentiable CBF-based safety layers, and investigate the performance of safe-by-construction NN controllers in learning-based control. Specifically, two formulations of controllers are compared: one is projection-based and the other relies on our proposed set-theoretic parameterization. Both methods demonstrate improved closed-loop performance over using CBF as a separate safety filter in numerical experiments.