Subgroup analysis with a nonparametric unimodal symmetric error distribution

Abstract

In clinical trials and moving toward precision medicine, a very key issue is identifying subgroups of patients with different responses to therapy, namely, testing if there exist subgroups of patients who may benefit the most from the treatment under investigation, e.g., treatment-favorable vs nonfavorable subgroups, and then classifying patients into such subgroups. Existing parametric methods are not robust against subjective assumptions, and the typical classification rules ignore the priority of the treatment-favorable subgroup. To address these issues, we propose a semiparametric model, with the subdensities specified nonparametrically. For semiparametric mixture identifiability, the subdensity is assumed to be symmetric unimodal to find its nonparametric maximum likelihood estimate, and it can be extended without the symmetry assumption for a wider application scope. The Wald statistic is used to test the existence of subgroups, and the Neyman-Pearson rule is used to classify each subject. Basic asymptotic properties of the estimators are derived, and simulation studies are performed to evaluate the finite sample properties of the method. Finally, the method is applied to the analysis of a real clinical trial result.