Profile repeatability: A new method for evaluating repeatability of individual hormone response profiles

Abstract

There is broad interest in determining repeatability of individual responses. Current methods calculate repeatability of individual points (initial and/or peak), time to peak value, or a single measure of the integrated total response (area under the curve), rather than the shape of the response profile. Repeatability estimates of response profiles using linear mixed models (LMM) generate an average repeatability for an aggregate of individuals, rather than an estimate of individual repeatability. Here we use a novel ad hoc method to calculate repeatability of individual response profiles and demonstrate the need for a more rigorous assessment protocol. Response profile repeatability has not been defined at the individual level. We do this using a new metric, Profile Repeatability (PR), which incorporates components of variance and the degree to which response profiles cross each other in a time series. Values range from 0 (no repeatability) to 1 (complete repeatability). We created synthetic data to represent a range of apparent time series repeatability, and 20 independent observers visually ranked those data sets by degree of repeatability. We also applied the method to real data on stress responses of European starlings Sturnus vulgaris. We then computed PR scores for the synthetic data and for real data from European starling corticosterone responses over time, and contrast the results to those from LMM. Finally, we assessed the sensitivity of PR to reductions in the number of time points in the corticosterone response, as well as reductions in the number of replicates per individual. We found the average PR scores for a group of individuals to be somewhat robust to reductions in points in the time series; however, the ranks of individuals (PR values relative to one another) could change substantially with reduction in the number of values in a time series. PR showed threshold sensitivity to losing replicate time series between 6 and 4 replicates. Surprisingly, human observers fell into two disparate groups when ranking repeatability of the synthetic data, and the PR score indicated that human observers may underestimate repeatability of data where replicates cross each other. In contrast to the average profile repeatability estimated using LMMs, our approach calculates individual repeatability. From our perspective, LMM does not provide a definitive idea of repeatability at the individual level; in essence, it concludes that suites of time series with low within-individual variance has high repeatability, regardless of replicate trajectories. LMM and PR have non-linear relationships between 0 and 1, but PR has greater discrimination for mid-values of repeatability. Consistent average group repeatability can be associated with substantial differences in individual ranks suggests that estimating individual repeatability is critical. The PR score should be useful in comparing repeatability of any type of nonlinear, including non-monotonic, response profiles over time, which are common in both physiology and behavior, and it demonstrates the specific needs for future improvements of a profile repeatability metric.