Panel and panelist agreement for product comparisons in studies of Temporal Dominance of Sensations

Abstract

Temporal Dominance of Sensations (TDS) is a sensory method to investigate the most dominant attribute in consumer perception over time. In this paper, we aim at investigating (dis-)agreement as a performance measure of the overall panel as well as of individual panelists. This helps to check the data quality and, if necessary, to direct the panel leader in subsequent training. Panel and panelist agreement with regard to product differences is a main aspect of panel performance. The interaction between panelist and product serves as a useful measure of this agreement. This interaction can only be determined if panelists evaluate at least some of the products repeatedly. A simple test statistic is used here to investigate possible panelist disagreement by means of randomization tests. We propose a test for overall panel agreement on product differences as well as tests for each panelist. All tests can be performed as well by sensory attribute, by point in time, and by attribute at each point in time. In addition, we propose a way to graphically display the many thousands of p values derived. The method is applied to data from a TDS study on six wheat flakes, indicating that there is substantial disagreement between the panelists. For all panelists, we identify the attributes on which they disagree with the remainder of the panel. If these disagreements are confirmed in other studies, the panelist might require additional training.