The analysis of temporal check-all-that-apply (TCATA) data

Abstract

Temporal check-all-that-apply (TCATA) extends classical check-all-that-apply (CATA) by adding a temporal dimension to the evaluation. From a data analysis point of view, TCATA data are similar to Temporal Dominance of Sensations (TDS) data but differ in that more than one attribute can be selected at any time point. Procedures for analyzing TCATA data can hence be generalized from methods for CATA as well as for TDS.TCATA data can be organized in a matrix format with 0s and 1s similar to what has been described before for TDS, but with the relaxation that the column sums can exceed 1. Consequently, the same randomization tests as suggested for TDS are suitable for TCATA data. For TDS data, ad-hoc chance and significance limits are frequently used, a practice that we review critically. We also show that these do not generalize easily to TCATA data. Instead, we suggest comparing individual products to aggregate data across all other products in the test, with that aggregation being assumed to provide an estimate of what is to be expected in the respective product category. This approach does not rely on the number of attributes considered and is also recommended for TDS data. We further suggest a new way to visualize the respective results.The results are compared to the naïve approach of applying standard CATA analyses to the data by time point. Results are reasonably close to warrant very similar interpretation, such that applying CATA analyses by time point to TCATA data seems a potential ad-hoc alternative in practice for preliminary data evaluation.Additionally, panel agreement for TCATA data is obtained over time using Gwet’s AC1 coefficient and its 95% confidence bands, along with raw agreement and mean citation rate. Results provide insight into the level of panel agreement across the various time segments.We exemplify the suggested methods by means of data from a study on Syrah wine finish.