In recent years, time correlators have received renewed attention, especially under the guise of identifiers of nonclassical correlations. However, the physical interpretation of these objects, and more generally of multi-times variables, remains ambiguous, which may be one of the reasons why they are so difficult to measure. In this work, we introduce and advance the perspective that a two-time correlator should actually be regarded as an average involving a novel single physical observable, one that cannot be rephrased in terms of the primitive ones, according to quantum principles. In particular, we provide examples showing that the presumed constituents of a two-time correlator and the proposed two-time operator itself cannot be simultaneous elements of the physical reality.