Semantic scaffolding: the co-construction of visualization meaning through reader experience

Abstract

Information visualization has practical value – it is a means to gain insights, reveal patterns, shape decision-making and provoke a reaction from the reader. However, current evaluation methods are conceived in terms of the transmission of information, rather than use of visualization in practice. This has resulted in a divergence between the factors known to be important in practice (such as engagement and the reader's visual literacy), and the factors considered in the formulation of visualization design guidance. Even for the relatively simple case of static visualization, a new theoretical contribution is needed to describe the interaction between visualization design choices and the reader's prior knowledge.In this research, a semiotics approach was used to extend the theory of visualization semantics, with the aim of developing an understanding of the context-specific meaning of visual features. The term semantic scaffolding is used in this thesis to describe the capacity for features in a visualization take on additional meaning when read in a specific context.The first part of the thesis concerns the use of visual conventions and figurative images (e.g. illustrations) to convey meaning in information visualization. Content analysis of visualization exemplars examined whether the reader's prior knowledge is anticipated in information visualization practice. The analysis investigated the prevalence of visual conventions and figurative images in data visualizations and infographics. The content analysis showed that figurative elements and conventions are commonly used in exemplars of visualization practice. Visual conventions anticipate the readersʹ familiarity with the prior meaning of visual features. Figurative images convey descriptive information by anticipating the reader's knowledge of and ability to recognise the visual forms of objects.Both figurative images and visual conventions are currently undervalued within the information visualization research community, because the way they contribute to meaning does not fit information transmission conceptualizations of visualization. By extending concepts from geospatial visualization, this research constructed a critical vocabulary for the descriptive information conveyed by figurative images. The critical vocabulary for figurative images articulates the increased expressive capabilities of information visualization layouts which use both figurative and abstract representation ('hybrid' information visualizations).For the second part of the thesis, a new conceptual model was developed through the synthesis of existing models and experiments on visualization construction and interpretation. The new conceptual model developed in this project – called the semantic scaffolding model – assigns a specific role to the reader’s prior knowledge in creating practical value. It is premised on the co-construction of meaning through visual encoding and the reader's context-specific knowledge (semantic scaffolding). Motifs, or intermediate level visual features (groups of marks or mark attributes encoding multiple data values), are a key component of the semantic scaffolding model. Motifs are created indirectly through the transformation of data into a visual form, but can be perceived as unitary visual forms with a coherent, context-specific meaning. Both intermediate level motifs and information-rich figurative images are visual features which can have practical meaning for the reader beyond the meaning encoded by the data-to-visual transformation.The semantic scaffolding model and associated research is important in providing a technical language and set of concepts which can shape future research in the field of information visualization. To prevent a widening rift between information visualization research and practice, the design of experiments, evaluation measures and heuristics needs to shift away from the current focus on the encoded meaning of individual visual values. Instead, research needs to be designed around visual features and their practical meaning for the reader.