Representations, Affordances, and Interactive Systems

Robert Rowe

doi:10.3390/mti5050023

Abstract

The history of algorithmic composition using a digital computer has undergone many representations—data structures that encode some aspects of the outside world, or processes and entities within the program itself. Parallel histories in cognitive science and artificial intelligence have (of necessity) confronted their own notions of representations, including the ecological perception view of J.J. Gibson, who claims that mental representations are redundant to the affordances apparent in the world, its objects, and their relations. This review tracks these parallel histories and how the orientations and designs of multimodal interactive systems give rise to their own affordances: the representations and models used expose parameters and controls to a creator that determine how a system can be used and, thus, what it can mean.

Highlights

Affordances, and Interactive Systems.Music is an inherently multimodal experience
This standardized representation is basically that of music as played on a keyboard—Musical Instrument Digital Interface (MIDI) was conceived as a communications protocol between keyboards and synthesizers, so sends out note numbers and velocities
We have considered the centrality of representations in shaping the possibilities for multimodal interaction

Summary

Introduction

A similar process converts sound waves into sampled amplitude values, and following our understanding of sampled representations, both can be extensively treated using digital signal processing Such approaches are multimodal in their combination of sensorial modalities to guide interaction between a human and a machine. We may consider different modalities of thought—in particular, between high-level, symbolic representations and the rule-like computations they afford, as opposed to raw signals feeding into networks that converge on a state able to recognize salient differences This contrast—between leveraging domain knowledge over a system of symbols and learning classifications and operations through exposure to examples—has been a primary axis of design focus throughout the history of artificial intelligence and has become a critical research question today, as the tools of AI are adapted to the use case of algorithmic composition. We will consider how the development of multimodal interactive systems could contribute to the decades-old question of how best to harmonize symbolic domain knowledge with purely data-driven models

Symbolic and Sub-Symbolic Representations

Algorithmic Composition

Artificial Neural Networks

Ecological Perception

Representations and Affordances

Conclusions