Direct Manipulation Programming Research Articles

Bidirectional Object-Oriented Programming: Towards Programmatic and Direct Manipulation of Objects

Many bidirectional programming languages, which are mainly functional and relational, have been designed to support writing programs that run in both forward and backward directions. Nevertheless, there is little study on the bidirectionalization of object-oriented languages that are more popular in practice. This paper presents the first bidirectional object-oriented language that supports programmatic and direct manipulation of objects. Specifically, we carefully extend a core object-oriented language, which has a standard forward evaluation semantics, with backward updating semantics for class inheritance hierarchies and references. We formally prove that the bidirectional evaluation semantics satisfies the round-tripping properties if the output is altered consistently. To validate the utility of our approach, we have developed a tool called BiOOP for generating HTML documents through bidirectional GUI design. We evaluate the expressiveness and effectiveness of BiOOP for HTML webpage development by reproducing ten classic object-oriented applications from a Java Swing tutorial and one large project from GitHub. The experimental results show the response time of direct manipulation programming on object-oriented programs that produce HTML webpages is acceptable for developers.

Penrose

We introduce a system called Penrose for creating mathematical diagrams. Its basic functionality is to translate abstract statements written in familiar math-like notation into one or more possible visual representations. Rather than rely on a fixed library of visualization tools, the visual representation is user-defined in a constraint-based specification language; diagrams are then generated automatically via constrained numerical optimization. The system is user-extensible to many domains of mathematics, and is fast enough for iterative design exploration. In contrast to tools that specify diagrams via direct manipulation or low-level graphics programming, Penrose enables rapid creation and exploration of diagrams that faithfully preserve the underlying mathematical meaning. We demonstrate the effectiveness and generality of the system by showing how it can be used to illustrate a diverse set of concepts from mathematics and computer graphics.

Bidirectional evaluation with direct manipulation

We present an evaluation update (or simply, update) algorithm for a full-featured functional programming language, which synthesizes program changes based on output changes. Intuitively, the update algorithm retraces the steps of the original evaluation, rewriting the program as needed to reconcile differences between the original and updated output values. Our approach, furthermore, allows expert users to define custom lenses that augment the update algorithm with more advanced or domain-specific program updates. To demonstrate the utility of evaluation update, we implement the algorithm in Sketch-n-Sketch, a novel direct manipulation programming system for generating HTML documents. In Sketch-n-Sketch, the user writes an ML-style functional program to generate HTML output. When the user directly manipulates the output using a graphical user interface, the update algorithm reconciles the changes. We evaluate bidirectional evaluation in Sketch-n-Sketch by authoring ten examples comprising approximately 1400 lines of code in total. These examples demonstrate how a variety of HTML documents and applications can be developed and edited interactively in Sketch-n-Sketch, mitigating the tedious edit-run-view cycle in traditional programming environments.

Object versioning to support recovery needs

We present object versioning as a generic approach to preserve access to previous development and application states. Version-aware references can manage the modifications made to the target object and record versions as desired. Such references can be provided without modifications to the virtual machine. We used proxies to implement the proposed concepts and demonstrate the Lively Kernel running on top of this object versioning layer. This enables Lively users to undo the effects of direct manipulation and other programming actions.

Can direct manipulation lower the barriers to computer programming and promote transfer of training?

Novices face many barriers when learning to program a computer, including the need to learn both a new syntax and a model of computation. By constraining syntax and providing concrete visual representations on which to operate, direct manipulation programming environments can potentially lower these barriers. However, what if the ultimate learning goal of the novice is to be able to program in conventional textual languages, as is the case for introductory computer science students? Can direct manipulation programming environments lower the initial barriers to programming, and, at the same time, facilitate positive transfer to textual programming? To address this question, we designed a new direct manipulation programming interface for novices, and conducted an experimental study to compare the programming processes and outcomes promoted by the direct manipulation interface against those promoted by a textual programming interface. We found that the direct manipulation interface promoted significantly better initial programming outcomes, positive transfer to the textual interface, and significant differences in programming processes. Our results show that direct manipulation interfaces can provide novices with a “way in” to traditional textual programming.

Integrated Development Environments and Visual Design Environments for ROV Mission Programming

Simplification of the remotely operated vehicles (ROVs) piloting in real time so as to carry out the scientific mission in an efficient, reliable and repeatable way, is an ambitious aim. With the presented ROV interface, adopting the integrated development environment (IDE), pilot and scientists have high-level facilities for their cooperation and synchronization besides the possibility of planning, debugging and executing mission. The capabilities are enhanced with a visual design environments (VDE) that allows the pilot to drive graphically the vehicle through the map. Such system as this, combining the language and action paradigms, integrates their advantages and forms the core of direct manipulation programming for ROV navigation.

QUICK: a tool for graphical user-interface construction by non-programmers

The last decade has placed the superiority of graphical user interfaces over traditional text-based approaches beyond dispute. In almost all contexts, users have found graphical interfaces easier to learn, faster to use, and less error-prone. However, it has been shown that the creation of powerful graphical interfaces takes up to 80% of the time required to develop an application. In our work, we seek to extend the benefits of graphical interaction to the next layer of computer user—the interface designer. Our work in this area is distinguished from other efforts by two important differences. First, while other efforts focus primarily on the design of “standard” user interfaces, our approach emphasizes the creation of unique and innovative interfaces by supporting, among other things, arbitrary user-designed graphical representations, direct specification of animation, and digitized sound. Second, our goal is to cater to the nonprogrammer. Thus, we address a challenging trade-off: maximizing power and flexibility in an extremely simple environment. We explore the utility of the prototype object-oriented paradigm, a high-level userinterface language, and a direct-manipulation programming environment in this context.