The structuring of systems using upcalls

Abstract

When implementing a system specified as a number of layers of abstraction, it is tempting to implement each layer as a process. However, this requires that communication between layers be via asynchnonous inter-process messages. Our experience, especially with implementing network protocols, suggests that asynchronous communication between layers leads to serious performance problems. In place of this structure we propose an implementation methodology which permits synchronous (procedure call) between layers, both when a higher layer invokes a lower layer and in the reverse direction, from lower layer upward. This paper discusses the motivation for this methodology, as well as the pitfalls that accompany it. 1 I n t r o d u c t i o n This paper is concerned with a methodology for program structure, a methodology suitable for operating system programs, especially programs dealing with communications and networks. This methodology arose out of our earlier research in the implementation of network protocols, in which recurring performance problems with protocol software led us to the conclusion that many operating systems failed to provide the correct runtime support for highly interactive parallel software packages such as protocols. This paper describes and motivates this methodology, and discusses the operating system, Swift, which we built to explore it. The methodology described in this paper is relevant to programs which have been modularized according to the This research was supported by the Advanced Research Projects Agency of the Department of Defense and was monitored by the Office of Naval Research under contrac~ N00014-75-C-0681 and N00014-83-K-0125. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. © 1985 A C M 0 8 9 7 9 1 1 7 4 1 12/85-0171 $ 0 0 . 7 5 principle of layering. Traditionally, a layer is thought of as providing services to the layer above, or the client layer. The client uses some mechanism for invoking the layer, perhaps a subroutine call. The layer performs the service for the client and then returns. In other words, service invocation occurs from the top down. As we will discuss below, there are organizational and modularity reasons why this downward flow of control is appealing in a layered system. However, the natural flow of control is not always downward. In a network driven environment, for example, most of the actions are initiated, not by the client from above, but by the network from below. The natural flow of control is thus upward, not downward. Especially where such an upward flow of control crosses a protection boundary, most systems do not permit this flow to be implemented as a procedure call. Instead, some more cumbersome and asynchronous mechanism must be used, such as an interprocess communications signal. Substantial inefficiencies and complexities can result from asynchronous upward flows. In our methodology, the system is organized so that the programmer has the choice as to whether an upward flow is implemented by procedure calls or asynchronous signals. We call this feature upcalls. We chose the word upcall to distinguish from the structured view of service invocation, organized around downward flow. An upcall need not go precisely upward. Our goal is that procedure flow should map on to the natural flow of control in the program, whether that is up, down, or sideways. With procedural invocation available in this way, there is thus no reason to use processes and interprocees communication unless there is an intrinsic source of asynchrony. The other half of this programming methodology is an approach towards structuring a layer. In many systems, a layer is implemented as a task or process. In our methodology a layer is organized as a collection of subroutines which live in a number of tasks, each subroutine callable as appropriate from above or below. Subroutines in different task that make up a layer constitute, in our terminology, a module. A multitask module also contains a collection of state variables, which are accessible, using shared memory, from the different tasks which execute the subroutines of the layer. The purpose of this paper, therefore, is to describe the programming methodology which we call upcalls and multi-task