Shell Commands Research Articles

Pipelines [Unix programming

One of the underlying philosophies of the Unix command shell is the idea of the pipeline: take a set of small, specialized programs and let them interact by feeding the output of one into the input of the next. By stringing simple programs together in this way, rich behavior can emerge. Pipelines can let you replace complex program logic with a chain of far simpler discrete steps.

Virtual private grid: a command shell for utilizing hundreds of machines efficiently

We describe design and implementation of virtual private grid (VPG), a shell that can utilize many machines distributed over multiple subnets. VPG works around common security policies (e.g., firewall, private IP, DHCP) that restrict communication between machines and even break uniqueness of IP addresses. VPG provides the following functions: (1) a unique nickname to each machine that does not depend on a DNS name or a fixed IP address; (2) job submissions to any nicknamed machine; (3) redirections from/to a file on any nicknamed machine; (4) pipes between commands executed on any nicknamed machine. VPG implements the above functions by constructing a self-stabilizing spanning tree among machines and forwarding messages via a path in the tree. We ran VPG on about 100 nodes (270 CPUs) and measured a turn around time of a small job submission with VPG and other tools: rsh, SSH, and globus-job-run. The experimental result shows that VPG can submit a job faster than SSH and globus-job-run, since VPG performs authentication only when it constructs a tree.

Host-based intrusion detection using dynamic and static behavioral models

Intrusion detection has emerged as an important approach to network security. In this paper, we adopt an anomaly detection approach by detecting possible intrusions based on program or user profiles built from normal usage data. In particular, program profiles based on Unix system calls and user profiles based on Unix shell commands are modeled using two different types of behavioral models for data mining. The dynamic modeling approach is based on hidden Markov models (HMM) and the principle of maximum likelihood, while the static modeling approach is based on event occurrence frequency distributions and the principle of minimum cross entropy. The novelty detection approach is adopted to estimate the model parameters using normal training data only, as opposed to the classification approach which has to use both normal and intrusion data for training. To determine whether or not a certain behavior is similar enough to the normal model and hence should be classified as normal, we use a scheme that can be justified from the perspective of hypothesis testing. Our experimental results show that the dynamic modeling approach is better than the static modeling approach for the system call datasets, while the dynamic modeling approach is worse for the shell command datasets. Moreover, the static modeling approach is similar in performance to instance-based learning reported previously by others for the same shell command database but with much higher computational and storage requirements than our method.

A procedure for landslide susceptibility zonation by the conditional analysis method

Numerous methods have been proposed for landslide probability zonation of the landscape by means of a Geographic Information System (GIS). Among the multivariate methods, i.e. those methods which simultaneously take into account all the factors contributing to instability, the Conditional Analysis method applied to a subdivision of the territory into Unique Condition Units is particularly straightforward from a conceptual point of view and particularly suited to the use of a GIS. In fact, working on the principle that future landslides are more likely to occur under those conditions which led to past instability, landslide susceptibility is defined by computing the landslide density in correspondence with different combinations of instability factors. The conceptual simplicity of this method, however, does not necessarily imply that it is simple to implement, especially as it requires rather complex operations and a high number of GIS commands. Moreover, there is the possibility that, in order to achieve satisfactory results, the procedure has to be repeated a few times changing the factors or modifying the class subdivision. To solve this problem, we created a shell program which, by combining the shell commands, the GIS Geographical Research Analysis Support System (GRASS) commands and the gawk language commands, carries out the whole procedure automatically. This makes the construction of a Landslide Susceptibility Map easy and fast for large areas too, and even when a high spatial resolution is adopted, as shown by application of the procedure to the Parma River basin, in the Italian Northern Apennines.

EMAN: Semiautomated Software for High-Resolution Single-Particle Reconstructions

We present EMAN (Electron Micrograph ANalysis), a software package for performing semiautomated single-particle reconstructions from transmission electron micrographs. The goal of this project is to provide software capable of performing single-particle reconstructions beyond 10 Å as such high-resolution data become available. A complete single-particle reconstruction algorithm is implemented. Options are available to generate an initial model for particles with no symmetry, a single axis of rotational symmetry, or icosahedral symmetry. Model refinement is an iterative process, which utilizes classification by model-based projection matching. CTF (contrast transfer function) parameters are determined using a new paradigm in which data from multiple micrographs are fit simultaneously. Amplitude and phase CTF correction is then performed automatically as part of the refinement loop. A graphical user interface is provided, so even those with little image processing experience will be able to begin performing reconstructions. Advanced users can directly use the lower level shell commands and even expand the package utilizing EMAN's extensive image-processing library. The package was written from scratch in C++ and is provided free of charge on our Web site. We present an overview of the package as well as several conformance tests with simulated data.

OpenSim: A Flexible Distributed Neural Network Simulator with Automatic Interactive Graphics

An object-oriented simulator called OpenSim is presented that achieves a high degree of flexibility by relying on a small set of building blocks. The state variables and algorithms put in this framework can easily be accessed through a command shell. This allows one to distribute a large-scale simulation over several workstations and to generate the interactive graphics automatically. OpenSim opens new possibilities for cooperation among Neural Network researchers. © 1997 Elsevier Science Ltd.

The HetNOS network operating system

The HetNOS network operating system is a set of software layers laid over "native" operating systems to provide a distributed programming platform. The environment is composed of the HetNOS shell command language and the system calls interface (accessed through a procedure library). In both levels of interaction with users, the set of machines integrated by HetNOS are seen as a distributed virtual machine.The HetNOS command interpreter, namely hsh , implements most functions present in more traditional command interpreters. It is possible to spawn, monitor, and terminate processes in any host in the network like in the local case. The HetNOS distributed kernel uses a symbolic, global, location independent, process identification scheme. Distributed applications are split into sequential processes, which interact with each other by message exchange. There are neither connections nor ports, being the communication mechanism strongly influenced by the process identification scheme. This paper briefly describes the HetNOS software organization, presents the HetNOS environment for distributed programming, and then compares HetNOS with related work.

LISA — a powerful program package for Listmode and spectral data analysis

LISA is a graphical program package which enables both off-line listmode and spectral data evaluation as well as on-line monitoring while multi-parameter experiments are running. It can be executed on every computer with a UNIX operating system and an X-WINDOW environment, running PV-WAVE from Visual Numerics Incorporation. This package is basically written in the language PV-WAVE CL, but integration of procedures written in the C-language and execution of UNIX shell commands lead to an additional increase of performance.

A unix tool for managing reusable software components

AbstractSloth is a simple Unix‐based tool for creating and maintaining large C programs built from reusable ‘modules’. Sloth is a collection of UNIX shell commands for creating and editing modules, and for producing executables by linking compiled code from the modules comprising a program. Sloth in a sense extends C by providing module facilities similar to those built in to newer languages such as Modula‐II. The Sloth ‘extensions’, however, are at the shell level only. No changes are required in C itself; in fact, the Sloth commands invoke the standard Unix C compiler.A Sloth module contains a set of C procedures and local declarations, a set of global definitions and variable declarations, an initialization routine and an ‘import’ list of other modules containing declarations referred to, but not defined by, the module in question. The Sloth link command has one argument, the name of a ‘root’ directory. The link command automatically identifies all those modules imported directly or indirectly by the root module, checks that the object code for each module is up‐to‐date and recompiles where necessary. It then creates a main program in which the various initialization routines are executed in the correct order (most primitive first), compiles it and links it with all the object files and produces an executable program.Individual modules can be compiled without any knowledge of the application that uses, directly or indirectly, the module in question. Two applications that both use a collection of modules can therefore share the object code for these modules.The use and implementation of Sloth are presented, as are the experience in using it, programming techniques developed to take full advantage of its facilities, as well as several extensions, either completed or planned.

Design and implementation of a distributed file system

We introduce a new model for replication in distributed systems. The primary motivation for replication lies in fault tolerance. Although there are different kinds of replication approaches, our model combines the advantages of modular redundancy and primary-stand-by approaches to give more flexibility with respect to system configuration. To implement such a model, we select the IBM PC-net with MS-DOS environment as our base. Transparency as well as fault-tolerance file access are the highlights of our system design. To fulfil these requirements, we incorporate the idea of directory-oriented replication and extended prefix tables in the system design. The implementation consists of a command shell, a DOS manager, and a recovery manager. Through this design, we can simulate a UNIX-like distributed file system whose function is compatible with MS-DOS.

SCAMP: A metabolic simulator and control analysis program

Previously developed metabolic simulation programs have concentrated on providing the facility to make time-dependent simulations of metabolic networks; few have been concerned with the steady state and even fewer with any steady state analysis, in particular metabolic control analysis. Here we describe a new simulation program SCAMP, which includes the facility to make time-dependent simulations, but in addition incorporates many of the concepts of metabolic control analysis. SCAMP makes available all the coefficients defined in the control analysis of Kacser and Burns [1] and Heinrich and Rapoport [2]. Thus control coefficients, response coefficients (including the conserved cycle coefficients defined by Hofmeyr et al. [3]) and elasticities (including kappa and pi elasticities [4]) can all be calculated easily. Two other special facilities have also been incorporated; conserved cycle identification and the inclusion of a simple data base of enzyme rate laws. SCAMP will accept models in terms of mass action reactions, user-defined (or data base-defined) rate law reactions or differential equations. SCAMP was primarily implemented for the Atari ST microcomputer since the ST supports a large continuous memory map. However, it is possible to run SCAMP on an MS-DOS type microcomputer provided there is at least 640K of memory. We also supply a simple command shell which integrates the various parts for both the Atari and MS-DOS version. The Atari version also includes a useful graphics plotting package.

A pipe has two ends …

A proposal for a flexible but easy to use syntax. Many mechanisms have been proposed for accessing data, and running processes external to APL. For some of these, the syntax is bizarre, for others a detailed knowledge of some other language (sometimes even assembler) is necessary to do the simplest thing. Some, such as shared variables, offer a highly efficient method of passing data between two processes in the same processor, without having to copy the data. A number of methods either allow the passing of data and/or commands to an external shell (such as the C-shell under Unix), or allow the results of commands to the shell to be imported into APL. These in effect connect APL to one end (but not both ends) of a pipe of shell commands. I offer for general consideration a device that allows data to be piped out of APL, through (a series of) shell commands, and back into APL. It works by connecting both ends of a pipeline of shell commands to an APL statement. The output of the first part of the APL expression fills the pipe connected to the standard input of the first command in the pipeline of shell commands; the next part of the APL expression receives the output of the pipe connected to the standard output of the the last command in the pipeline. The commands may be executed in the same processor, or remotely, with no change in syntax.

Into STELLAR space

Previously developed metabolic simulation programs have concentrated on providing the facility to make time-dependent simulations of metabolic networks; few have been concerned with the steady state and even fewer with any steady state analysis, in particular metabolic control analysis. Here we describe a new simulation program SCAMP, which includes the facility to make time-dependent simulations, but in addition incorporates many of the concepts of metabolic control analysis. SCAMP makes available all the coefficients defined in the control analysis of Kacser and Burns [1] and Heinrich and Rapoport [2]. Thus control coefficients, response coefficients (including the conserved cycle coefficients defined by Hofmeyr et al. [3]) and elasticities (including kappa and pi elasticities [4]) can all be calculated easily. Two other special facilities have also been incorporated; conserved cycle identification and the inclusion of a simple data base of enzyme rate laws. SCAMP will accept models in terms of mass action reactions, user-defined (or data base-defined) rate law reactions or differential equations. SCAMP was primarily implemented for the Atari ST microcomputer since the ST supports a large continuous memory map. However, it is possible to run SCAMP on an MS-DOS type microcomputer provided there is at least 640K of memory. We also supply a simple command shell which integrates the various parts for both the Atari and MS-DOS version. The Atari version also includes a useful graphics plotting package.

Basic Pre‐ and Post‐processors for fortran Programs

The BASIC language, available on almost all personal computers in interpreter form, has a number of strengths and some major weaknesses. It is a powerful language in its ability to read to and from the monitor screen, to manipulate strings, and to perform simple graphics. It is a weak language for use in computationally intensive programming, and, in the form usually provided with microcomputers, cannot access the math coprocessor. The FORTRAN language is good for working with large arrays and for performing extensive computations with high precision numbers, but is weak in the areas where BASIC is strong.A typical engineering program consists of three phases, data input, computation, and data output. The user of a microcomputer program expects to be offered easy interactive data entry and correction, fast computation, graphical review of results where applicable, and foolproof program operation including graceful recovery from errors accompanied by meaningful error messages. Satisfying these desires can be accomplished by the programmer who properly meshes the complementary strengths of BASIC and FORTRAN. This paper describes how to write an engineering program package which uses BASIC in accomplishing data input through data‐entry screens, FORTRAN in the number crunching phase, and BASIC in the post‐processing graphical display phase. Associated problems of formatting files for use by both BASIC and FORTRAN programs, entry of default values during initial data input or data change, and data base management are discussed.The BASIC discussed herein is a Microsoft product available on IBM machines as IBM BASIC and on many compatible machines as GW BASIC. In either case the version must be recent enough that the SHELL command is available. The FORTRAN compiler used in writing the example program discussed later is the Microsoft Optimizing Compiler, version 4.0.