\documentstyle[12pt,final,sf,sltt,sfheads]{rccldoc}
\input Cmode.tex
\defcvmacro\cverbatim{\tt}{\baselineskip}
\defcvboxedfilemacro\cverbatimfile{\tt}{12truept}
%\pagestyle{empty}
%\textwidth=14.3truecm
%\textheight=23.8truecm
%\advance\oddsidemargin by 1truecm
%\advance\evensidemargin by 1truecm

\begin{document}

\begin{center}
\large\bf
Real Time Control Under UNIX \\
for RCCL
\end{center}
\bigskip
\begin{center}
\it 
(3rd International Symposium on Robotics and Manufacturing \\
Vancouver, Canada, July 18-20, 1990)
\end{center}
\begin{center}
\vspace{1truecm}
John Lloyd and Mike Parker \\
{\small McGill University, Research Center for Intelligent Machines} \\
\vspace{.5truecm}
Gaylord Holder \\
{\small University of Pennsylvania} \\
\vspace{1truecm}
{\bf Abstract}
\end{center}

\begin{quote}
RCI (Real-time Control Interface) is a C/UNIX package which
gives a programmer the ability to develop real-time control
applications within a UNIX environment. It allows functions
linked into a standard UNIX program to be executed as 
real-time ``tasks'', either on the UNIX (host) CPU or on
an auxiliary CPU connected to the host by shared memory.
RCI has been implemented on MicroVAX-II and Sun-4 systems,
and is used as a platform for the robot programming library RCCL.
The paper outlines the system's design and discusses aspects
of its implementation.
\end{quote}

\chapter{Introduction}

This paper describes RCI (Real-time Control Interface), 
a software package for creating simple real-time tasks in a UNIX
environment. It was developed over the last few years at McGill
University and the General Electric Advanced Technology Laboratory.

RCI was designed as a substrate for implementing RCCL (Robot
Control C Library), a software package for doing robot control
from UNIX [Lloyd, et al.\ 88, Hayward and Paul 86]. In RCCL, robot actions
are requested from a UNIX process, and the trajectories
to implement these actions are computed by a real-time
control task, which communicates with the UNIX process using shared
memory. The control task typically runs at 50 to 100 Hz.,\ which is
beyond the real-time capabilities of most current stock UNIX systems.

The RCI package makes it possible to create such tasks. Depending
on the system, RCI tasks can be
run either on the UNIX CPU itself (with some hooks introduced into
UNIX to make this possible), or on auxiliary CPUs which are
attached to the system backplane and utilize a real-time kernel.

Versions of RCI capable of running tasks on the UNIX CPU have
been implemented on MicroVAX-II systems running 4.3 BSD,
and Sun SPARC systems running SUNOS 3.2 and 4.0.3. A version
capable of running tasks on auxiliary CPUs has been implemented 
for the MicroVAX-II system, using KA630 or KA620 CPU boards attached
to the system Q-bus and running a special kernel written by Mike Parker.
A version capable of using auxiliary SPARC CPUs attached to a Sun system
and running the VxWorks real-time system is currently under development.

\chapter{RCI Functionality and an Example}

From a software point of view, an RCI task looks very much like
a signal handler: a particular {\it control\/} function is designated 
to be executed either on a particular hardware interrupt, or off
of a system clock tick. The control function runs to completion, and
communicates with the parent UNIX process (known as the {\it planning}
task) through shared memory. RCI tasks which run on the
UNIX CPU are executed directly at interrupt level (see section \ref{unixcpu}),
in order to achieve the necessary real-time response. RCI tasks running on 
an auxiliary CPU execute as tasks under the real-time kernel for that CPU.
In either case, control functions do not have access to normal UNIX
system services, although this is not a great loss since such services are
typically time consuming. RCI tasks can directly access all or part
of the external system bus through mapped memory (an idea also used
in the {\tt /dev/bus} project at the University of Pennsylvania
[Holder 89]).

Space limitations prevent a detailed description of the RCI primitives
[Lloyd 88].
Instead, we will give an example RCI program in which a task
running at 100 Hz.\ computes a running average of data collected
from a bus register. The averaged value is read back by the
planning task.

The planning and control tasks can be compiled together as
one program, although the control level portion needs to be placed in a
separate module(s).

\cverbatimfile{exampleCode/plan.c}

\cverbatimfile{exampleCode/control.c}

Compilation is done with the {\tt rcc} command, a front end
to {\tt cc} that contains RCI extensions. To allow
RCI to keep track of what the control level needs,
all control level modules should be placed to the right of
the special keyword {\tt CTRL} on the {\tt rcc} command line. 
The example here could be compiled with the command
\begin{verbatim}
        rcc plan.c CTRL control.c
\end{verbatim}
which will produce a single executable named {\tt plan}.

The program uses {\tt rciCreate()} to set up a task on CPU 0.
{\tt allocMem()} is used to allocate a piece of shared memory
to store the averaged data value.
{\tt controlFxns()} sets the control function to
be {\tt taskFxn}. To arrange for the task to be run at 100 Hz.,\ its
scheduling discipline is set to {\tt ON\_CLOCK} using {\tt
rciScheduling()}, and its execution interval is set to 10 msec.\ using
{\tt rciRate()}.

After initialization, the RCI task is set running with a call to
{\tt rciStart()} ({\tt \~{}0} is a bitmask specifying
all control tasks). The planning task
then reads out the value computed by the control task
for a few seconds before shutting it down with {\tt rciRelease()} and
exiting.

The control task consists of the function {\tt taskFxn()}. 
It uses {\tt getMemByName()} to locate the
shared memory. 
The quantity being averaged is read from location
{\tt REG\_ADDR} on the system bus, which is available through
mapped memory starting at the base address {\tt rciBus}.

\chapter{Scheduling, Communication, and Utilities}

An RCI task may be run according to one of four principal scheduling
disciplines, established using {\tt rciScheduling()} and a few auxiliary
routines:

\begin{description}

\item[\tt ON\_CLOCK] --
The task is run off the system clock at a periodic rate.

\item[\tt ON\_FUNCTION] --
The task is run whenever a polling function, driven by a device interrupt,
returns true.

\item[\tt ON\_TRIGGER] --
A combination of {\tt ON\_CLOCK} and {\tt ON\_FUNCTION}: a trigger
function, executed off of the system clock, wakes up some 
external entity, and this in turn wakes up the task with an interrupt.

\item[\tt ON\_SIGNAL] --
The task is run off of a UNIX signal, in user mode. This is used
for simulation and debugging applications.

\end{description}

Inter-task communication is done primarily through shared memory.
The functions
\begin{verbatim}
        mem = allocMem (name, size, pool)
        freeMem (mem)
\end{verbatim}
may be used to allocate or free a piece of shared
memory that is accessible to any of the program's tasks. 
The allocation routines can be called from the planning level only.
Since RCI tasks may run on different CPUs, shared memory addresses
may differ from one CPU to another. 
Because of this, shared memory addresses should be looked up using
a reference function, such as {\tt getMemByName(name)}, which uses the
memory object's name as a 
reference, or {\tt getMemByAddr(addr)}, which uses its planning
level address. To avoid name space collisions, it is common to
use {\tt getMemByAddr()} and not name the memory objects at all.

Shared memory objects, or portions of them, may be atomically read or
written using the primitives
\begin{verbatim}
        readMem (mem, buf)
        writeMem (mem, buf)
        copyMem (mem, src, dst, size) 
\end{verbatim}
While one task is referencing an object through one of these primitives,
all other references through these primitives to the same object are
blocked.

The {\tt allocMem()} facility is optimized for handling small
chunks of memory (typically 1 Kbyte or less).
To allocate larger amounts of memory, it may be better to use the
primitives
\begin{verbatim}
        rciGlobalMemory (name, size, pool)
        rciSharedMemory (name, size, pool)
\end{verbatim}
The memory allocated by these routines cannot be recycled. However,
the memory allocated by {\tt rciGlobalMemory()} can be accessed
by other UNIX processes or RCI programs, which is useful for data logging
or monitoring applications.

A simple message passing mechanism has also been built into RCI, 
but has not been used very much. 

A primitive which has proven to be extremely useful is
{\tt rciSignal(num)}, which sends a UNIX signal to the planning task.

For communicating with the outside world, portions of the system
bus are mapped into memory, with base locations at {\tt rciBus}
and {\tt rciIOpage}. What part of the bus is mapped depends on the system:
on MicroVAX-II systems, {\tt rciBus} is the entire Q-bus and {\tt
rciIOpage} is the Q-bus I/O space. On Sun systems, {\tt rciBus}
is a portion of VME24d16 space, and {\tt rciIOpage} is the 
VME16d16 space.
Direct bus access removes the need to go through the system in order
to interface with peripheral devices.

UNIX system calls, as well as most of the RCI allocation and task
control primitives, are not available to the control
level. However, the control level is provided with a simple set of
utility functions, such as:
\begin{verbatim}
        rciPrintf(fmt, ...)     -- diagnostic print routine
        rciMalloc(size)         -- basic memory allocation
        rciFree(mem)
        rciTime()               -- read system time (in msec.)
        rciBadAddr(addr,size)   -- check memory on the bus
        rciAbort(code,msg, ...) -- abort the program 
\end{verbatim}

\chapter{Implementation of RCI on the UNIX CPU}\label{unixcpu}

A good way to implement RCI tasks on the UNIX CPU
would be to invoke the control function from interrupt
level using UNIX signals. Unfortunately, this is
too slow ([Holder 89]). 
Instead, the RCI task is executed directly 
at interrupt level, in kernel mode. This requires
attending to the following issues:

\begin{enumerate}

\item {\it Context switching\/}.
The memory context must be set to that of the
RCI process (since the system may be inside any context when the 
interrupt arrives). The RCI interrupt code masks out all other system
interrupts, switches to the RCI process context, executes the 
control function, restores the original context, and then unmasks interrupts.
This switching can be done quickly since we are only concerned with the
address space context. On the MicroVAX-II system, the page table base and
length registers ({\tt p0br}, {\tt p0lr}, {\tt p1br}, and {\tt p1lr})
are switched. On Sun/SPARC
systems, the memory management unit (MMU) context number is switched (it is
also necessary to lock the context number to the process so that it won't
be given to another process).

\item {\it Memory locking\/}.
All memory referenced by an RCI task must be physically resident.
Those portions of the program's text and data used by the control level
must be locked down. To do this, a pair of locking primitives,
{\tt memlock()} and {\tt memunlock()}, were added to
the MicroVAX-II and Sun kernels. 

\item {\it Exception Handling\/}.
When an exception (such as a divide by zero or a memory access fault) occurs
in kernel mode, the normal response of the system is to crash, since it
indicates a kernel bug. Since this is unacceptable when running
RCI applications, a hook was put in the system trap code to catch
exceptions occurring within an RCI task and terminate the program
more gracefully.

\end{enumerate}

Shared memory between tasks running on the arbiter CPU
is trivial to implement because the tasks and the UNIX process
share the same address space. 
Creating shared memory is simply a matter of allocating it and
locking it down. Shared memory between an RCI program and other UNIX
processes is done by allocating memory out of a
pool of kernel memory whose pages have been made
user-accessible. The bus interface ({\tt rciBus} and {\tt rciIOpage})
is created by mapping the appropriate bus region
into system virtual space and making it user accessible.

\chapter{Implementing RCI on auxiliary CPUs}

Implementing an RCI capability on auxiliary CPUs is much ``cleaner''
than piggy-backing RCI onto UNIX, since the auxiliary CPUs
can be booted with a real-time kernel that provides the necessary 
performance characteristics. Communication between the auxiliary CPUs
and the arbiter is done using shared memory over the system bus.

To simplify the implementation, each auxiliary CPU may be attached
to only one RCI process at a time.
When a call to {\tt rciCreate()} first references
a particular CPU, that CPU is checked for availability
and booted with the real-time kernel if necessary.
The CPU is then loaded with those portions of the RCI process
text and data segments used by the control
level\footnote{This assumes that the auxiliary CPU
can execute from the same image as the RCI process itself, which is true
for implementations to date. In cases where this was not true, the {\tt rcc}
command could produce a separate image for the auxiliary CPU.}.
A monitor is then started which accepts instructions from the UNIX
CPU to set up RCI tasks and run them when the appropriate 
scheduling conditions arise.

Communication between auxiliary CPUs and the arbiter CPU is done through
shared memory. Each CPU exports a region of memory onto the external
bus for access by all the other CPUs. The other CPUs
import this memory into a specific region of their virtual
address space.
It would also be possible to use separate
memory modules, although this has not yet been done.

\chapter{Discussion}

RCI provides a connect-to-interrupt facility for UNIX, allowing
processes to create high-priority tasks for doing data acquisition or process
control. Its principal use has been to provide the trajectory
generation task for RCCL. The system concept is very simple, but this
has allowed it to be implemented on a variety of hardware systems.

The main reason the system has been useful
is probably because of the well-defined scheduling structure 
of the tasks it runs: RCI control tasks are assumed (more or less) to execute
periodically at a
known sample rate. Such tasks are
easy to synchronize with, and they are also conducive to shared memory
communication (a certain frequency of data access can be assumed, double
buffering is feasible, etc.). This is also the reason that the RCI message
passing primitives have not been used much.

\chapter*{References}
{
\footnotesize
\def\ref #1{\par
\noindent {\footnotesize\bf #1}\quad\hangindent=2\parindent\hangafter=1\relax}

\ref{[Holder 89]}
Gaylord Holder,
``A General Device Driver for ULTRIX or Leave the Driver to /dev/bus''.
Technical Report MS-CIS-89-10, Grasp Lab 174,
Department of Computer and Information Science, 
University of Pennsylvania, Philadelphia, Pa.
\medskip
\ref{[Hayward and Paul 86]}
Vincent Hayward and Richard Paul,
``Robot Manipulator Control Under UNIX: RCCL, a Robot Control C Library''.
{\it International Journal of Robotics Research},
Winter 1986, pp. 94 -- 111.
(Vol. 5, No. 4)
\medskip
\ref{[Lloyd et al. 88]}
John Lloyd, Mike Parker, and Rick McClain,
``Extending the RCCL Programming Environment to Multiple
Robots and Processors''. 
{\it IEEE Conference on Robotics and Automation}, 
Philadelphia, Pa., April 24-29, 1988, pp. 465 -- 469
\medskip
\ref{[Lloyd 88]}
John Lloyd, ``RCI User's Guide''.
Report CIM-88-22, McGill Research Center for Intelligent
Machines, McGill University, Montreal, January 1988.

}
\end{document}
