\documentstyle[12pt,final,twoside,times,hyphentt,sfheads]{rccldoc}
% 
% reduce ident for refitems, and boast the tolerance, to make
% TeX's jobs a little easier ...
%
\refItemIndent=1truein
\tolerance=1000
%
\begin{document}
\pagenumbering{roman}

\title{MicroVAX-II Kernel Support for RCI}
\author{John Lloyd and Mike Parker}
\date{November 1987}
\maketitle
\tableofcontents
\newpage
\pagenumbering{arabic}
\parskip = 2ptplus3pt

\chapter{Purpose}
The Robot Control Interface (RCI) is a software package which has been 
embedded into Berkeley UNIX 4.2 and 4.3 at McGill University to allow
the creation and execution of real-time robot control tasks. 

This document explains what was involved in doing this, as an aid in 
porting RCI to other UNIX systems. It assumes an intimate knowledge
of UNIX, UNIX kernel internals (particularly 4.2/4.3 bsd and its
derivatives), and of course the programming language C.

\chapter{Requirements}

Those aspects of robot control which are concerned with trajectory
generation and servoing are usually implemented by synchronous,
periodic control tasks, which execute at rates typically ranging from
10 to 1000 Hz. Work presently being done at the Jet Propulsion Laboratory 
(JPL) requires the concurrent execution of two robot control tasks
at rates of around 200 Hz. Since a fairly regular sample rate is
required for good control, the latency associated with these tasks
must be absolutely minimal, around 0.5 msec or less.

\chapter{Description}

RCI allows the user to create and execute a robot control task
as a part of a main UNIX program. The control task is a function
that executes much the way a signal handler tied to an 
interrupt line would. I/O with the robot is done automatically by RCI and
communicated to the control functions through blackboard variables. Usually,
RCI reads feedback information such as joint positions
(encoder values) from the robot, calls the control task,
and sends back commands such as motor current values or encoder setpoints 
(there is usually a very low level controller on the robot side that can
either send currents directly to the joints or execute very simple PID 
joint-level commands).

To make it complicated, we actually allow the user to specify
two control functions, which are interleaved with the robot I/O.
This allows the programmer to optimize computation (more on this
later).

A simplified illustration of how it all works is shown below:
\begin{verbatim}
/* ------------  a (very) stylized RCI program ------------------ */

ctrlFxn1();                 /* the robot control functions */
ctrlFxn2();

struct CONTROL_BUF ctrlCom; /* big fluffy structure with all sorts of
                               parameters and things that the main prog.
                               uses to interact with control functions */
main()
{
   ... initialize, print "Welcome to program", etc ...

   rciOpen();                              /* set up RCI system */
   rciControl(ctrlFxn1, ctrlFxn2);         /* start control task */

   while (notFinished)
    { 
      ... wait around, query user, write stuff to files ...

      prepareRobotCommand (&ctrlCom);      /* issue robot command and
      ctrlCom.set = 1;                        wait for control function
      while (ctrlCom.set)                     to handle it */
         ;
     }
   rciRelease();                           /* stop control task */

   ... unwind, print "bye", etc ...
}

ctrlFxn1()
{
   if (ctrlCom.set)
    { 
      ... do whatever the control structure says ...

      ctrlCom.set = 0;                     /* acknowledge */
    }
}

ctrlFxn2()
{
   ... do support computations for first control function ...
}
\end{verbatim}

The {\tt rciControl()} call connects the control functions to an appropriate
device or timer interrupt which is associated with the robot controller.
When it is ``time to control'', the control function is
executed as an interrupt handler. The main program and the control 
function share the same memory space, so they can communicate using
global variables.

RCI also provides a communication interface between the robot
and the control function: at interrupt time, data is read in from
the robot into a global blackboard (called {\tt how} for historical
reasons) which the control function can then read randomly to decide
what commands to send back to the robot. The commands are specified
by writing into a command blackboard (called {\tt chg}---more history),
which RCI looks at after the control function exits to determine the
appropriate command packet to ship back to the robot. The ``driver''
for the RCI task then looks something like this:

\begin{verbatim}
struct HOW how;         /* blackboard variables with venerable historical
struct CHG chg;            names */

int (*ctrlFxn1Ptr)();   /* pointers to user's control functions */
int (*ctrlFxn2Ptr)();

onInterrupt()           /* driver routine attached to control interrupt */
{
   readFromRobot (&how);   /* get feedback information from robot */
   (*ctrlFxn1Ptr)();       /* execute user control function */
   writeToRobot (&chg);    /* send commands (posted in chg) to robot */
   (*ctrlFxn2Ptr)();       /* execute second control function */
}
\end{verbatim}

The reason for having two control functions is now a little clearer:
the first function provides immediate response to feedback information,
while the second allows longer computations to be done at the
same time as the robot is digesting its most recent batch of commands.
By doing different things in the two functions, the user can balance
response vs.\ parallelism.

RCI offers the advantage that the programmer can think of the robot
control task as ``just another part'' of the robot control program. There
is no need to worry about cross compiling or downloading code to 
another processor, and the interfacing of the main program to the
control task, through global variables, is trivial. The only
real restriction is that the control task cannot execute system 
calls, since it executes at high priority (just how high we will
get into shortly). This is not usually too much of a disadvantage, 
however; system calls often involve lengthy operations, which the control
task cannot afford anyway. File I/O, such as for data logging, 
can be accomplished by writing information to a buffer, which is
then flushed by the main program when appropriate.

\chapter{Implementing RCI on UNIX: the Problems}

If using RCI is straight forward, implementing it is not, principally because
it demands real-time performance from UNIX that Ken and Dennis never
even remotely conceived of. Before we go on, we should make a historical
note that UNIX itself began as an attempt to ``get around'' the limitations
designed into the tools (sic) that existed then.

The cleanest way to implement RCI on UNIX would be to write a 
device driver which, on receipt of a device interrupt, sends a signal
to the robot control program, where a handler has been set up which
execute the control tasks. The driver would look like this:

\begin{verbatim}
onintr()                        /* driver interrupt routine */
{
 readFromRobot(&how);
 signalUser(signal1);           /* send signal to user task */
}

ioctl()
{
 case USER1DONE:                /* this ioctl is called by user task in */
    writeToRobot(&chg);         /* reply to the signal; it is now OK to */
}                               /* write data back to the robot.        */
\end{verbatim}

Unfortunately, this latency associated with
this is too long. Signals are not normally considered to be time critical,
and one may wait until the next system clock tick, or beyond, before
the signal is delivered.
The other problem associated with the basic latency of the signal
mechanism is that user mode priority is usually not
sufficiently high to give the response required for the control
tasks. UNIX often spends several milliseconds executing at high priority,
either doing critical sections or communicating with
devices (this seems to be particularly bad on systems with extensive network
connections). 
Consequently, it is necessary (on 4.2 and 4.3) to run
at an elevated priority within the driver (figuring robotics research
is an important discipline, we use {\tt splhigh()}). In fact, we execute the
entire robot control task from within the driver interrupt routine,
at {\tt splhigh()}, on the interrupt stack. This usually gives the required
response. We will end the paragraph here to give the reader a chance to calm 
down.

Doing this sort of thing is quite feasible but presents us with
a few problems. In particular, we have have to address four issues:

\begin{list}{}{}
\item[1.]{\it Memory context} --
At interrupt time, the current process may not be
the process running RCI, which means that the memory context will be
different. It then becomes necessary to restore the RCI process memory
context for executing the control task, since the control functions and data
exist in that memory.

\item[2.]{\it Memory locking} --
All memory accessed by the control task must be
resident in physical memory; there is not enough time to make a trip to
the swap disk and pick up whatever isn't there. (This problem exists
even if we aren't executing in kernel mode at high IPL. The VMS
installation at NASA Goddard Space Center runs the control tasks in
user mode, but locks all the associated memory).

\item[3.]{\it System crashing} --
Errors (or ``exceptions'', to use the appropriate
euphemism) which occur in kernel mode are normally considered to
be ``can't happen events'' which should result in a system crash.
Of course, the RCI control task is user written and is usually
rife with ``exceptions''. So to prevent system crashes, we have
to modify the system trap handler so that during the execution of
a RCI control task, exceptions simply cause the appropriate UNIX
signal (such as SIGILL or SIGFPE) to be sent to the RCI program.
\end{list}

While we are on the topic of system crashes, we should mention
a couple of things. RCI always presents the possibility of crashing
the system (although it very rarely happens nowadays). The exception
trapping described above does not prevent the possibility that a
bad memory reference will take a shot into some important system data 
structures. This fact often causes system people to get upset. Our
response is twofold: first, it has very rarely occurred (I can think
of no instance in memory where the system crashed because of a
user bug in an RCI program), and second, it doesn't really matter.
A system running RCI is assumed to be essentially a three or four
person work station. If you crash your PC, nobody cares; why worry
here? Besides, even that large jacket-and-tie operating system,
VMS, supports kernel mode user functions (given the appropriate
privileges), and VMS has a lot more data structures than UNIX to 
shoot at. As far as security is concerned, withstanding malicious users 
has never been a design goal of RCI.

\begin{list}{}{}
\item[4.]{\tt Resources} -- As a last item, 
RCI provides the user with a few utility routines
(like a version of {\tt printf}) that can be executed from kernel mode.
This is important since the RCI control task cannot execute system
calls. RCI also provides a couple of variables, {\tt rciIObus} and
{\tt rciIOpage}, that give the programmer direct access to the bus
and I/O space address. This can be used for doing direct I/O without
having to go through a interface routine, or referencing shared
memory that has been set up with another system. (This feature does
not make sense on all systems, such as those where the hardware
does not support memory mapped I/O).
\end{list}

The next section will describe how each of these has been implemented
on UNIX, and describes possible alternatives.

\chapter{Implementing RCI on UNIX: the ``answers''}

The whole of an RCI task is executed inside an interrupt routine that
looks a little like this:

\begin{verbatim}
driverInterrupt()
{
   s = splhigh();          /* shut the door and lock it */

   switchToRCImemory();    /* set appropriate memory context */
   setKernelUserTraps();   /* tell kernel not to crash */

   readFromRobot (&how);
   (*ctrlFxn1Ptr)();
   writeToRobot (&chg);
   (*ctrlFxn2Ptr)();

   clearKernelUserTraps();
   releaseRCImemory();     /* restore previous memory context */

   splx (s);
}
\end{verbatim}

There is also a mechanism (not shown) whereby the control functions can
send signals back to the parent task to notify it of some condition.
This happens, for instance, in the case of programs detected errors.

\section{memory context}

The is embodied by the {\tt switchToRCImemory()} call.

George Gobel, when he wrote the first RCI driver at Purdue University
in 1983 (it was called RTC in those days) referred to the operation
of quickly switching memory contexts, at high priority when the operating 
system wasn't looking, as ``Kentucky'' switching, a name which reflects the
disdain that residents of Indiana have for their neighbors
to the south. 

In theory, an elegant alternative to Kentucky switching
would be to double map the memory associated with the control function 
into system space, and execute from system virtual memory, but this
cannot be done on Berkeley UNIX since the C compiler does not generate
relocatable code (ughh!).  (And requiring that all control functions be
written by hand in assembly is clearly unreasonable.)

Kentucky switching on a VAX is relatively easy: first, you mask all
interrupts ({\tt splhigh()}) so that no one can catch you at it.  Next,
you simply save the memory context registers and replace their
contents with the context of the process you wish. There are 4 such
registers on a VAX: {\tt p0br}, {\tt p0lr}, {\tt p1br}, and {\tt p1lr},
which describe the base address and length of the pages table areas
associated with the two different user memory regions ({\tt p0} and
{\tt p1}).  The problem is finding the correct values of these
registers to load for the desired process. One can simply remember the
values at some
prescribed time, which is fine as long as they don't change, but if
UNIX rearranges the page tables for that process, they will change.
Instead, it is better to obtain the values by referencing the
{\tt proc} structure directly, where the values will always be
correct.

Well, almost$\dots$.

When a process calls {\tt vfork()}, however, its memory context is
temporarily removed and given to the child process until the child
calls {\tt exec()} or {\tt exit()}.  This means that if we look in the
parent process' {\tt proc} structure, we will find nothing useful
there.  Fortunately, however, the parent {\tt proc} structure contains
a pointer to the child's {\tt proc} structure ({\tt p\_xlink}) when this
is true, so we can find out where the memory really is.  (If the child
process calls {\tt vfork()} again during the first {\tt vfork()}, the
memory will be handed to the third process; we just chase down the
pointers until we come to the end.)

Well, almost$\dots$.

It turns out that there are a few spots during the execution of a {\tt
vfork()} where {\it neither} process has the memory context; it has
been removed from one process and not yet given to the other.
(Normally, these never affect anything, for neither process is ever
runnable during this time.  But we are running in an interrupt routine,
remember?)  Fortunately, these ``windows'' are small, and not difficult
to detect, though doing so requires adding another flag to the {\tt
proc} structure.  When this condition is detected, we fall back on
saved copies of the {\tt p0br}, {\tt p1br}, {\tt p0lr}, and {\tt p1lr}
registers.

\section{memory locking}

The original RCI driver solved the memory locking problem by locking
entire address space of the RCI program into memory, using the kernel
function {\tt vslock()}.  However, this prohibited the program from
later growing the memory area (by calls to {\tt brk()}).  The
resolution of this problem was to add system calls to UNIX which permit
the explicit locking and unlocking of memory (though in retrospect, it
might have been easier to use the existing locking facilities).  These
sys calls involved adding a few bits to the {\tt cmap} structure (which
represents a page of physical memory) to indicate which page frames are
locked and how many times each such has been locked.  To lock memory,
the memory is first {\tt vslock()}ed to make sure it is present in
physical memory during the following operations, then each page frame
has its ``user lock'' bit (one of the additions) set and its
``reference count'' (the other addition) incremented.  The memory is
then unlocked with {\tt vsunlock()}.  The process is marked as
non-swappable when this occurs, and the paging routines were modified
to ensure that page frames with the ``user lock'' bit set do not get
paged out to disk; and for robustness, the memory free routine {\tt
memfree()}, which is ultimately called to free memory when a process
exits, was modified to deal with such pages correctly.

\section{system crashing}

The is embodied by the {\tt setKernelUserTraps()} call.

The original RCI driver ignored this issue, merely warning users that
misbehavior such as accessing invalid memory locations would result in
a system crash.  However, after a few fifteen-minute waits for the
system to reboot after a silly mistake in the user code, we said
``Surely there's a better way$\dots$''.

Thus we added a global variable to the kernel.  When this variable is
set, traps such as divide-by-zero or memory access faults are permitted
in kernel mode; they simply store some information about the fault and
throw back (with the kernel version of ``longjmp'') to a catch point
set up by the RCI code.  Since the RCI interrupt service functions run
totally uninterruptably, such special trap handlers can never be
nested, which means the variables controlling this behavior can be
global to the entire system.  The catch point then uses the saved
information to send an appropriate signal to the user process, as if it
had incurred the corresponding fault during normal user-mode execution.

\section{resources}

A driver I/O control is defined which gives the user program the address
of certain kernel level routines and data areas which can be accessed
by the control functions. These include the kernel {\tt printf()}
routine (indispensable for debugging, but obviously not much use in
real-time), an RCI routine which will send signals to the parent
process, the kernel {\tt badaddr()} routines (for checking bus
addresses), an address of a location in memory where there is a
reliable wall clock, and the addresses (in virtual memory)
of the I/O bus areas. 

\chapter{Changes to the Berkeley UNIX 4.3 Kernel}

In reading this next section, it is assumed that the user has a copy
of the modified 4.3 source files. These can be obtained from the
RCCL distribution in the directory {\filefont \$RCCL/sys}.

\section{Basic Remarks}

RCI kernel installation consists of two parts.  There's the main body
of RCI and there are the auxiliary pieces, which are necessary but not
integrated into the rest of RCI.  That is, if existing facilities
provide equivalent functionality, the existing mechanisms can be used
instead.

In particular, RCI requires the ability to lock pieces of a user
process into core, because there's no time at interrupt time to fetch
them from swap.  The facilities for this are provided by the
file {\filefont rciext/map.c}, with support from the following files:
\begin{verbatim}
        h/cmap.h
        rciext/section.h
        sys/init_main.c
        sys/kern_exit.c
        sys/kern_exec.c
        sys/vm_mem.c
        sys/syscalls.c
        sys/vm_page.c
        sys/init_sysent.c
\end{verbatim}
The facilities actually provided
are fancier than are currently used, for historical reasons. 
In particular, they include system calls for creating shared
memory sections between processes. 
This support is conditionalized with {\tt \#ifdef MEMORYMAPPING};
searching for {\tt MEMORYMAPPING} should locate the required code in the
aforementioned files.

If it turns out to be impossible to implement memory locking,
you may be able to get away with simply touching everything to
page-fault it into memory and then hoping it doesn't get swapped out.
This is not recommended if it can be avoided.

RCI needs a few other pieces of support which are conceptually distinct
from RCI, but in the current implementation are rather deeply
intertwined with RCI proper.  These are the kernel-user trap handling,
Kentucky process switching, and the auxiliary CPU interface.

Kernel-user trap handling is a kludge necessitated because user code is
being executed on the interrupt stack at elevated processor priority.
Normally, only kernel code is executed in such a state, so an error
such as division by zero or a segmentation fault causes a panic.  Since
this is user code executing, though, such errors really ought to be
reflected back to the user process as if it were executing normally.
The kernel-user trap handlers allow the RCI code that calls the user
code to set things up so that traps merely cause a kernel {\tt
longjmp()} and send the appropriate signal to the process executing
(which is set up as part of the process of establishing kernel-user
trap handling).  Support for this is conditionalized on {\tt
KERNEL\_USER\_TRAPS} and is mostly in {\filefont vax/trap.c}, though a few
pieces are elsewhere, notably the code to handle machine checks due to
access to invalid Q-bus locations, which is in {\filefont vax/locore.s}.
Searching for {\tt KERNEL\_USER\_TRAPS} should locate all
of the code responsible for this stuff.

Kentucky process switching (the name is historical in origin) consists
of changing memory contexts without doing a full process switch.  This
is necessary because the user code being executed resides in
per-process memory, just like the rest of the user program, and when an
interrupt comes in and we want to execute some RCI control-level task,
it is likely that the process that got interrupted is not the one out
of whose memory context we must execute.  We can get away with this
only because the user code in question is executed at maximum IPL, so
we cannot be interrupted for anything, so nobody can catch us playing
fast and loose with the memory contexts.  (We can't double-map the user
memory into kernel space and execute it there because (a) the C
compiler doesn't generate position-independent code (!) and (b) because
of pointers.)  This code is conditionalized on RCI just like the rest
of the RCI code, because nobody else uses it and it's tailored for
RCI's needs at a fairly basic level.

The auxiliary CPU interface applies only when there are auxiliary CPUs
available.  The system on which development was done is a MicroVAX-II
with auxiliary KA620s plugged into the Q-bus backplane; if similar
hardware facilities are available, RCI can use them.

The rest of the RCI code is mostly interfacing these two pieces of
user-level code---the control task and the planning task---to the
kernel.

\section{File-by-file breakdown}

Here's a file-by-file breakdown of what's been changed and why.  First
we have the changes which are directly related to RCI or its support
code, then some other things which I think deserve mention, for example
because they would otherwise confuse someone trying to understand the
code.

\subsection{rciext}

The directory {\tt rciext} contains the code that {\bf we} wrote\footnote
{We actually means mostly Mike Parker.}.

\begin{verbatim}
rciext/drvclk.c
rciext/drvclkreg.h
\end{verbatim}
\refitem{}
This is the RCI clock device driver. It interfaces to a DRV11
with some timing logic attached, and is used to provide a 
system time base for RCI tasks running on the arbiter CPU. The
driver is very simple: when the device is open, interrupts are
accepted from the DRV11 and are used to increment the RCI clock
for base for the arbiter CPU. Interrupts are disabled when the device is
closed. The interrupt routine uses a count in the device input register to
check for  lost interrupts. If the device is found by the probe routine, it
sets a variable to let the rest of the RCI system know that there is a clock 
present and that the system clock does not need to be used.

\begin{verbatim}
rciext/drvj.c
rciext/drvjreg.h
\end{verbatim}
\refitem{}
An RCI polling interrupt driver for the DEC DRV11-J. The driver for the
DRV11 is located in {\filefont vaxuba/pri.c}.

\begin{verbatim}
rciext/genrci.c
\end{verbatim}
\refitem{}
Generic RCI device driver.  This is an RCI device which
corresponds to no hardware, but provides all the other support
any RCI device must.  The RCI system uses these when it wants
RCI support but has no device handy to use.  As an RCI device,
this driver calls on the support provided by {\filefont rciext/rci.c}.

\refitem{}
This is not a good driver to use as a template RCI device,
because it has had all the hardware support stripped out.
There is a file---{\tt rci\_template.c}---which exists for precisely
this purpose.

\begin{verbatim}
rciext/rci.c
\end{verbatim}
\refitem{}
RCI support routines.  There are routines to initialize the RCI
system, setting up mappings as necessary and such, to determine
whether RCI is alive, to get the interface routines, timer
services, the Kentucky process switch stuff, sending signals,
reflecting Q-bus errors to the user appropriately, utility
functions for RCI device drivers, and internal support stuff.
\refitem{}
This file uses the kernel-user traps facilities provided by
trap.c and contains a good deal of MicroVAX-specific code (e.g.,
assumptions about the Q-bus: its size and layout in the
physical address space).  This stuff is mostly called rather
than caller, though it does call some primitives like {\tt timeout()}
and {\tt mapin()}.  It also calls on some support provided by the {\tt ka}
driver in {\filefont rciext/ka.c}.

\begin{verbatim}
rciext/rci.h
\end{verbatim}
\refitem{}
This file contains the definition of the struct which is used
to interface a device driver to the RCI code.  The bits of code
in {\tt rci\_template.c} illustrate the ``advertised'' uses of this
interface.

\begin{verbatim}
rciext/rci_template.c
\end{verbatim}
\refitem{}
This is a template for building RCI device drivers.  It's a
normal device driver with all the hardware-specific code
stripped out.

\begin{verbatim}
rciext/rciioctl.h
\end{verbatim}
\refitem{}
Ioctl definitions for the RCI-specific ioctls.  This is used by
RCI device driver ioctl routines and by user code when using
the facilities so provided.

\begin{verbatim}
rciext/rcitimeout.h
\end{verbatim}
\refitem{}
Definition of the RCI timeout structure.
(Why is this a separate file?  Presumably there's some reason, but I
can't seem to recall it.)

\begin{verbatim}
rciext/ka.c
\end{verbatim}
\refitem{}
KA device driver.  This provides a software interface to the
auxiliary CPUs: booting them, halting them, downloading
software, etc.  In addition to being a normal device driver for
these purposes, it contains some support routines used by other
files.  The management routines for that part of the Q-bus that UNIX
doesn't use, for example.  (This is called the KA driver because DEC
CPUs are generally called KAnnn for some number nnn, and the
MicroVAX CPU is no exception: KA630 or KA620.)

\refitem{}
Sleeping in the {\tt ka} driver is messy, because each unit can be in
use by multiple processes (this is particularly likely with the
arbiter, unit 0).  So most sleeping which would normally be on
a per-unit cell is on a global cell {\tt kasleepers}, which is
awakened whenever anything interesting happens.

\refitem{}
This driver also knows the memory layout described in {\tt kacomm.h}.

\refitem{}
The minor device number is split up: the {\tt 0x80} bit is a flag
indicating that the device may be opened by more than one
process at a time and the rest of the bits are the CPU number
(because the MicroVAX architecture restricts us to 3 auxiliary CPUs,
this number must be in the range 0--3, 0 corresponding to the arbiter
CPU).

\begin{verbatim}
rciext/kacomm.h
rciext/kacrash.h
rciext/kaioctl.h
rciext/kaipcr.h
rciext/kamsgs.h
rciext/kareg.h
\end{verbatim}
\refitem{}
Include files for KA communication.  These are primarily for
{\tt ka.c}, but other files include them as well when they need to
refer to data structures defined herein ({\tt kacomm.h} for the
shared memory areas, {\tt kacrash.h} for the crash info structure,
etc).

\begin{verbatim}
rciext/local.c
\end{verbatim}
\refitem{}
Local syscalls ({\tt \#ifdef LOCAL}) and two support routines,
{\tt read\_physical} and {\tt write\_physical}, which shuffle memory mapping
hardware to permit reading and writing random physical memory
locations.

\refitem{}
The only thing used from this file is one of the local
syscalls.  It's known as {\tt cmkrnl()} from user mode (the name is a
hangover from VMS) and the kernel routine is {\tt firewall\_hole()}.

\begin{verbatim}
rciext/map.c
\end{verbatim}
\refitem{}
The central file for memory mapping support.  This includes all
the syscall interfaces and all the real work of memory mapping
and locking.

\begin{verbatim}
rciext/section.h
\end{verbatim}
\refitem{}
Structure and constant definitions for the memory mapping code
in {\filefont rciext/map.c}.  Not of interest to anything outside there,
except for debugging purposes.

\begin{verbatim}
rciext/mchk_frame.h
\end{verbatim}
\refitem{}
Machine check frame definitions, moved here from {\filefont vax/machdep.c}
so they're available to other files as well.

\begin{verbatim}
h/cmap.h
\end{verbatim}
\refitem{}
Added two new fields for memory mapping code.  {\tt c\_ulock} says the
page is locked in-core, either from a lock syscall or because
it's mapped between processes; {\tt c\_refcnt} is a count of the
number of references to this page.  It is used to tell when we
can free up the corresponding physical memory.

\refitem{}
Conditionalized {\tt SZ\_CMAP} to match the structure definition.
({\tt SZ\_CMAP} is for {\tt locore.s}, where {\tt sizeof()} can't be used.)

\begin{verbatim}
h/errno.h
\end{verbatim}
\refitem{}
Added errnos {\tt EALIGN}, {\tt EMISMATCH}, {\tt ETABLEFULL}, 
{\tt EDUPLICATE}; these
are returned by the memory mapping code.

\begin{verbatim}
h/ioctl.h
\end{verbatim}
\refitem{}
Changed {\tt IOCPARM\_MASK} from {\tt 0x7f} to {\tt 0x1ff}, to allow larger
structures to be passed back and forth to/from {\tt ioctls}.  This
was done for the {\tt RCIIOC\_GETCRASH ioctl}.

\begin{verbatim}
h/proc.h
\end{verbatim}
\refitem{}
Added {\tt SLOSTVM} for the Kentucky process switch mechanism.  As
the comment indicates, this flag means that the process is in
the middle of a {\tt vfork()} and happens to be in one of the small
sections during which there is no way for the interrupt level
to get hold of the memory context.  (The memory context isn't
totally lost, of course; it's present on the kernel stack, in
local variables in some function.)

\begin{verbatim}
vax/conf.c
\end{verbatim}
\refitem{}
Don't forget to add whatever drivers you're using (such as the
{\tt ka} and {\tt genrci} drivers) to {\tt cdevsw}.

\begin{verbatim}
vax/locore.s
\end{verbatim}
\refitem{}
Raised {\tt NISP} (number of interrupt stack pages) from 3 to 8.
(Because otherwise RCI control tasks tend to run out of stack.)

\refitem{}
Added {\tt jsb} to {\tt save\_kut\_regs} to the machine check handler
(so that Q-bus machine checks can be caught by the
normal kernel-user trap mechanism without losing register contents).

\refitem{}
Added {\tt soft15} for kernel-user traps.  This is necessary for
kernel-user traps to work properly sometimes.  When?  When the
user process is executing on the kernel stack, for example in
the {\tt RCIIOC\_EXECUTE ioctl}, and a trap can put the machine on the
interrupt stack.  {\tt Longjmp()} doesn't work right when executing
on the interrupt stack and jumping to a {\tt setjmp()} established on
the kernel stack.

\refitem{}
{\tt jsb save\_kut\_regs} added to {\tt alltraps:} code.  (Again, for
kernel-user trap handler.)

\refitem{}
{\tt save\_kut\_regs} added.  This saves the machine registers in a
global area for the kernel-user code to pick up later and stuff
into a {\tt CRASHINFO} structure (see {\filefont rciext/kacrash.h}).

\begin{verbatim}
vax/machdep.c
\end{verbatim}
\refitem{}
Added {\tt \#include "ka.h"} and 
{\tt \#include "../rciext/kacomm.h"},
the latter conditional on {\tt (NKA > 0)},
for Q-bus machine check handling.

\refitem{}
Moved machine check frame declarations from in-line struct
definitions here to {\filefont ../rciext/mchk\_frame.h} (so the kernel-user
machine check handler can include it as well).

\refitem{}
Added handling of Q-bus machine checks in trap handler (so
that bad Q-bus accesses from control functions appear as traps
rather than panics).

\begin{verbatim}
vax/scb.s
\end{verbatim}
\refitem{}
Added names for unused soft interrupts; the only one RCI uses
is {\tt soft15}.

\begin{verbatim}
vax/trap.c
\end{verbatim}
\refitem{}
Added {\tt kernel\_user\_traps} variables; added support for them in
{\tt trap()}; added the function-call and {\tt soft15} interrupt interface
mechanisms.  The soft interrupt is necessary because
kernel-user traps works only when executing off the interrupt
stack, because we can't {\tt longjmp()} from the interrupt stack to a
{\tt setjmp()} on the kernel stack, at least not reliably.  So we use
a soft interrupt request to get onto the interrupt stack to
call the function.

\begin{verbatim}
vax/vmparam.h
\end{verbatim}
\refitem{}
Upped {\tt USRPTSIZE} to {\tt 96*NPTEPG} from {\tt 32*NPTEPG},
conditional on {\tt RCI}.  I forget what {\tt USRPTSIZE} controls; it
looks as if it's the size of the user page table, but that's just a
guess based on the name.  I don't remember doing this.

\begin{verbatim}
vax/inline/machpats.c
\end{verbatim}
\refitem{}
{\tt bfill()} support added.  Some of the RCI kernel code uses
{\tt bfill()}, which isn't provided by default.  You can either put
it into your analog of {\filefont vax/inline} or write a {\tt bfill()} routine.
The functionality expected from {\tt bfill()} is equivalent to
\begin{verbatim}
                bfill(p,n,f) char* p; int n; char f;
                 { for (;n>0;n--) *p++ = f; }
\end{verbatim}

\begin{verbatim}
sys/init_main.c
\end{verbatim}
\refitem{}
Call {\tt sectioninit()} to initialize the memory locking/mapping
code (if, of course, it's turned on at compile time).

\begin{verbatim}
sys/init_sysent.c
\end{verbatim}
\refitem{}
New syscalls.  The ones RCI uses are
\begin{verbatim}
                firewall_hole()
                syscall_memlock()
                syscall_memunlock()
\end{verbatim}
\refitem{}
which are, respectively, the {\tt cmkrnl()} mentioned under
{\filefont rciext/local.c} and the syscall interface to the memory locking
code.

\begin{verbatim}
sys/kern_clock.c
\end{verbatim}
\refitem{}
Added an {\tt \#include ka.h} for the doorbell backup interrupt.
Added an {\tt \#include ../vax/cpu.h} for ditto.

\refitem{}
Added {\tt ka} doorbell backup and RCI timer tick in {\tt hardclock()}.  The
former is so that if we miss a doorbell interrupt (because of
the hardware bug), we will make up for it next timer tick.  The
latter is so that RCI can use the arbiter's hardware timer to
run the RCI clock.

\begin{verbatim}
sys/kern_exec.c
\end{verbatim}
\refitem{}
Call {\tt free\_all\_mapping} as well as {\tt vrelvm()}, if memory mapping is
enabled.  This must be done everywhere vrelvm is called; other
files do this as necessary.

\begin{verbatim}
sys/kern_exit.c
\end{verbatim}
\refitem{}
File close code moved to before memory free code; this is
necessary because otherwise an interrupt occurring after the
memory is freed but before the device close routine on an RCI
device has been called can cause us to try to Kentucky switch
to a nonexistent memory context.

\refitem{}
Call {\tt free\_all\_mapping} as well as {\tt vrelvm()} 
(see {\filefont sys/kern\_exec.c}).

\begin{verbatim}
sys/kern_fork.c
\end{verbatim}
\refitem{}
Set and clear {\tt SLOSTVM} as appropriate.  This flag is used to
mark certain times in a {\tt vfork()} during which the memory has
been taken away from the parent and not yet given to the child.
If the parent is running an RCI task and an interrupt strikes
during this time, we have to know it so we don't use invalid
memory context information.  (Yes, parents running RCI tasks do
sometimes {\tt vfork()}, a shell escape from the keytree matcher
being the canonical example.)

\begin{verbatim}
sys/syscalls.c
\end{verbatim}
\refitem{}
Add names for all the new syscalls (see {\filefont sys/init\_sysent.c}).

\begin{verbatim}
sys/vm_mem.c
\end{verbatim}
\refitem{}
Add some {\tt printf\/}s to {\tt vmemfree()} and {\tt memfree()} for
debugging.

\refitem{}
Add support for the new fields in the {\tt cmap} structure (see
{\filefont h/cmap.h}) in {\tt memall()} and {\tt meminit()}.

\refitem{}
Make {\tt vslock()} and {\tt vsunlock()} ignore out-of-range {\tt cmap}
structures, since virtual pages mapped to the Q-bus appear to
point to out-of-range physical pages.

\begin{verbatim}
sys/vm_page.c
\end{verbatim}
\refitem{}
Add support for the new fields in the cmap structure (see
{\filefont h/cmap.h}) to {\tt pagein()}, {\tt dpageout()}, {\tt checkpage()},
and {\tt klok()}.

\begin{verbatim}
sys/vaxuba/
\end{verbatim}
\refitem{}
Added some drivers: {\tt pri}, {\tt ranyan}, and {\tt shm}; and {\tt 
fiforeg.h} (the name is historical) which is used by them.  (John, can
you explain more?)

\section{Some more minor changes}

The changes described here are not directly RCI-related, 
though some of them were
necessary to get RCI working.

\begin{verbatim}
h/vmmac.h
\end{verbatim}
\refitem{}
Fixed {\tt MLOCK} and {\tt MUNLOCK} to not break if their argument happens
to have side-effects (eg, {\tt *cmapp++}). This bug was discovered
when working on the memory mapping/locking code.

\begin{verbatim}
vax/cpu.h
\end{verbatim}
\refitem{}
Added {\tt struct cpu620} to {\tt union cpusid} (for KA620). This was
preparatory to making UNIX run on a KA620, a project which has
been (temporarily, I hope) shelved.

\refitem{}
Added {\tt VAX\_620} (value is 16); raised {\tt VAX\_MAX} to match.
See also {\tt locore.s}.

\begin{verbatim}
vax/emulate.s
\end{verbatim}
\refitem{}
Rewrote everything but the {\tt editpc} and {\tt crc} emulation.  Shuffled
pieces of interface with {\tt locore.s} (q.v.).  In order to run 4.3 on
a MicroVAX, you must use a fixed {\tt emulate.s}; the one on the 4.3
tape (our 4.3 tape at least) is badly broken.  (It doesn't have
to be {\it this} {\tt emulate.s}, but if you use another one, make sure
it agrees with {\tt locore.s} on the interface between them---I moved
some pieces of this interface when doing this.)

\begin{verbatim}
vax/locore.s
\end{verbatim}
\refitem{}
Added {\tt VAX\_620} to case on cpu type in machine check handler (more
UNIX-on-620 support).

\refitem{}
Added {\tt soft1} through {\tt soft15} (except for 2, 8, 12, and maybe 15)
as stray interrupts.  (Not strictly necessary.)

\refitem{}
{\tt \_emuvec} table moved to {\tt emulate.s} (q.v.).
Emulate SCB vector handler moved to {\tt emulate.s} (q.v.).

\begin{verbatim}
vax/machdep.c
\end{verbatim}
\refitem{}
Added {\tt dumphangs} variable to make kernel {\tt dump()} routine hang for
investigation rather than dumping.  (For debugging.)

\refitem{}
Added support for {\tt dumphangs} in {\tt dump()} routine.

\begin{verbatim}
vax/pcb.h
\end{verbatim}
\refitem{}
Added {\tt \#ifndef LOCORE {\rm/} \#endif} around {\tt struct pcb}
definition, so {\tt locore.s} could include it for the {\tt \#define\/}s.

\begin{verbatim}
vax/inline/langpats.c
\end{verbatim}
\refitem{}
{\tt splimp()} changed to use IPL 17.  This is necesary on a MicroVAX,
because all device interrupts arrive at IPL 17 regardless of
the actual bus level requested.  Thus calling {\tt splimp()} can
actually {\it drop} IPL sometimes if it uses IPL 16.

\end{document}
