\documentstyle[12pt,twoside,proof,sf,sltt,cvarl]{rccldoc}

\pagestyle{myheadings}

\begin{document}

\title{
Modification of the SUN O/S Kernel to Provide Real-time
support for RCCL/RCI}
\repnumber{}

\author{
John Lloyd}

\date{March 25, 1990}

\maketitle

\chapter{Introduction}

This memo is intended to summarize the kernel modifications 
that were performed on SUN O/S 3.2 (by Mike Parker and John Lloyd)
to enable that system to run RCI. RCI (Real-time Control Interface)
is a software package which allows the creation
of real-time control tasks attached to a UNIX process.

The main problem with running a real-time task under UNIX is
one of task priority: it is generally not possible to schedule
execution of any particular piece of code with a time accuracy less than a
large fraction of a second. 

We get around this problem by offering a ``connect to interrupt''
facility, which allows application control code to be executed
by the interrupt routine of a specially prepared device driver.
The execution occurs in kernel mode, with all interrupts masked
out. This provides the necessary performance characteristics, but
creates the following problems:

\begin{itemize}
\begin{enumerate}
\item {\it Memory Context}.
When an interrupt is received by the system, the current
memory context is not necessarily that of the RCI process which
contains the control code. It is therefore necessary to 
switch to the context of the RCI process.

\item {\it Locked Memory}.
It is necessary that the memory containing the control
code is not swapped out, either as a result of paging, or
of the entire RCI process being swapped out. It is
therefore necessary to lock down the memory containing the
control code and data.

\item {\it Exception Handling}.
When an exception (such as a divide by zero or a bad
address fault) occurs inside the control code, it is
necessary to catch this exception ourselves and handle
it in some reasonable way (such as by sending an appropriate
signal to the RCI task). Otherwise, when an exception
occurs within kernel mode, UNIX responds by bringing down the
system (because errors are not supposed to occur within the
kernel). 
\end{enumerate}
\end{itemize}

The handling of each of these issues on the SUN4, under O/S 3.2,
is now discussed in some detail.

\chapter{Implementation}

\section{Memory context}
On SUN systems, the memory management unit (MMU) maintains a small
number of contexts (usually 8 or 16, depending on the system).
Each context defines a complete address space for a particular
process. Switching from one context to another is done simply
by setting the context register in the MMU. Since the
number of processes on the system usually exceeds the number
of MMU contexts, the system will juggle contexts around,
trying to keep them assigned to the most active processes.
Setting up or changing a context within the MMU is considerably
more expensive than just switching context numbers, in particular
because making changes within a context often requires flushing
entries in the memory cache.

For RCI, some support was added to the kernel which effectively
``locks'' the context number of an RCI process, so that at interrupt
time it is only necessary to set the context register to the
appropriate number and the RCI process address space instantly appears.

The context locking was implemented by adding a 'flags' field
and a lock bit to the memory context structure,
and inserting code which uses this in the routines {\tt ctxalloc()} and {\tt
ctxfree()}, in the file {\filefont sun4/vm\_machdep.c}.

\section{Locked memory}
It is also necessary to make sure that the system does not
try to swap out the RCI process, or individually pageout any
of its pages.

The former is handled by setting the flag {\tt SULOCK} in the
system process structure. The latter is handled by adding some
code to the paging algorithm, telling it not to remove any
pages from a process which has the flag {\tt SULOCK} set.

It was also necessary to modify the routine {\tt ptexpand()},
in {\filefont os/vm\_pt.c}, which is responsible for expanding
an existing memory context. This routine normally frees the
process's memory context, and will also swap it out if the
resources needed for expanding are not available. This was changed so that
the context for an RCI process is not freed, the changes to the
context itself are done non-interruptably (so that the context
will not be in an indeterminent state when an interrupt comes
along), and the process is simply put to sleep if the 
resources are not available.

\section{Exception handling}

Exceptions generally result in a call to the kernel routine
{\tt trap()}. This routine was modified to check a flag that
is set whenever RCI control level code is running. When an
exception occurs and this flag is set, 
an appropriate signal is sent to the RCI process,
rather than the usual procedure of bringing
down the system.

The code for {\tt trap()} lives in the file {\filefont sun4/trap.c}.

\section{Miscelaneous items}

A small number of other changes were made. These were:

\begin{itemize}
\begin{enumerate}
\item
The file close code in {\tt exit()} ({\filefont os/kern\_exit.c})
was moved to a point before the call to {\tt relvm()}. 
This ensures that the close code in the RCI device drivers
will be able to shutdown the RCI control level, and unlock
the memory context, before the system explicitly tries to free the
memory context be calling {\tt relvm()}.

\item
The size of the interrupt stack (defined in {\filefont sun4/locore.s})
was grown to accomodate the execution of RCI control code.

\item
The various {\tt spl} 
routines which set the system interrupt level (defined
in the file {\filefont sun4/subr.s}) were modified so that the level is
never dropped while we are executing RCI control code.

The main reason this was necessary was because the control
level will sometimes make use of the kernel {\tt printf()}
routine (for printing diagnostic messages and such).
The kernel {\tt printf()} in turn calls other routines 
which try to lower the system interrupt level using
the {\tt spl} routines.

\item
Similarly, a change was introduced into {\filefont sun4/float.s} so that
the floating point exception code does not drop the interrupt
level while RCI control code is running.

\item
In order to provide RCI with its own clock, a private RCI
routine was added to the {\tt hardclock()} routine in
{\filefont os/kern\_clock.s}.
\end{enumerate}
\end{itemize}

\chapter{Summary of Changed Files}

The source files in SUN O/S 3.2 that were modified are listed below:

\section{Critical changes}

\begin{description}

\item[\filefont os/kern\_clock.c:]
Added a hook in the {\tt hardclock()} routine so that RCI
can have its own clock function.

\item[\filefont os/kern\_exit.c:]
Moved the file close code before {\tt relvm()}.

\item[\filefont os/vm\_page.c:]
Prevent paging if a process has {\tt SULOCK} set (its seems
that we might also have been able to simply set {\tt SFAVORD} instead).
	
\item[\filefont os/vm\_pt.c::]
Modify {\tt ptexpand()} so that page tables for an RCI process are expended
without releasing the context or swapping the process out.

\item[\filefont sun4/float.s:]
Prevent dropping of interrupt level in floating exception code.

\item[\filefont sun4/locore.s:]
Grow the interrupt stack size. Make the beginning and
end of the stack global variables so we can locate the stack. 
On SUN4-110 systems, patch the
VME interrupt code to the required vector, since we couldn't get the
vectoring to work otherwise.

\item[\filefont sun4/subr.s:]
Put hooks into the following routines:
{\tt splzs()}, {\tt spl0-7()}, {\tt splsoftclock()}, {\tt splx()}, 
{\tt splr()}, so that interrupt level is not dropped during execution
of RCI control code.

\item[\filefont sun4/trap.c:]
Added special code to properly handle exceptions from RCI control
level.

\item[\filefont sun4/vm\_machdep.c:]
In {\tt ctxalloc()}, do not allocate a locked context. Panic if
we run out of contexts. Explicitly clear the lock flag when a context
is freed in {\tt ctxfree()}.
Mike also appears to have tightened up the context allocation algorithm.

\end{description}

\section{Changes in files normally provided}

\begin{description}

\item[\filefont os/init\_sysent.c:]
Added a new system call, {\tt cmkrnl()}, which allows a function to
be executed in kernel mode.

\item[\filefont sys/ioctl.h:]
Enlarged the maximum size of the argument buffer used by {\tt ioctl()}.

\item[\filefont sys/proc.h:]
Added flag marking a process as an RCI process.

\item[\filefont sun4/conf/files:]
Declared new source files for the system.

\item[\filefont sun/conf.c:]
Added drivers for RCI devices.

\item[\filefont sun4/mmu.h:]
Added a flags field to the context structure ({\tt struct context}),
and defined the flag {\tt CTX\_LOCK} which indicates that the 
context is locked.

\end{description}

\section{New support code}

The RCI drivers and support code live in the new directory
{\tt /local}.

\chapter{Upgrading the Changes to SUN O/S 4.0}

It should be possible to upgrade the changes described here to SUN O/S
4.0 or higher. For SUN O/S 4.0, it would be desirable to have access
to the following source code directories:

\begin{verbatim}
		os/
		sun4/
		sun/
		sys/
		vm/
\end{verbatim}

We need access to much of the source because the virtual memory code
was rewritten on 4.0, and we don't know exactly what we would have to touch
to effect our changes.

There might be an easier solution. 
For instance, if Sun supports the {\tt pagelock()} software special on
O/S 4.0, it may be possible to make do with that extra feature, plus
access to only the following source files:

\begin{verbatim}
		os/kern_clock.c
		os/kern_exit.c
		sun4/float.s
		sun4/locore.s
		sun4/subr.s
		sun4/trap.c
		sun4/trap.h
\end{verbatim}

\chapter{``Doing It All'' with a Special Driver}

It also looks like it should be possible
to write a device driver which provides RCI services
without requiring any modification to SUN O/S. Doing this relies
on several things:

\begin{itemize}
\begin{enumerate}
\item We can compile programs (using the the {\tt -PIC} option available
on SUN O/S 4.0) for which the code is position independent. It would
then be possible to copy the control level portion of the RCI program
into the kernel and execute it there. This eliminates all the problems
associated with swapping, paging, and locking memory contexts.
It appears that this will work, but a full test is still pending.
For instance, it is unclear whether the libraries in {\tt libc} have been
compiled  
with the {\tt -PIC} option, and the RCI control code does use some 
of the routines stored there.

\item We can replace the system trap table with our own (when running
the RCI control code). Exceptions which occur inside the RCI control
level would then be handled by our own code, with no need to modify
the system exception code in {\filefont sun4/trap.c}. The trap table would
be restored to the system default value when the control function was
finished. This appears to be feasible, based on our experience
with a test driver that does exactly this.

\item We obtain an external hardware clock and interface it to the
SUN system. This provides RCI with an independent time base, and removes
the need to put a hook into the {\tt hardclock()} routine
in {\filefont os/kern\_clock.c}. This is definitely
feasible.

\end{enumerate}
\end{itemize}

We still have to contend with the following issues:

\begin{itemize}
\begin{enumerate}
\item The interrupt stack is very small, and it will be necessary to
switch to a larger stack when processing the RCI control level code.

\item The kernel {\tt printf()} routine will still drop interrupt level when
used from the control level. The {\tt spl()} routines would have to
be patched to get around this problem.
\end{enumerate}
\end{itemize}

\end{document}
