Mono LibRT - The Linux Mono Real Time Library
=============================================

Alex Tsariounov <alext@novell.com>
2008/11/25

Contents
--------
  Introduction
  Requirements
  Fair Disclosure
  Installation
  Mono.LibRt Class Wrapper Overview
  Using the Mono.LibRt Class Wrappers in Your Own Applications
  The Monocyclic Latency Measurement Tool


Introduction
------------
The mono-librt package includes two subpackages:

1) The Mono.LibRt and Mono.LibRt.HRT real time facilities class
   wrappers.  These classes wrap interfaces to real time policy and
   priority settings, task processor affinity settings, and the high
   resolution timer system.

2) The monocyclic latency measurement tool.  This tool is based on the
   C-language tool called cyclictest.  The tool measures wake up
   latencies in the Mono runtime and so allows latency problem
   diagnosis and demonstration.

The mono-rtlib project is under SVN and hosted on the Novell developer
site at:

  http://developer.novell.com/wiki/index.php/Rtmono

You can check out the latest code from SVN with this command:

  $ svn checkout https://forgesvn1.novell.com/svn/rtmono/trunk

Requirements
------------
Mono-librt is optimized for use with Mono 2.0 or greater, although it
may function with an earlier release.

Even though the class wrappers and monocyclic will function on a
non-realtime Linux (so long as high resolution timers are available),
to take full advantage of latency reduction, you will need to run a
real time Linux kernel.

One suitable RT Linux distribution is SLERT (SUSE Linux Enterprise
Real Time, http://www.novell.com/realtime).


Fair Disclosure 
--------------- 
If you run an RT Linux with this package, you will see a reduction in
latencies in your Mono applications; however, note that this reduction
will not be as complete as you may like it to be.  The reason for this
is that this package does not modify the Mono runtime in any way.
Specifically, a Real Time Garbage Collector (RTGC) is not included.

Since garbage collection can have an extremely drastic effect on
latencies seen by your applications, one cannot truly call this system
"real time", at least not yet.

In spite of this, you can get a nice reduction in typical latencies
seen by using an RT Linux kernel and these libraries.  In your quest
to reduce latencies, the monocyclic tool as well as the typical Mono
profiling tools will be most useful.  Pay particular attention to
memory usage patterns revealed with the profiler.  Minimization of
memory allocations will also minimize the non-realtime effects of
garbage collection.  See this URL for more information:

  http://www.mono-project.com/Performance_Tips


Installation
------------
The package as a whole is set up for autotools.  To create a local
build and install, use the following commands:

  $ configure
  $ make
  $ make check
  # make install

This configures the project for an installation prefix of /usr/local,
builds all libraries and the monocyclic tool, runs unit tests on the
libraries, and installs everything under /usr/local.  Note that you
usually need root permissions for the last installation step.

If you want to install to a different location, then specify a
different prefix for the configuration step, for example:

  $ configure --prefix=/usr


Mono.LibRt Class Wrapper Overview
---------------------------------
The missing RT API wrappers are organized into two assemblies: Mono.LibRt and
Mono.LibRt.HRT.  LibRt houses the CPU affinity and scheduling wrappers.
LibRtHRT houses the High Resolution Timers wrappers. All code in both
Mono.LibRt and Mono.LibRt.HRT also has NUnit unit tests that are part of the
project.

Real Time Scheduling Policy and Priority Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Real time scheduling policy and priority control is effected by the C-library
functions sched_getparam(1), sched_getscheduler(1) and sched_setscheduler(1).
These functions are called via P/Invoke mechanism in the Mono.LibRt assembly
and are then wrapped with the LibRt.Scheduler class. A helper class,
LibRt.SchedSpec is also created. LibRt.SchedSpec is used to specify scheduling
policy and priority. An enumeration, LibRt.SchedPolicy, contains the four
possible scheduling policies: SCHED_OTHER, SCHED_FIFO, SCHED_RR and
SCHED_BATCH. The LibRt.SchedSpec class makes sure that policies outside the
ones allowed or incorrect priorities are not specified.

Processor Affinity Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Processor affinity is controlled by the sched_setaffinity(1) and
sched_getaffinity(1) C-library functions.  These are called via P/Invoke in the
Mono.LibRt assembly and are wrapped with the LibRt.LinuxAffinity class. Basic
access to get and set either the calling thread's or any other thread on the
system is provided via the C# methods of LibRt.LinuxAffinity.GetAffinity() and
LibRt.LinuxAffinity.SetAffinity().

High Resolution Timers Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The High Resolution Timers facilities present in SLERT are wrapped in the
LibRt.HRT assembly with the LibRt.HRT.HrtClock and the LibRt.HRT.HrtTimer
classes. Methods exist to query the HRT resolution, to nanosleep for a number
of nanoseconds, to get and set the current time from the HRT system. HRT clocks
of CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_PROCESS_CPUTIME_ID and
CLOCK_THREAD_CPUTIME_ID are supported and defined in the LibRt.HRT.HtrClockType
enumeration. A helper class of LibRT.HRT.TimeSpec is defined to house time
specifications to nanosecond precision. This class allows mathematic
manipulation of times while keeping the results guaranteed valid. Both absolute
and relative time specifications are allowed. The LibRt.HRT.HrtTimer is modeled
after the System.Threading.Timer interface. It is currently implemented with a
nanosleep rather than the Unix signal that the HRT subsystem uses.


Using the Mono.LibRt Class Wrappers in Your Own Applications
------------------------------------------------------------
Since the mono-librt class libraries are considered "Libraries with
Unstable APIs" at this point, the assemblies are not installed into
the GAC.  Instead, in accordance with the Mono's Guidelines for
Application Deployment[0], the project includes pkg-config files to
permit simple access to the assemblies for your use.

There are two basic ways to use the libraries:

1) Copy the dlls to your project and re-distribute them.  This is done
   when you want to decouple the release schedules of mono-librt and
   your application.

2) Link to the dlls directly.  This is done if you do not care about
   release schedule decoupling, do not want to add the dlls to your
   distribution and simply want to link to the dlls on the system,
   whatever and wherever they are.

For case 1, add a target similar to the following to your project's
makefile:

  update-libraries:
       cp $(pkg-config --variable=Libraries mono-librt) .
       pkg-config --libs mono-librt > mono-librt.flags

  Then, in the target for your application, do the following:

  Demo.exe: Demo.cs mono-librt.flags
       mcs -out:$@ Demo.cs $(cat mono-librt.flags)

For case 2, simply use pkg-config to locate the libraries in your
application's makefile target:

  Demo.exe: Demo.cs
       mcs -out:$@ Demo.cs $(pkg-config --libs mono-librt)

The above assumes that the mono-librt pkg-config files are installed
in a place where pkg-config can find them.  If you are using a
packages rpm of mono-librt, then this will be the case.  If you built
and installed mono-librt by hand, you may need to make some
adjustments.

[0] http://www.mono-project.com/Guidelines:Application_Deployment#Libraries_with_Unstable_APIs


The Monocyclic Latency Measurement Tool
---------------------------------------
Monocyclic is the port of the C-language based command line tool
cyclictest to Mono. Cyclictest has become the standard method of
measuring latency on RT Linux, see this URL for more information:
http://rt.wiki.kernel.org/index.php/Cyclictest

Cyclictest, and hence monocyclic, measures latency, or jitter, by
spawning threads that sleep for a defined interval and measure the
time before and after the sleep with the High Resolution Timing
facility. The threads then calculate how far off the target wake up
time they actually do wake up. The ideal number for this is zero,
which means that there is no latency, no jitter. In reality, this time
varies and this time variation is what we call “jitter.”

Monocyclic gathers this timing data and outputs minimum, average and
maximum values. It can also collect a histogram of latencies that can
be converted into a graph.

If you just execute monocyclic without any options, a default set of
options is used that make sense for latency measurement. For example,
use this command line:
  # mono monocyclic.exe

Sample output looks like the following:

 [zuul:Debug]$ s mono monocyclic.exe
 NOTE: Using default arguments of: -t -a -n -r -m -p95
 0.21 0.89 2.12 1/366 27082
 T: 0 (27079) P:95 I:1000 C:  2228  Min:   3 Act:   6 Avg: 6 Max: 425
 T: 1 (27080) P:95 I:1000 C:  2228  Min:   3 Act:   5 Avg: 6 Max: 431
 T: 2 (27081) P:95 I:1000 C:  2228  Min:   3 Act:   6 Avg: 6 Max: 466
 T: 3 (27082) P:95 I:1000 C:  2228  Min:   3 Act:   4 Avg: 6 Max: 462

The output above shows latency minimum, average and maximum times in
microseconds for four threads on a 4-core machine. Monocyclic is using
default arguments of “-t -a -n -r -m -p95”. These mean the following:

  -t      Use as many threads as there are CPUs on system
  -a      Affinitize threads to separate CPUs
  -n      Use LibRt.HRT.Nanosleep() for sleeping time instead of 
          the cyclic timer
  -r      Use relative sleeping times
  -m      Lock process in memory
  -p95    Use a real time FIFO priority of 95

Monocyclic supports most of Cyclictest's options, with exception of
kernel tracing. For online help, type “monocyclic --help”,
the output of which is listed in the next figure.

----End.
