Capabilities

OpenPGM C programs require system capabilities from this list in order to function.

* Clock selection is specified by two system environment variables PGM_TIMER governing how to get the current time, and PGM_SLEEP^† governing how to wait for an interval of time.

Clock Selection

Kernel timer resolution is in a period of change, kernels can be found with standard resolutions of 1-4ms, with high resolution timers providing microsecond to nanosecond resolution. RDTSC is a very cheap mechanism for applications to get the current time in high resolution, however multiple core systems may suffer from drift between each core, some systems may vary frequency in power saving states. Particularly prone to error are Hyperthread based processors from Intel, it is highly recommended to disable hyperthreading on all processors wishing to run high speed PGM based messaging. The High Precision Event Timer, HPET, was created to provide a system wide stable timer in multi-core systems to resolve TSC irregularities.

An operating system calculates the current time of day based off various sources to provide an accurate, stable, and monotonic source to applications. On Linux the system time is generated from the APIC PM Timer or HPET with interpolation using TSC which can cause the time to become non-monotonic. Calculating a stable system time is not cheap and historically many applications have been re-engineered to minimise the calls to functions like gettimeofday() due to this overhead, using RDTSC has been a popular method to bypass this overhead, with multi-core systems either process affinity has to be used to keep using the same TSC source, or reading from the HPET device instead but at a ~500 nanosecond cost.

The RTC device /dev/rtc cannot be shared between multiple applications but offers a stable time source external to the TSC and is available on all systems, compared with limited availability of HPET.

On Linux TSC is calibrated using the details provided by the kernel in /proc/cpuinfo which is ratio calculated at boot time. By default TSC will be calibrated by the method defined by PGM_SLEEP, and either value can be overridden by defining RDTSC_FREQUENCY in kilohertz. It is recommended to set this frequency or use an alternative timing mechanism on Solaris.

* Default settings. † Removed in OpenPGM version 3.0.

Examples

Use the PGM protocol via raw sockets using the CAP_NET_RAW capability,

$ sudo execcap 'cap_net_raw=ep' pgmsend moo

On a single core Intel system, use TSC and usleep.

$ PGM_TIMER=TSC PGM_SLEEP=USLEEP pgmsend moo

or,

$ pgmsend moo

On a multi-core Intel system with HPET device, use HPET and usleep.

$ sudo chmod a+r /dev/hpet
$ PGM_TIMER=HPET PGM_SLEEP=USLEEP pgmsend moo

or,

$ sudo chmod a+r /dev/hpet
$ PGM_TIMER=HPET pgmsend moo

On a hyper-threaded Intel system without HPET device, use the RTC device and usleep.

$ PGM_TIMER=RTC PGM_SLEEP=USLEEP pgmsend moo

or,

$ PGM_TIMER=RTC pgmsend moo

On a multi-core system without HPET and for multiple applications, use gettimeofday and usleep.

$ PGM_TIMER=GTOD PGM_SLEEP=USLEEP pgmsend moo

or,

$ PGM_TIMER=GTOD pgmsend moo

On a Windows system, use the system performance counter and Sleep.

C:\pgm\bin> set PGM_TIMER=TSC
C:\pgm\bin> set PGM_SLEEP=MSLEEP
C:\pgm\bin> pgmsend moo

or,

C:\pgm\bin> pgmsend moo

On a Solaris system with a 3.2Ghz core, set the TSC frequency first.

$ RDTSC_FREQUENCY=3200 PGM_TIMER=TSC PGM_SLEEP=USLEEP pgmsend moo

or,

$ RDTSC_FREQUENCY=3200 pgmsend moo