TimeKeeper Client

Use PTP, NTP, PPS, GPS - simultaneously

TimeKeeper can track time from just about any source. It can track PTP from a Grandmaster on the network, at the same time it tracks NTP from several different stratum servers, while also using a PPS and/or GPS input. This is useful to ensure proper delivery of timing data through the network - detecting network asymmetries, noisy clocks, faulty hardware, and more in addition to providing local clock accuracy in the nanosecond range.

Terminology

As mentioned above, TimeKeeper tracks ‘time sources’, which are defined in the configuration. We’ll cover a few common examples here to show how these sources work, with more specific detail as we go.

You’ll see the words ‘hardware timestamping’ throughout these documents. That refers to the ability of some network cards to more accurately timestamp packets as they’re received and sent. TimeKeeper will automatically make use of any of these features for a more accurate sync if they’re found. This allows for a more accurate and stable NTP and PTP sync, often in the low hundreds of nanoseconds.

NTP client

TimeKeeper supports nearly any form of NTP, and can achieve sub-microsecond accuracy with the protocol, which is a surprise to many. (NTP as a protocol is accurate, but many implementations of the protocol are not.)

As a client, TimeKeeper supports NTP versions 1, 2, 3, and 4, and can track NTP from any type of server, with accuracy primarily dependent on the quality of the NTP implementation on the server. (Generally, to achieve accuracy in the range of hundreds of nanoseconds, the NTP server providing time should be running TimeKeeper.)

A basic example of an NTP source, configured in TimeKeeper’s /etc/timekeeper.conf, is:

SOURCE0() { NTPSERVER=hostname; }

This will track NTP from ‘hostname’ as the primary source of time. There are a number of NTP options that can be enabled, which we’ll cover later - however, this is all that is needed to configure TimeKeeper for NTP.

PTP client

Similar to NTP, TimeKeeper will track IEEE 1588 PTP Grandmasters and boundary clocks from nearly any vendor, whether it’s a hardware appliance, a software server or a PTP-aware switch. TimeKeeper supports both PTP version 1 and version 2. Key features when using TimeKeeper as a PTP client include:

Support for the default, telecom, and hybrid PTP profiles
Ability to track many Grandmasters via many domains/interfaces simultaneously for controlled failover. This is in addition to the limited scope of the PTP BMC algorithm
Ability to match network speed properly to avoid silent biases caused by network asymmetries

Below is a straightforward example that will listen for PTP domain 0 (the default) using the default profile as source 0, the default time source. This builds on our NTP example by keeping our NTP source, but demoting it to a secondary priority (source 1) for failover:

SOURCE0() { PTPCLIENTVERSION=2; }
SOURCE1() { NTPSERVER=hostname; }

PTP is very reliant on multicast traffic, particularly in the default profile. In this mode, time is distributed from the Grandmaster via multicast, and every client that communicates with the Grandmaster uses multicast. As you might expect, this does not always scale well - with thousands of nodes, every client seeing every other client’s Grandmaster request consumes a lot of network and processor bandwidth.

A hybrid mode often makes more sense, where the Grandmaster distributes time via multicast to clients, and the clients that need data (like delay information) from the Grandmaster make unicast queries directly to the Grandmaster. Again, let’s build on our example, by adding a new source, tracking PTP domain 1 as the primary source, with our previous PTP source as the secondary source, and our NTP source as tertiary:

SOURCE0() { PTPCLIENTVERSION=2; UNICAST=1; PTPDOMAIN=1; }
SOURCE1() { PTPCLIENTVERSION=2; }
SOURCE2() { NTPSERVER=hostname; }

Since a hybrid PTP source is often used to avoid creating extra multicast traffic, normally if theres a hybrid source like source 0, a fully multicast source like source 1 would be removed. However, this is just is just an example so it’s left for comparison.

Full unicast (telecom profile) is also an option. In this case, the client subscribes to unicast updates from the Grandmaster and communicates with the Grandmaster entirely via unicast traffic. This can be ideal for longer links where multicast is not an option. Building on our previous example, now with our primary source being a fully unicast PTP feed available on PTP domain 2, with fallbacks to a hybrid PTP source, a fully multicast PTP source, and then NTP:

SOURCE0() { 
	PTPCLIENTVERSION=2; 
	PTPSERVER=gm_hostname; 
	PTPDOMAIN=2; 
}
SOURCE1() { PTPCLIENTVERSION=2; UNICAST=1; PTPDOMAIN=1; }
SOURCE2() { PTPCLIENTVERSION=2; }
SOURCE3() { NTPSERVER=hostname; }

PTP is a fairly complex protocol, and many other options are available to finely tune your deployment. We’ll cover other optional capabilities later in this document.

TimeKeeper communicates with just about any PTP Grandmaster, boundary clock, and client on the market. This includes but isn’t limited to TimeKeeper Grandmasters, Symmetricom/Microsemi products such as the S3XX line (like the S350), SSU2000 models, TPXXXX models (like the TP5000), Spectracom hardware like the SecureSync and VelaSync, and also hardware from EndRun, Oscilloquartz, Arista, Cisco, and so on. If you have any specific questions about your hardware, please contact us at support@fsmlabs.com.

PCIe cards

A PCIe card can also be used to deliver time to a TimeKeeper client. The specific method will vary by the card’s capabilities, but may be a direct GPS feed, an IRIG-B distribution, PPS, and so on.

Note: In addition to these examples, some TimeKeeper users get time from their custom in-house hardware.

TimeKeeper can generally use the card in any supported mode, allowing the card to be configured as needed. Local system time can be driven based on the card’s time and then redistributed, whether that’s via one of the card’s outputs or with TimeKeeper serving PTP and NTP on the network.

Out of the box support is provided for a number of cards from Spectracom, Symmetricom, and others.

In these cases, the card can be configured as needed to select timing inputs with the vendor supplied tools. Once configured, TimeKeeper will track the time provided by the card, along with any other configured sources.

Spectracom PCIe TSync GPS/IRIG Timecode Processor

A driver for this card is included with TimeKeeper and is loaded automatically when the Spectracom TSync card is selected as a source. You may also load and configure a driver for this device from the vendor. TimeKeeper does not change the reference table stored in the card so the reference table must be set up before starting TimeKeeper.

Adding a Spectracom TSync PCIe card to our existing example at the end of the PTP section would look like the following:

SOURCE0() { PPSDEV=spectracom; }
SOURCE1() { 
	PTPCLIENTVERSION=2; 
	PTPSERVER=gm_hostname; 
	PTPDOMAIN=2; 
}
SOURCE2() { PTPCLIENTVERSION=2; UNICAST=1; PTPDOMAIN=1; }
SOURCE3() { PTPCLIENTVERSION=2; }
SOURCE4() { NTPSERVER=hostname; }

In this case, the local card would be used first, with automatic failover to the PTP profiles and to NTP. Of course, even without failover, TimeKeeper will provide clear per-source timing details allowing you to cross check and alarm based on the behavior of all of your timing sources.

When using a PPS input the MAJORTIME option should be used to combine the PPS with time of day information from another source. More details on how to do this can be found in the section, “Major/minor sources.”

Symmetricom/Microsemi BC635PCIe/BC637PCIe IRIG, GPS and PPS cards

Similarly, a Symmetricom/Microsemi BC635/BC637 PCIe card could be used as a local source.

TimeKeeper can use these cards to receive a PPS, GPS or IRIG time references, depending on how the card has been configured. Only one card can be used at a time, and as with the Spectracom card, if only a PPS is delivered, the source should be combined with another source that provides time of day information, detailed in the “Major/minor sources” section.

Here, we’ve configured it to be the last source to be used, if all of the others fail, just as an example:

SOURCE0() { 
	PTPCLIENTVERSION=2; 
	PTPSERVER=gm_hostname; 
	PTPDOMAIN=2; 
}
SOURCE1() { PTPCLIENTVERSION=2; UNICAST=1; PTPDOMAIN=1; }
SOURCE2() { PTPCLIENTVERSION=2; }
SOURCE3() { NTPSERVER=hostname; }
SOURCE4() { PPSDEV=symmetricom; }
SYMMETRICOM_DRIVER_PATH=/opt/bc637_driver_release

In this configuration the drivers for the BC635/BC637 card must be installed separately from TimeKeeper and configured as desired with the driver’s tools. The driver’s location must be named using the SYMMETRICOM_DRIVER_PATH config option. This is the path to directory containing the subdirectories, dvrv/, samples/, etc.

EndRun GPS/Network/CDMA time reference device

TimeKeeper can use EndRun devices as a time source. The time source device should be configured to produce a NMEA ZDA string on the RS-232 port at 4800 baud. Using the FSMLabs provided cable, connect the BNC connector to the “1 PPS” output of the EndRun device and the RS-232 connector to the male DB9 port on the EndRun device labeled “Serial Time”. You will need a null modem adapter between the supplied cable and the “Serial Time” port.

Below is an example configuration using the reference input connected to the first serial device on the system, where the device is the last priority source in the system. You can also use any other serial port on the system as long as you update “PPSDEV” to reflect that device. You may not use a USB to RS232 converter for this connection since they introduce a great deal of delay and jitter on the connection which will result in poor accuracy.

SOURCE0() { 
	PTPCLIENTVERSION=2; 
	PTPSERVER=gm_hostname; 
	PTPDOMAIN=2; 
}
SOURCE1() { PTPCLIENTVERSION=2; UNICAST=1; PTPDOMAIN=1; }
SOURCE2() { PTPCLIENTVERSION=2; }
SOURCE3() { NTPSERVER=hostname; }
SOURCE4() { PPSDEV=/dev/ttyS0; }

GPS, PPS tracking

One of TimeKeeper’s strengths is that it can consume data from the network and a direct GPS or PPS input or output. A sole PPS input can be combined with another source for time of day information (like a NTP or PTP source). Benefits of GPS/PPS inputs include:

Highly accurate reference inputs for serving time or providing cross check
Low cost - sub-microsecond accuracy is possible with just a PPS via a serial input

A PPS can be easily pulled into a system, whether it’s for an accurate time source directly or as a cross check for other sources. In some cases it’s provided via a PCIe card, described above. It can also be delivered directly to an FSMLabs NetCard device or via a serial line input.

As with the other scenarios, informing TimeKeeper of a source is simple. If a PPS is brought in on the serial port, TimeKeeper can combine it with another source that delivers time of day information (major time) even if the source of major time is relatively noisy. The result will be a very accurate clock based on the delivery of the second boundary from the PPS.

SOURCE0() { 
	PPSDEV=/dev/ttyS0; 
	MAJORTIME=SOURCE3;
}
SOURCE1() { 
	PTPCLIENTVERSION=2; 
	PTPSERVER=gm_hostname; 
	PTPDOMAIN=2; 
}
SOURCE2() { PTPCLIENTVERSION=2; }
SOURCE3() { NTPSERVER=hostname; }

OCP-TAP Timecard

TimeKeeper can use the OCP-TAP Timecard device http://www.timingcard.com as a time source.

One can access the Timecard via the Linux kernel system file system or sysfs. The exposed parameters can be viewed by:

ls /sys/class/timecard/ocp0/

Where /sys/class/timecard/ocp0/ is the instance of the exposed Timecard device. If a second Timecard is installed, it would be available as /sys/class/timecard/ocp1/ etc.

To configure the Timecard, add the source using the PPSDEV option. It should use the base path of the Timecard directory. Below is an example configuration.

SOURCE0 () { PPSDEV='/sys/class/timecard/ocp0'; }
SOURCE1() {
	PTPCLIENTVERSION=2;
	PTPSERVER=gm_hostname;
	PTPDOMAIN=2;
}
SOURCE2() { NTPSERVER=hostname; }

In this example, additional sources identify PTP and NTP servers for comparison, failover, etc.

Hyper-V support

On a virtualized host time may be available from the hypervisor directly. This is the case in some Hyper-V environments like cloud systems hosted in Azure.

To configure for Hyper-V, a source should be added using the PPSDEV option. Then just like in the other examples, additional sources can identify PTP and NTP servers for comparison/reporting/etc.

SOURCE0() { PPSDEV=hyperv; }
SOURCE1() {
	PTPCLIENTVERSION=2;
	PTPSERVER=gm_hostname;
	PTPDOMAIN=2;
}
SOURCE2() { NTPSERVER=hostname; }

Other sources can also be configured, as in the other examples. Here the primary source is the Hyper-V time, with failover/cross check/compliance reporting sources over both PTP and NTP.

This option is supported on Linux systems hosted in a Hyper-V environment.

Failover and Sourcecheck

In all of the above scenarios, TimeKeeper uses sources in the order configured. Meaning, the system clock will be driven to match source 0 (or the lowest numbered source) as long as that source is delivering timing data. If source 0 stops providing data, TimeKeeper will fail over to source 1, 2, and so on.

If there’s a source failure and TimeKeeper begins using source 1, it will continue to use source 1 until either source 1 fails or source 0 begins responding again. If source 0 returns TimeKeeper will begin tracking it again. All of this occurs regardless of what protocol or source is in use - NTP fails over to PTP, PTP can fail over to PPS, and so on.

TimeKeeper can also actively compare time sources and proactively reject bad ones using the Sourcecheck feature. This feature applies to TimeKeeper clients, servers, and Grandmasters. For details on enabling and using Sourcecheck, please refer to the “Sourcecheck” section.