Tips

Collected here are some short notes about various aspects of testing, drivers, and general practices. If you have any specific questions about any of these items, please contact support@fsmlabs.com.

Assumptions about time in the past being invalid

When receiving time from any time source TimeKeeper will validate that time to make sure that it can be trusted. One of these checks includes making sure the time is not earlier than the release date of the TimeKeeper version being used. This can catch problems with time sources that are reporting erroneous time but it also means that it is not possible to run simulations with time in the past. If you need to be able to use simulated time that is in the past please contact support@fsmlabs.com and we can explain how to configure TimeKeeper to allow that.

Holdover

Some time devices are capable of keeping good time while not actively receiving time updates. This includes the Spectracom TSync GPS/PPS card, Symmetricom BC637 GPS/PPS card and the TimeKeeper GPS/Oscillator. By default TimeKeeper will remain in holdover for 2 hours (7200 seconds) and continue to use the time source even when it reports no GPS/PPS signal. That time limit can be changed with the HOLDOVER_LIMIT=X setting where X is number of seconds. Once that time limit expires TimeKeeper will begin comparing all time sources to determine if the holdover time is out of range with other time sources by using the Sourcecheck algorithm (even if Sourcecheck is disabled). To facilitate a possible failover, you must configure TimeKeeper to have at least three sources so that Sourcecheck can find a quorum and therefore a source to failover to.

Clock adjustment and steering

TimeKeeper will smoothly adjust the clock to correct offsets in most cases. However, when the offset is greater than 5 seconds TimeKeeper will reset the time to avoid the delay necessary to slew the clock. Once the clock is set in this way it will continue tracking and slewing as necessary to keep the time synchronized. In practice the only time this happens is when TimeKeeper is first started and it begins synchronizing the clock with a remote time source.

Once TimeKeeper is active, setting the system time is not recommended. TimeKeeper will stop other timing daemons but it is important that other utilities do not also try to steer the clock. That includes the command line tool “date” and other system tools that set the time and time synchronization software such as ntpd that use system calls such as “settimeofday” and “adjtime”.

Windows

The Windows Time service, W32Time, includes an NTP daemon. It can cause problems in a couple of ways. First, it could be trying to steer the system clock in conflict with TimeKeeper. Second, even if it isn’t steering the clock, it could be bound to the same port that TimeKeeper uses. The following instructions are just a starting point for dealing with these issues. You should research Windows Time service options to determine what best suites your needs.

You can stop W32Time and prevent it from automatically starting on boot with these two commands:

sc stop w32time
w32tm /unregister

However, another process, e.g., the Windows Domain Controller, could start it up again, so it’s up to you to assure that this can’t happen. If this becomes a problem, e.g., you require Windows Domain Controller, you can leave W32Time running but prevent it from steering the clock with a registry setting (set Type to NoSync). There are three ways to do this.

  1. You could use this command:

    reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters /t REG_SZ /v Type /d NoSync /f

  2. use regedit to make the same changes,

  3. or use this command:

    w32tm /config /syncfromflags:NO /update

After making any changes to the W32Time configuration, you should restart it with this command:

net stop w32time && net start w32time

If you have TimeKeeper configured to use NTP, there could still be a problem. W32Time could be bound to the NTP port even though it’s not using it. In that case, TimeKeeper cannot use NTP sources. Configure TimeKeeper to use something else instead, such as PTP.

Identifying inaccurate sources

Accuracy issues may manifest as a noisy primary clock - usually as noise in the smoothed offset plot on the web interface, or in the second column of data in timekeeper_0.data.

Comparing the noisy data file to a less noisy one can help identify where the noise is coming from. If your primary source is SOURCE0, and it appears to be smoothly handled in *timekeeper_0.data", and SOURCE1 is noisy in the web interface and *timekeeper_1.data", you can be fairly certain that either the network or the time server itself in SOURCE1 is noisy.

This is worth noting because if the noisy source is your primary and TimeKeeper is chasing that clock, the reported offsets for your secondary and other sources may appear to be noisy also. Put another way, if your SOURCE0 is noisy but SOURCE1 is stable, chasing the noise in SOURCE0 can appear in some situations to also be noise in SOURCE1.

Where it’s an option, configuring your primary source to be a clean and stable PPS can quickly indicate which sources are actually noisy. When TimeKeeper is not chasing a noisy clock and is instead tracking a stable PPS, much of the noise will be removed, allowing for more accurate measurements.

Sync rate

The default sync rate of 1 and 0.9 packets/second for PTP and NTP, respectively, is a good tradeoff between network traffic, CPU load, and synchronization accuracy. A higher sync rate won’t necessarily increase accuracy. If you’re considering it, experiment first in your environment to determine if it actually results in better accuracy than with the default. Also note that higher sync rates increase storage requirements for Compliance reporting. Please refer to the Compliance “Disk space requirements” section.

Unresponsive NTP server

When acting as a NTP client TimeKeeper will wait for many milliseconds for a response. If there is no response from the server in that time the server is assumed to not be responding and no time synchronization will occur during that query of the server. This could be a problem for synchronization of a very slow WAN connection. If your environment and network have a very long round-trip between the client and server please contact support@fsmlabs.com for configuration information.

Hardware timestamping network cards

Hardware timestamp support allows TimeKeeper to most accurately measure receipt and transmission of timing data. This is supported automatically if:

Hardware timestamping is supported on these cards (but not limited to them):

In the case of Red Hat Enterprise Linux 6, upgrade the provided driver with the most recent IGB version available from Intel since the driver shipped with the distribution does not support all cards. Version 2.4.8 of the IGB driver or above is recommended. Some version of these drivers do need to be loaded, configured with an IP address to initialize them (with ifconfig), unloaded and then loaded once again for proper behavior when used for hardware timestamp assist.

Hardware clock

TimeKeeper also updates the hardware clock every 10 minutes on Linux to keep it in sync with the current system time. This ensures there’s a good reference for any components that depend on the hardware being in sync with the system clock.

Identifying accuracy problems

Poor NTP implementations

Many NTP implementations are of very low quality and are very noisy, even if the network delivering the NTP data is fast and deterministic. In these cases it’s recommended that TimeKeeper be configured to minimize the effect that these bad NTP servers can have on your network. This can be accomplished in several ways:

Firewalls

Although uncommon, occasionally firewalls are used between timing clients and servers. While the clients can still sync through the firewall, generally this reduces the potential accuracy of the client greatly.

Timestamping accuracy

TimeKeeper will accurately serve or consume time on any supported host, but there are some configurations that provide more accurate results.

TimeKeeper will enable the highest resolution timestamping it can, and report which features were enabled in /var/log/timekeeper at startup. On windows, this will be logged to %ProgramFiles%\timekeeper\var\log\timekeeper.log. If the host distribution supports the feature, the driver can enable hardware timestamping, and the device supports the feature, timestamping will be done by the card. If the card or the driver can’t support the feature, software timestamps will be requested from the network stack, if the host can provide them.

Hardware timestamps are the most accurate, followed by software timestamps. TimeKeeper uses this data to factor out propagation delays in incoming time, and also to provide more accurate data when delivering time.

It is important to understand TimeKeeper does not need special support for your particular hardware - it will find and use the most accurate timestamping available, automatically, to the best ability of your host distribution, driver, and hardware.

If you have any questions about how to make sure you’re getting the best performance in your environment, please ask support@fsmlabs.com.

Bonded Ethernet

Bonding is supported, including the ability to use hardware timestamps with the devices in the bond. However, care should be taken to make sure that the underlying devices are of the same device type so that there’s a consistent feature set and performance across the members of the bond.

Best Practices

Here is a brief list of TimeKeeper best practices. These may not apply to all deployments, but they apply to many that depend on TimeKeeper for accurate and resilient timing services.