Hardware timestamps

Network cards with hardware timestamp capability have clocks on them that timestamp a packet when it arrives. This can greatly improve the accuracy of timestamps because there are no software delays in those timestamps. For those timestamps to be useful the clock on the network card must have good time. TimeKeeper manages the clocks on these devices and steer them to the correct time without any user interaction or configuration.

TimeKeeper transparently detects and take advantages of any network card hardware timestamp features when the operating system, drivers and hardware supports them. Hardware timestamps have been supported with TimeKeeper on Linux for many years, and is now supported on Windows too, starting with Windows 11 and Windows Server 2022. TimeKeeper does not support hardware timestamp features on Solaris.

Enabling hardware timestamps

When running on a platform that supports hardware timestamps there is no additional work needed to configure TimeKeeper to use them. It will enable and verify these features by default if they’re enabled in the operating system. You can verify that once TimeKeeper is running with the tkstatus command. The output should look something like the below when hardware timestamps are working. The TIMESTAMPING_HW indicates that hardware timestamps are used. Column 9 in the data files will show the same value.

If the network card doesn’t provide hardware timestamps (or hardware timestamp data isn’t usable for some reason), the driver/stack can provide a software timestamp which is indicated by ‘TIMESTAMPING_SW’. If the driver doesn’t provide either, TimeKeeper falls back to its own internal timestamps, indicated by ‘INTERNAL’. Internal timestamps are similar to software timestamps but can be a little noisier because they’re taken further up the stack and are subject to more scheduling noise.

TimeKeeper Status
================================================================================
TimeKeeper version X.Y.Z
Primary source is: 0
Primary source offset: 0.000000072 seconds
Primary source type: PTP(10.0.0.12:0,unicast,telcom)
Last time update: 0 seconds ago

Src|offset      |source type                     |update age(s)|timestamp type
================================================================================
  0  0.000000072  PTP(10.0.0.12:0,unicast,telcom)             0 TIMESTAMPING_HW

The file timekeeper_stamps.log in the configured LOGDIR directory will show a per-source and per-server summary of hardware timestamps. This will show a snapshot of the last 30 seconds of data as well as historical data since the last time TimeKeeper was restarted.

Enabling hardware and software timestamps (Windows-specific)

On Windows the same steps apply as above regarding log files and formats, but hardware and software timestamping may need to be enabled. The below steps in PowerShell (as Administrator) will enable software timestamping for a device named Eth4. For completeness this includes steps required to install components needed for HW/SW timestamping. This is provided as an example only, please refer to the Microsoft documentation for details.

PS C:\> Set-ExecutionPolicy remotesigned # Enable execution of below steps in some environments
PS C:\> Install-Module SoftwareTimeStamping
PS C:\> Import-Module SoftwareTimestamping
PS C:\> Get-DscResource -Module SoftwareTimeStamping
PS C:\> Enable-SWTimestamping -NetAdapterName Eth4
PS C:\> Restart-NetAdapter -Name Eth4

You can disable software timestamping on a device with:

PS C:\> Disable-SwTimestamping -NetAdapterName Eth4

Similarly, hardware timestamping may need to be enabled explicitly with the following commands to display, enable, or disable the feature:

PS C:\> Get-NetAdapterAdvancedProperty -Name Eth4 -Displayname 'PTP Hardware Timestamp'
PS C:\> Set-NetAdapterAdvancedProperty -Name Eth4 -Displayname 'PTP Hardware Timestamp' -DisplayValue 'Enabled'
PS C:\> Set-NetAdapterAdvancedProperty -Name Eth4 -Displayname 'PTP Hardware Timestamp' -DisplayValue 'Disabled'

On Windows either hardware or software timestamping must be enabled on an interface, but not both. So if hardware timestamping is intended make sure to disable software timestamping on the interface, and vice versa if software timestamping is preferred. Having both enabled will not prevent TimeKeeper from using any configured time sources, but will cause INTERNAL timestamps to be used rather than software or hardware.

Network card clock logging

Every update to the network clocks is logged just like updates to the system clock. Instead of the log files being timekeeper_$SOURCE.data where $SOURCE is 0 through 49, 100 through 149 are used. This allows for up to 50 network card clocks to be controlled. The log file format is the same as for any other time source and is described in the “File formats” section.

The accuracy listed is when comparing the network card clock against the system clock. The one-way trip times are when reading the network clock (typically across a PCIe bus). The exception to this is noted below on Linux with the steering nic optimization.

On Windows, the network clocks are managed differently and currently not logged in the .data files as they are on Linux.

Steering NIC optimization (Linux)

Normally TimeKeeper receives a time update and uses that to adjust the system clock. It then will adjust all the network card clocks by steering them to match the system clock. In some cases TimeKeeper may use an optimization.

The optimization is to steer a given network card clock directly to the incoming NTP or PTP feed rather than to the system clock. This can improve the accuracy somewhat since it avoids one extra step (through the system clock). This will only be done when:

When the optimization is being used you will see a different message in the data file associated with that network card clock. An additional value will appear in column 12. A ‘sourceN’ value will appear in which indicates this clock is being directly steered to the Nth time source. If that was source0 then the information may show up as:


1635174614.904756863 -0.000000173 -0.000000021 -0.000000152 0.000051843 0.000000000 0.000000000 -0.000000173 N/A 0.000001102 0.000019956 PHC(002590d096d2,freqcorr:-0.000051848,source0) 0.000051800 0.000000425 AUX 0.000051680 0.000000000 0.000000188 0.000000166

When a network clock is in this mode the logs will not show the accuracy, round-trip time and similar in reference to comparing the clock against the system clock but rather the values when comparing the clock against the remote time source. So accuracy is against the remote source and the one-way trip times are over the network to the remote time source.

This optimization can be disabled with the following option in the timekeeper.conf file:

ENABLE_STEER_NICTOSOURCE=0

Hardware timestamp problems

TimeKeeper does significant cross-checks and validation of hardware timestamps as there can be device and/or driver issues that can otherwise cause significant accuracy issues. These errors will be logged in the main timekeeper log file /var/log/timekeeper on Linux and C:\Program Files\timekeeper\var\log\timekeeper.log on Windows. The first indication that there is a hardware timestamp problem is that software or internal timestamps are being used when hardware is expected.

One common error is that a network card/driver provides hardware timestamps for transmit but not receive, or vice-versa. The main TimeKeeper log file will show something similar to:

1512507850.617854123: Source 0: timestamp mismatch occurred: sync RX: TIMESTAMPING_HW, delay request TX: TIMESTAMPING_SW, able to correct mismatch
1512507859.276330604: Source 0: timestamp mismatch occurred: sync RX: TIMESTAMPING_HW, delay request TX: TIMESTAMPING_SW, able to correct mismatch
1512507859.277407383: Source 0: timestamp mismatch occurred: sync RX: TIMESTAMPING_SW, delay request TX: TIMESTAMPING_HW, able to correct mismatch
1512507864.693405652: Source 0: timestamp mismatch occurred: sync RX: TIMESTAMPING_HW, delay request TX: TIMESTAMPING_SW, able to correct mismatch

The above error is due to a driver/card problem but is being corrected by TimeKeeper. TimeKeeper will fall back to software timestamps if available to avoid asymmetry errors introduced by having hardware timestamps one direction but software the other direction.

Disable hardware timestamps

For testing and avoiding network card/driver bugs it is sometimes necessary to disable hardware timestamps. One can disable hardware timestamps on all network cards with ENABLE_HWTSTAMPS=0 in the TimeKeeper configuration file. This will configure each network card seen to disable (in hardware) the hardware timestamps. This is useful when some older drivers (with bugs) cause kernel issues when hardware timestamps are enabled.

It’s also possible to leave the hardware timestamps on in the hardware but on a source-by-source basis simply ignore those hardware timestamps. This is helpful for testing sometimes. This can be done with ENABLE_HWTSTAMPS=0 in each source block to ignore hardware timestamps for that source.

There are yet other cases where it’s necessary to disable hardware timestamps on the card itself for specific cards but not others. The above configuration variables won’t allow that but AVOID_IFACES=… will. For example, if you have one of the network cards that advertises hardware timestamp support but contains a bug that causes serious networking problems or reboots you may want to use this option. In those cases you can enter those network cards in the AVOID_IFACES= value (comma separated list) and TimeKeeper will ignore these devices. It could route network traffic through them but will not attempt to configure them, will not track the time on them and will not try to steer the time on them. This is one possible work-around for buggy drivers.

TimeKeeper also maintains an internal set of known unstable drivers and avoids versions known to be unstable by default. This is automatically done but can be overridden with the ENABLE_NIC_BLACKLIST configuration option. More details on this and AVOID_IFACES is available in the configuration section, “Global options.”