TimeKeeper Grandmaster

This section of the Grandmaster documentation will just focus on managing the device - from initial setup to applying updates. For data on individual features, please refer to the section on that feature.

TimeKeeper Grandmaster setup

Connect a GPS antenna to the female SMA GPS (Grandmaster generation 2) or BNC (Grandmaster generation 1) connector on the back of the device (see photos below). The GPS receiver provides 5v DC to the antenna.

Connect a network cable to the first RJ45 ethernet port (eth0 in photos) and navigate with a web browser to 192.168.1.1. You can log in with account “admin” and password “timekeeper” (Note: In earlier versions of TimeKeeper the default password was “fsmlabs”) to configure the network ports, TimeKeeper mode and options. Change the admin user’s default password immediately for security, please. If this is an upgrade and you have been using previous versions of the Grandmaster GUI, it is recommended that you force a full page reload (Ctrl-Shift-R on many browsers). This will make sure your browser loads the latest web interface from the Grandmaster and provides all of the latest features.

To change network interface addresses navigate to the Configuration tab and then Networking subtab, select the options you would like then click Save network changes and then Restart Network to apply the updates. The changes will take a moment to apply after which you can reconnect to the newly configured address.

By default (and as shipped) the Grandmaster does not enable SSH access. That can be enabled through the web interface, however. Please see below for details on enabling additional access methods.

The device will then synchronize with the GPS and once done will answer NTP queries and provide PTPv2 service on the default domain (domain 0) on all network interfaces. The GPS startup process can take up to 15 minutes in some cases. You can monitor progress on initialization via the Status tab in the web GUI.

No further configuration of the server should be necessary.


Grandmaster Generation 1


Grandmaster Generation 2


Grandmaster Generation 2 with Additional Network Interfaces

Shipped network configuration

Each network port is configured with a unique static IP address when shipped. eth0 is configured for 192.168.1.1, eth1 for 192.168.2.1, and so on. This includes optional 10G SFP+ ports which are eth5 (192.168.6.1), eth6 (192.168.7.1), etc.

Please note, the TimeKeeper Grandmaster does not support having multiple network interfaces on the same subnet or multipath routing, and does not support netmasks more restrictive than /30 (255.255.255.252) to avoid common routing issues.

Modifying the server configuration by changing the console ports, otherwise modifying the Linux kernel, BIOS settings, installed devices or installed software applications may cause problems in performance or correctness. This can create support issues beyond FSMLabs' support obligations. Please contact support@fsmlabs.com before attempting to make any changes to the server software or hardware configuration.

Upgrading TimeKeeper Grandmaster hardware

The following steps describe how to upgrade via the web interface. In addition to TimeKeeper, you can also upgrade the basepackage, which is a set of components that upgrade the Grandmaster platform and provide new features for TimeKeeper to use. Please note, however, that when updating TimeKeeper and the basepackage on a hardware Grandmaster, TimeKeeper must be upgraded first, followed by the basepackage.

  1. Download the installer for the current release from FSMLab’s website
  2. Log in to the web interface (see the “Web interface authentication” section for details)
  3. Click on the “Configuration tab” on your GM
  4. Click on the “Update” tab under that
  5. Click “Update TimeKeeper install”
  6. Click “Choose File”
  7. Navigate to and select the installer you downloaded in step 1
  8. Wait about 20 minutes while the system reboots twice to complete the upgrade process
  9. Log back in to the web interface
  10. Confirm via the “Status” tab that it has the right version
  11. If you also want to install the basepackage, repeat except click “Update TimeKeeper base package” in step 5

RAID alerting and recovery

TimeKeeper Grandmasters have built in RAID support, and the state of the RAID will be reported on the status page of the TimeKeeper web GUI. If configured, TimeKeeper will also send alarms about disk failure and recovery through the normal configured alerting methods - syslog, email, SNMP, detailed in the section, “Alerting - Log files, SNMP, syslog, email, Windows event log.” Alarms will also be sent if the S.M.A.R.T. diagnostics of a disk reports that it may fail.

In the event of a disk failure, the Grandmaster will emit periodic alarms until the failed disk is replaced, but will continue operating. Once the failed disk is replaced, a reboot is required in order to initiate reconstruction of the RAID array.

Managing Grandmaster systems

TimeKeeper Grandmaster systems can have some of their settings saved, set, and restored with the tkctl tool. tkctl can also be used to change individual settings directly on the command line.

To use tkctl log in on the console or through ssh:

tkctl -h

All current Grandmaster settings can be printed with this command, as the admin user:

tkctl -a

Redirect this output to a file to save the current grandmaster settings:

tkctl -a > /tmp/tkgm_settings.tkctl

This file can be saved externally as a backup of the system state and modified if needed to reflect intended settings. This file can be applied with the -l option:

tkctl -l /tmp/tkgm_settings.tkctl

Settings are applied in the order they’re defined in the file. For example, if a line specifies the IP address of a VLAN interface, that VLAN interface must already be created so any line needed to create the device should be specified first. More detail on device creation is listed below.

These steps of saving off the configuration and applying the saved configuration can also be done with the web interface, under the Configuration tab, subtab Service and System Management. The section Configuration Management has an option to download and save the current configuration and also one to upload an existing configuration file to have it applied.

Individual settings can also be set or retrieved, like this to get the IP address set for eth0:

tkctl -g gm.network.devices.eth0.ipaddr

or to set it:

tkctl -s gm.network.devices.eth0.ipaddr=10.45.220.31

tkctl options

tkctl controls 2 different types of behavior on TimeKeeper Grandmasters - settings and actions. Settings are individual feature configurations like network options. Actions are steps involving an action and may relate to a setting, like restarting the network to apply configuration changes recently made. Here’s an example where you set the configuration for eth0, then restart the network in order to make that setting active:

tkctl -s gm.network.devices.eth0.ipaddr=10.45.220.31
tkctl -s gm.network.devices.eth0.netmask=255.255.255.0
tkctl -s gmaction.restart=gm.services.network

Or similarly a bond device can be created/configured/deleted:

tkctl -s gmaction.create=gm.network.devices.bond0
tkctl -s gm.network.devices.bond0.ipaddr=10.45.220.31
tkctl -s gm.network.devices.bond0.netmask=255.255.255.0
tkctl -s gm.network.devices.eth5.slave=1
tkctl -s gm.network.devices.eth5.master=bond0
tkctl -s gm.network.devices.eth7.slave=1
tkctl -s gm.network.devices.eth7.master=bond0
tkctl -s gmaction.delete=gm.network.devices.bond0

Here are the list of names/settings supported by tkctl and can be set (with -s) if applicable or retrieved (with -g). A setting that isn’t present or doesn’t have a value set may not be listed when using tkctl -a. If tkctl -g is used to query a setting that isn’t present, an empty value may be printed in addition to the expected values specified below.

A number of options can be taken using a ‘set’ using the gm.delete name to remove a name/setting. This can be done as above with a command like:

tkctl -s gmaction.delete=X

where X is the value to be cleared or removed. Names that allow deletions include:

Here are a list of sample actions possible:

These operations are reported over syslog along with some additional events in order to provide records of changes made over time by different users. Please refer to the alerting documentation for details on what these alerts look like and how they’re transmitted.

Note: tkctl -a does not include deletions of any deletable settings (see gmaction.delete above) including SSH listen addresses, syslog servers, RADIUS/TACACS+ servers, bonds, VLANs, routes, and rules. That means, upon using tkctl -l to load a backup saved with tkctl -a, you will end up with a union of the old and new configurations with respect to these settings. If you don’t want this union, be sure to delete these settings before restoring from the backup file. You can do this from the web interface or with the tkctl command, e.g.,

tkctl -s gmaction.delete=gm.auth.radius.3.host
tkctl -s gmaction.delete=gm.auth.radius.2.host
tkctl -s gmaction.delete=gm.auth.radius.1.host

Similarly, tkctl -a includes settings for any configured bonds and VLANs, e.g., gm.network.devices.bond0/1.enabled = 1, but it does not first create those bonds or VLANs. To prevent tkctl -l from failing, be sure to first create any bonds and VLANs referenced in the backup file before loading it. You can do that from the web interface, via the tkctl -s command, or by adding creation settings to the configuration file, e.g.,

gmaction.create=gm.network.devices.bond0
gmaction.create=gm.network.devices.bond0/1

for the above case.

Quick steps for Grandmaster replacement

A common use of tkctl is to save and restore entire Grandmaster configurations. Saving the configuration is important to provide continual backups of the entire Grandmaster configuration in case of failure. Restoring the configuration with tkctl is helpful when there’s been an inadvertent change, or when an existing Grandmaster needs to be replaced with a new unit running the same configuration.

To recap, a Grandmaster’s current configuration can be saved off in a file with:

tkctl -a > /tmp/gm_config

from the command line, or from the web interface select Configuration tab, subtab Service and System Management and click Save Grandmaster configuration to download the current settings as a file.

Once this file is in hand it can be applied to a new GM to get it up and running as a replacement in production, or as an identically configured unit to be used in test. This is done at the command line with:

tkctl -l /tmp/gm_config

from the command line, or from the web interface select Configuration tab, subtab Service and System Management and click Apply Grandmaster configuration file to upload and apply the existing file.

When using the web interface, the upload will also trigger a restart of both the network and TimeKeeper automatically. The tkctl command line interface will leave that step for the user. To make sure all settings are applied follow the tkctl load step above with:

tkctl -s gmaction.restart=gm.services.network

Alternatively, the provided configuration file could have gmaction.restart=gm.services.network included as the last line to automatically restart both the network and TimeKeeper.

Footnotes

  1. This takes effect after restarting TimeKeeper ↩︎

  2. Changes here take effect immediately ↩︎

  3. The numbers refer to list position, not array index. Only matters when using -s to add or delete entries. ↩︎

  4. This takes effect after restarting the network (this also restarts TimeKeeper) ↩︎