Compliance Configuration

Compliance is enabled or disabled via the GUI by selecting the Configuration tab and selecting the “Enable Compliance” checkbox or by adding the following to the TimeKeeper configuration file /etc/timekeeper.conf:

ENABLE_COMPLIANCE=1

Once Compliance is enabled, reports can be configured in order to provide selective summaries on particular sets of hosts. Instances of those reports are referred to as audits. For example, there is a default report that includes all visible clients on the Compliance instance. This is referred to as the ‘Compliance global report’. Instances of this report for particular day/week/month/year is an audit that provides reporting data on all hosts for that particular time period.

To add custom reports restricted by IP address or hostname you can add an array of COMPLIANCEREPORT() entries in timekeeper.conf as follows:

COMPLIANCEREPORT0() { TITLE='1 host'; CLIENTSET='10.78.20.4'; ALERTTHRESHOLD=0.001; }
COMPLIANCEREPORT1() { TITLE='NY4 hosts'; CLIENTSET='*ny4*'; ALERTTHRESHOLD=0.00005; }
COMPLIANCEREPORT2() { TITLE='Subnet 10.10.*'; CLIENTSET='10.10.*'; ALERTTHRESHOLD=0.00005; }
COMPLIANCEREPORT3() { TITLE='Network 4.*, 3.*'; CLIENTSET='4.*,3.*'; ALERTTHRESHOLD=0.00005; }

These can be configured in /etc/timekeeper.conf directly or via the web interface. The web interface input would look similar to the following:

Each COMPLIANCEREPORT entry creates an individual PDF for each time period with the ‘CLIENTSET’ and ‘TITLE’ entered. This allows you to create custom reports based on your specific network architecture, hostname groupings, and so on.
The “*” wildcard matches any or no character in the IP address field when used in a CLIENTSET and only those matching the expression are used in each PDF report.

Additionally, multiple criteria in the same COMPLIANCEREPORT’s CLIENTSET can be specified by separating them with commas as in the last example. This way you can group all NY2 and NY4 hosts together for example, while keeping Aurora and London hosts separate. This could look like:

COMPLIANCEREPORT0() { TITLE='NY hosts'; CLIENTSET='*-ny4.domain.com,*-ny2.domain.com'; }
COMPLIANCEREPORT1() { TITLE='Aurora hosts'; CLIENTSET='*-aur.domain.com'; }
COMPLIANCEREPORT2() { TITLE='London hosts'; CLIENTSET='*-ldn.domain.com'; }

Excluding hosts from a report can be done using “EXCLUDEDCLIENTSET”, e.g.:

COMPLIANCEREPORT0() { TITLE='NY hosts'; CLIENTSET='*-ny4.domain.com,*-ny2.domain.com'; EXCLUDEDCLIENTSET='www.*'; }
COMPLIANCEREPORT1() { TITLE='Aurora hosts'; CLIENTSET='*-aur.domain.com'; EXCLUDEDCLIENTSET='www.*'; }

This would omit “www.west-aur.domain.com” but include “timeserver.west-aur.domain.com”.

If hostnames/IPs need to be broken down by function rather than geographic region that can be configured for also.

DNS resolution of hostnames for the audits can be enabled or disabled using the configuration key

ENABLE_COMPLIANCE_DNS='1';

turning off ENABLE_COMPLIANCE_DNS by setting it to 0 results in improved performance but hostnames are not resolved during compliance reporting.

Because different regulations have different reporting thresholds, Compliance allows for different warning and error thresholds. The ALERTTHRESHOLD is an option defined per report so that any client in the named CLIENTSET will be flagged as having an alert threshold breach if the reported accuracy exceeds the named ALERTTHRESHOLD (specified in seconds).

For example, a report may be configured to alert if any ‘trd.domain.com’ hosts report an offset greater than 50 microseconds:

COMPLIANCEREPORT0() { TITLE='Trading hosts'; CLIENTSET='*-trd.domain.com'; ALERTTHRESHOLD=0.00005; }

Similarly, WARNINGTHRESHOLD allows the configuration of a generally lower threshold that should be watched for as a possible warning of potential problems. For the above case, a warning threshold may be set at 25 microseconds:

COMPLIANCEREPORT0() { TITLE='Trading hosts'; CLIENTSET='*-trd.domain.com'; ALERTTHRESHOLD=0.00005; WARNINGTHRESHOLD=0.000025; }

This way every daily report will look for clients exceeding a 25us offset and report them as being in a warning state so they can be inspected for issues before exceeding the ALERTTHRESHOLD of 50us. For simplified submitted reports, the warning threshold can be skipped entirely, or configured on a separate ‘internal only’ set of reports.

The ‘Compliance global report’ uses an internal default of 1ms for a warning threshold, and 50ms for an alert threshold. If the global report isn’t needed or wanted in favor of just custom reporting, the global report can be disabled with:

ENABLE_COMPLIANCE_GLOBAL_REPORT=0

Compliance can also report on end to end accuracy - meaning that in addition to tracking the local accuracy/offset to the client’s time source, it can factor in inaccuracies in getting the time to your local time source. This allows you to build audits based on the reported accuracy values across the entire network.

Configuring a report to generate audits with this feature enabled looks like this:

COMPLIANCEREPORT0() { TITLE='Trading hosts'; CLIENTSET='*-trd.domain.com'; ALERTTHRESHOLD=0.00005; ENDTOENDACCURACY=1; }

With this enabled hosts will be reported on both reported local accuracy to their source and any inaccuracies reported by the sources themselves. This way audits will reflect the sum of inaccuracies across the network at each point in time.

Compliance reports may also specify the times of the day during which Compliance is to produce reports. These are configured using the parameters STARTTIME and ENDTIME which are specified as offsets from midnight UTC (00:00:00 hours). The maximum duration for a report is 24 hours (86400 seconds).

For instance, New York City weekdays 8-4 time can be represented as follows:

COMPLIANCEREPORT0() { TITLE='New York City'; CLIENTSET='*.nyc.your.org'; STARTTIME='13:00:00'; ENDTIME='21:00:00'; DAYS='1,2,3,4,5'; }

Tokyo, 8-5 plus daylight savings time, every day except Sunday

COMPLIANCEREPORT4() { TITLE='Tokyo'; CLIENTSET='*'; STARTTIME='23:00:00'; ENDTIME='32:00:00'; DAYS='0,1,2,3,4,5'; }

For localtimes that change (viz. daylight savings time), using offsets that include both intervals is recommended, starting an hour early for one season and ending an hour late for the other.

DAYS can be specified either as numbers (0-6 inclusive) or three letter English abbreviations (Sun, Mon, Tue, Wed, Thu, Fri, Sat). The default, blank, is equivalent to “*” which includes all days.

For a full set of Compliance configuration options, please see the rest of the configuration documentation in the “Configuration” section.

Reporting subsets of alerts

Compliance retains a subset of alerts and gaps in data depending on the configuration used. For example, the default gap length is 180 seconds and any client gaps 180 seconds or longer will be summarized in an an audit. A gap of 60 seconds won’t be included in the audit with the 180 second threshold set.

Similarly, clients can only provide alerts if there have been enough within a day to require reporting. TimeKeeper can limit alerts with the MINALERTCOUNT configuration option (labeled “Minimum client alerts required” in the web interface) on a per-report basis, as shown below. Please see FINRA CAT requirements for recording issues with persistent errors and thresholds for reporting to see if this feature will be useful for your compliance reporting requirements.

Clients that have less than the configured number of alerts in a configured day (10 in this case) will not be reported on, and their alerts will not be retained or included in summaries. The day considered for alerts by default is 24 hours starting at UTC midnight but may be changed using the STARTTIME and ENDTIME options described above. If a client has enough alerts, they’ll be retained in detail and reported on normally.

It can be helpful to also have all of the detail on the individual alerts even if there were fewer than needed to self-report. This is similar to the case where a gap of 180 seconds is reportable and shorter gaps are not recorded but you may also want to also be aware of 60 second gaps internally. An easy way to handle this is to configure additional reports to cover different levels of reporting. One report can be set with tighter restrictions and thresholds for internal tracking, with another set to industry standard thresholds required for external reporting. This way there’s a single clear set of data to report with, and another set that can be used to identify issues before they become part of a larger problem.

This type of internal versus external reporting might look like:

COMPLIANCEREPORT0() {
	TITLE='External report with minimal alert counting';
	CLIENTSET='';
	STARTTIME='13:00:00';
	ENDTIME='21:00:00';
	DAYS='1,2,3,4,5';
	ALERTTHRESHOLD=0.0001;
	MINALERTCOUNT=10;
}
COMPLIANCEREPORT1() {
	TITLE='Internal reporting with stricter minimal alert count';
	CLIENTSET='*';
	STARTTIME='13:00:00';
	ENDTIME='21:00:00';
	DAYS='1,2,3,4,5';
	ALERTTHRESHOLD=0.0001;
	MINALERTCOUNT=5;
}

It’s important to note that these alert counts are restricted to alerts that happen within the current day as configured with the optional STARTTIME and ENDTIME options. To report on alert counts that occurred before the start of the configured day, alternate Compliance reports spanning the earlier time window may be configured. Or alternatively, the MINALERTCOUNT value can be set at a lower value for more specific reporting, all the way to a MINALERTCOUNT value of 0 where all alerts are reported. The MINALERTCOUNT=0 value is the default behavior for Compliance reports and audits.

Optimizing Compliance configurations

In many cases, adding the following line to your /etc/timekeeper.conf is helpful:

CPU=-1

This allows TimeKeeper to run Compliance and the rest of the TimeKeeper suite on all CPUs on the system, which allows for higher throughput and quicker response times.

Also, as Compliance requires a lot of log analysis, disk speed is important. The faster the disk storage is, the faster TimeKeeper Compliance will be able to generate reports on your client data.

In the next section, we’ll go over how and where these reports are stored, and how they should be managed.