Bug 837 - root.key and root.hints regular file update enhancements
root.key and root.hints regular file update enhancements
Product: unbound
Classification: Unclassified
Component: server
All Linux
: P5 enhancement
Assigned To: unbound team
Depends on:
  Show dependency treegraph
Reported: 2016-09-28 07:17 CEST by Eric Luehrsen
Modified: 2017-02-21 11:01 CET (History)
4 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Eric Luehrsen 2016-09-28 07:17:25 CEST
Unbound-anchor and internal autotrust tools could use some improvement. Maybe I am missing something, but this is what I think I am seeing.

(1) on embedded flash ROM system each root update could add trouble. Unbound.conf quote: "...Write permission to the  file,  but also  to  the  directory  it  is in (to create a temporary file, which is necessary to deal with filesystem full events)." Unbound should really be writing working data to /tmp mount point (or windows %temp% env) which on ROM systems is tmpfs in RAM. This saves the amount of thrashing on systems not really meant for regular R-W access.

(1-a) the temporary file needs to be fixed name or /tmp/subdir/ locationso that chroot/bind jail splicing can work in these systems.

(2) on embedded flash ROM systems the root update rate could add trouble. It appears that the root key is updated on each application restart and regular intervals. However there is no configuration for this. if "ifup" events are used to restart or reload Unbound to so it can bind to new interfaces, then this file is written again. This is hard on flash. There should be a unbound.conf option for how old the root key needs to be (last DL date could just be in special ";" comment).

(3) the anchor tools update root.key but they dont appear to update root.hints. It would seem natural and necessary that both are updated at the same interval. It is particularly odd now as the anchor won't change for a while, but root servers do get hosted at different locations.
Comment 1 Wouter Wijngaards 2016-09-28 09:12:36 CEST
Hi Eric,

Unbound is trying to keep the file up-to-date for you, and it does that by writing every time the root key is seen by the resolver.  That is more than you want.

Then, you should take up the maintenance of the root.key file.  You can do this, and stop unbound from writing to it, by using the keyword trust-anchor-file: root.key in unbound.conf.  Unbound now reads the file but does not write to it.  

To update the root.key you could use unbound-anchor from cron (once a day), you could configure that unbound-anchor to work on a tmpfs copy of the root.key.  Copy it back if the lines that grep 'DNSKEY' are different between the current and worked-on copy of the root.key file.  (this skips the comments at the top, one of which is the 'last time of probe' field that would always be updated).

The root.hints are managed by making (some-of-) them the same for years, so that you do not need to update that file for the lifetime of the device.

Best regards, Wouter
Comment 2 Eric Luehrsen 2016-09-30 04:11:16 CEST
Hi Wouter,

Yes, I already have such a routine in Bash. All the file i/o in std-C would allow the same thing. Actually Unbound already has the bulk of the necessary routines, and they just need to be chained together with the right options. This is why I am making the request.

(1) It is poor form and security to write working files from the internet directly into /etc/. They should always be written to /tmp/ respective mount point and inspected before copying to /etc/.

(2) There is no reason to update the root key _EVERY_ time it is encountered. This can burden file i/o in general, and the special case of physical burden on flash ROM. Knowing the date a file was last written, it is a simple if() test to inhibit the file write except for N days later. 

(3) Certain root-servers shouldn't go away like VeriSign, but poor management and bankruptcies do happen. Unbound already has routines to spit out its built-ins and selected cache. Why not optionally chain such a routine so that root.hints updates with root.key?

In all, I would envision an related set of options:

auto-trust-anchor: "/etc/unbound/root.key"
auto-root-hints: "/etc/unbound/root.hints"
auto-trust-update: 30d
Comment 3 Wouter Wijngaards 2016-09-30 09:21:53 CEST
Hi Eric,

So, we have to write the root key every time, because unbound-anchor needs to know the time when it was last seen on the wire.  Based on that time and the RFC5011 hold-down timer, it can work out that a 'cannot see anything' is either 'just network outage' (file recent), or 'fetch new trust anchor' (>30 days).

This is why it writes to the file every time.  It also writes to it because this makes sure the file is writeable.  Many setups do not have writeable /etc, and this is necessary if a rollover happens.

Maybe surprisingly for you, but the root hints on the wire are not signed with the root key.  They are glue and glue does not get DNSSEC signatures.  The data plane does have signatures and this protects the file data contents.

Not as if such an option wouldn't be interesting.  But the root servers (as a set) are stable enough that this is not necessary today.  If you really want it, run an ftp (or http) get of the file from cron.

Anyway, back to the topic, you want to stop it from wearing down the drive.  So, instead of auto-trust-anchor-file, use trust-anchor-file.  Then, install a cron job that updates it if necessary.  Or update it by hand when the root key changes (i.e. schedule an update via the package management system at that time).

Best regards, Wouter
Comment 4 Eric Luehrsen 2016-10-01 19:38:22 CEST
Hi Wouter,

Wrt. RFC5011 I entirely agree, and I very much dislike tools that don't follow standards. However, I think we only are miscommunicatung based on an approach from different perspectives.

While the key may or may not be changing, the process of chasing and validating the key is really an internal function of the resolving server. This includes the hold down time. If it is desirable to maintain hourly incremental results as one might imply by RFC5011 2.3, then those incremental results should be stored in the respective /tmp/ mount point provided to Unbound. If we want even, then keep a rolling set of the last 10 hours worth (/tmp/unbound/root-timehash.key). The point is (A) keep the busy work in /tmp/; (B) most hourlies are no-change, trash, and should be in /tmp/; (C) it is just good practice, discipline, and security to inspect "from the wire" data in /tmp/ or other non-production area first (not /etc/).

The process of copying the key from /tmp/ to /etc/ should be more judicious. RFC5011 2.2 sort of implies this. The new key needs to appear and validate some other sets. Only when this succeeds should a /tmp/ copy to /etc/ occur. If a major key compromise or poisoning occurs in an enterprise network, then it may be necessary to reboot all servers and we only want a true-good copy in /etc/. Obviously, all the /tmp/ copies that are residual from another Unbound session must be ignored.

Also when the /etc/ copy is stable (usually) we may wish to allow Unbound know that it is reasonably fresh. That is the next time Unbound needs to be restarted. A file touch, or modifying a special ";" comment as to the last confirmation date is all that is required. This may be an optional time and/or while Unbound is terminated normally. Just as it prints stats to syslog, it could give /etc/ copy a touch on condition it was believed fresh data.

Therefore only on rare occasion of a true root update, or on some long self notation interval would /etc/unbound/root.key be modified.

Yes, the glue is not signed. Only the ". IN NS [each].root-servers.net" is and thats trivial. However, the one-by-one "[each].root-servers.net IN A" is signed. Then assemble a new root-hints based on valid IN-A. We can also fall back on https-get to internic.net just like icann.org. Again, it just seems convenient and to flow naturally to update root.key and root.hints together. When a major shift in the root-zone is detected either by IN-DNSKEY or IN-A (substantial address changes), then that is a time to update both.

The option I propose for update is really an override, such that if we don't have an good excuse to copy a update, then just do a full update process anyway (30 days).

And again, I have been working with a cron bash script prior to my suggestion. That is why I make the suggestion. It seems something that should be part of the feature set in any comprehensive name resolver.
Comment 5 Robert Edmonds 2016-10-01 21:33:20 CEST

I'm really confused by this bug report. An auto-trust anchor file is not a static configuration file, so it should not be written to /etc. For a Linux distro that tries to follow the Filesystem Hierarchy Standard, the closest match for Unbound's auto-trust facility is the /var/lib "Variable state information" directory (http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLIBVARIABLESTATEINFORMATION) which is used to store "data that programs modify while they run", and which should be stored on a persistent filesystem mounted read-write.

On Debian/Ubuntu systems, we set auto-trust-anchor-file: "/var/lib/unbound/root.key" in the default Unbound configuration shipped by the unbound package, and we use the following scheme to keep that file up to date:

1) The unbound package has a dependency on the dns-root-data package, which is an implementation-neutral package that contains static root hint and trust anchor information. It ships the root trust anchor in /usr/share/dns/root.key. The contents shipped by the dns-root-data package are only updated when the package is upgraded. The dns-root-data package should only be updated very infrequently, i.e. when root nameserver hint addresses change or when the root trust anchor changes.

2) Each time the unbound daemon is started by the system, we invoke a helper script that checks the freshness of unbound's root auto-trust anchor file (/var/lib/unbound/root.key) and the static root trust anchor file shipped by the system (/usr/share/dns/root.key). It performs the following logic:

a) If the static root trust anchor exists (/usr), and the auto-trust anchor does not (/var/lib), or the static copy is newer than the auto-trust copy, the static copy is copied to the auto-trust path.

b) The unbound-anchor utility is invoked on the auto-trust anchor file (/var/lib) to refresh it.

3) Then the unbound daemon is started.

This only requires a few lines of shell script to implement, see the do_root_trust_anchor_update() function in this script: https://anonscm.debian.org/cgit/pkg-dns/unbound.git/tree/debian/package-helper?h=debian/1.5.9-3.

This scheme allows the static system copy of the root anchor to be stored on a read-only persistent filesystem (/usr), while the copy of the trust anchor that is updated at runtime is kept on a read-write persistent filesystem (/var). It would be very easy to extend this scheme so that the runtime copy used by unbound is kept on a read-write nonpersistent filesystem (e.g. a tmpfs /var/run or /run) and only occasionally copied back to the read-write persistent filesystem (/var), e.g. after the unbound daemon is stopped, or by a scheduled job that runs every N days or so.

Basically, IMO, Unbound already offers all the trust anchor update functionality needed by both general purpose and embedded Linux distros, and it is up to the distro maintainers to configure where the auto-trust anchor file is stored.

But, I could definitely see an argument for a variant of the auto-trust-anchor-file option that reads from one path and writes to another, e.g.:

    # Only read permission is needed for this file:
    input-file: "/var/lib/unbound/root.key"

    # But this is the file that is actually updated:
    output-file: "/run/unbound/root.key"

Or even:

    # The contents of these two files are used as the combined input:
    input-file: "/usr/share/dns/root.key"
    input-file: "/var/lib/unbound/root.key"

    # This is the file that is updated:
    output-file: "/run/unbound/root.key"

(You would still need some distro-specific logic to decide whether to e.g. copy back /run/unbound/root.key to /var/lib/unbound/root.key after the daemon is stopped, or periodically.)
Comment 6 Eric Luehrsen 2016-10-02 04:12:36 CEST
Hi Robert,

>>so[auto-trust anchor] should not be written to /etc
In a full blown server, agreed. Sometimes though simplicity is just simplicity. We'd also need to chroot/bind jail all those locations. Ugh. I don't want to get hung up on the directory names. Whether its /etc or /usr/local/share or where ever, the discussion is about persistent storage and production areas versus non-persistent and okay to scratch in areas. 

>>Unbound already offers all the trust anchor update functionality needed...
Almost. The pieces are there, but maybe they can be better arranged. Its why I instigated this. Its also why I like your idea better.

I like this "input:" and "output:" idea. I like letting Unbound run the auto-trust function and outputting to an appropriate directory mounted in a tmpfs. I like not having an external program/script to to that periodically. I like not even having to copy a file up on launch. Let Unbound follow RFC5011 freely as it wills. Write to /var/lib all day, who cares. Boot strapping the auto-trust could be more robust then too, and just fall back if any file is missing or malformed.

    # If you specify input files, then the builtin is excluded
    # A fall back option _untouched_ from upstream package
  input-file: "/usr/share/dns/root.key"
    # A file you mess with locally - try second
  input-file: "/var/lib/unbound/root.key"
    # A file you can find on NAS - try first
  input-file: "//enterprise/deploy/unbound/root.key"
    # A file to free wheel RFC5011 into; but not fail if missing
  output-file: "/run/unbound/root.key"

    # If you specify no input files, then it uses the builtin
input-file: "/var/lib/unbound/root.key"
Comment 7 Eric Luehrsen 2016-10-02 04:14:15 CEST
[scratch that previous]

    # If you specify no input files, then it uses the builtin
    # and this file after it seeds. Almost carry over function.
  output-file: "/var/lib/unbound/root.key"
Comment 8 Eric Luehrsen 2017-02-21 05:52:26 CET
In OpenWrt/LEDE we are now using init scripts to configure, adjust, and jail (chroot) Unbound to only /var/lib/unbound. This is TMPFS and then we have tear-down scripts that copy root.key back to a permanent location when needed. The problem with this approach is that Unbound and unbound-control have no obvious way to default to /var/... and they each require the -c option. This is annoying, but not the topic of this ticket. Due to the desire to use a jail, the concept of an "in:" and "out:" option are less important. It would also be nice to have the root servers DNSSEC validated as individual lookups and stored to a fresh hints file. But will be a the subject of another ticket.
Comment 9 Wouter Wijngaards 2017-02-21 11:01:47 CET
Hi Eric,

Good to hear this stuff works.  Because the root key is going to get rolled later this year, so being able to pick up the new key is going to be important then.

There are configure time options to set the default location for a bunch of stuff,  --with-conf-file=path   --with-run-dir=path   --with-chroot-dir=path  --with-rootkey-file=filename .

Best regards, Wouter