Bug 756 - NSEC3 insecure delegation validation issue
NSEC3 insecure delegation validation issue
Status: ASSIGNED
Product: unbound
Classification: Unclassified
Component: server
unspecified
x86_64 Windows
: P5 enhancement
Assigned To: unbound team
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-04-22 04:00 CEST by Scott Morizot
Modified: 2016-04-25 18:30 CEST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Scott Morizot 2016-04-22 04:00:25 CEST
http://dnsviz.net/d/otcnet.fms.treas.gov/VxO6ww/dnssec/

Last week, an attempted move to a different signing product exposed the bug captured in the above DNSViz results in the new product. The NSEC3 record for the insecure delegation did not have the NS bit set. (See RFC 6840 Section 4.4) The issue with the signing of the zone above (and other related zones) was resolved last weekend.

However, during the troubleshooting process I found that Unbound incorrectly validated the insecure validation from treas.gov. Since the NS bit was not set, the insecure delegation should have failed validation and the unsigned responses from the delegated zone should have failed validation in turn. BIND, BIND-based products, and Secure64 correctly validated the DS response as bogus. Unbound did not.

I had recently replaced the server I typically use to troubleshoot issues like this one and had not yet installed Unbound on the new server. Since this was a critical outage I was helping Treasury DNS staff identify and resolve, I didn't have time to install and configure my own installation. I tested against DNS-OARC ODVR instead. I didn't see the version of Unbound listed, but I believe DNS-OARC keeps it pretty current.

https://www.dns-oarc.net/oarc/services/odvr

If you have any questions, let me know.

Thanks,

Scott
Comment 1 Wouter Wijngaards 2016-04-22 16:27:43 CEST
Hi Scott,

Somehow I have failed to reproduce this.

I test with lookup for foo.www.example.com.   The answer for that is unsigned.  The www.example.com. no-DS reponse has no NS flag in the NSEC3.  When queried for foo.www.example.com. DS it also responds with the same as the www.example.com. contents (and a proof that foo.www.example.com. did not exist).

Was the response to foo.www.example.com. DS really somehow an NSEC3 that says foo exists?  For you, the otcnet.fms.treas.gov had its own NSEC3 (separate from fms.treas.gov?) (that is really mis-signed, but I am trying to figure out how unbound make it insecure for you while it is bogus for me).

The validator also complains that the optout flag is not set.  I see in the dnsviz output NSEC3 without optout flag; is this true?  Because if the optout flag was set, unbound was correct - it is very lenient for weird optout cases.

Best regards, Wouter
Comment 2 Scott Morizot 2016-04-22 17:28:19 CEST
Yes, optout is not used in .gov nor is it generally set in any lower level US government authoritative zone. So optout is not involved.

This was exposed by a signing bug. octnet.fms.treas.gov is insecurely delegated from treas.gov. (There is no intervening fms.treas.gov delegation in the public zone.) After the rollback, below is the response for the DS record query and the referral from an A query to an authoritative nameserver for treas.gov. As you can see, the NS bit is correctly set in the NSEC3 record.

dig @ns1.treasury.gov otcnet.fms.treas.gov ds +norecurse +dnssec

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> @ns1.treasury.gov otcnet.fms.treas.gov ds +norecurse +dnssec
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14974
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;otcnet.fms.treas.gov.          IN      DS

;; AUTHORITY SECTION:
treas.gov.              300     IN      SOA     nstsm1.perimeter.bpd.treas.gov. webadmin.bpd.treas.gov. 2010410930 10800 1080 1209600 1800
treas.gov.              300     IN      RRSIG   SOA 7 2 300 20160423151414 20160422141414 21512 treas.gov. 9aH3Pq4uJ6v19DB6tHPYlCxT+TMxoju2wg5NSxBheQFU2HwBTvqwu26Y IG3EhDJkwSWk96S1HD7JRmR5Kcx5fgMDAwPiNLKUoCVMUNPTAvoV752T YYRLJ0Y9PxEC2HuHCzRCKaJjKogFar98AYLFTixdB0QQ28RKvxriZKwM 5O4=
K5GCLHLJDBPPM4HOHCFQ72AGPDHB6IR9.treas.gov. 1800 IN NSEC3 1 0 10 44370D9B05A6C1D65CF7B85C4A K6RHUO4USBEIAHQE702HE2E8C0VB7DOC NS
K5GCLHLJDBPPM4HOHCFQ72AGPDHB6IR9.treas.gov. 1800 IN RRSIG NSEC3 7 3 1800 20160423125428 20160422124506 21512 treas.gov. hHmlun8sf/v1B9Bjt6fezDUB/l6nS2KN15doOzGOnrgmgwuMXUBsEFhB RuDl4zpPdN6EcBpMk2I8AuzK3JGatktYbho5BCBbXa6H+Ff9SaiQazjX 6Af7PlebZD9WKD+p4hMz0ELuujji235wLeGL3I61gxHOLBv+7IF2uPCU Hwo=

;; Query time: 81 msec
;; SERVER: 2610:108:4000:100a::249#53(2610:108:4000:100a::249)
;; WHEN: Fri Apr 22 11:15:16 EDT 2016
;; MSG SIZE  rcvd: 540

dig @ns1.treasury.gov otcnet.fms.treas.gov a +norecurse +dnssec

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> @ns1.treasury.gov otcnet.fms.treas.gov a +norecurse +dnssec
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57233
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;otcnet.fms.treas.gov.          IN      A

;; AUTHORITY SECTION:
otcnet.fms.treas.gov.   900     IN      NS      ns2.twai.gov.
otcnet.fms.treas.gov.   900     IN      NS      ns1.twai.gov.
K5GCLHLJDBPPM4HOHCFQ72AGPDHB6IR9.treas.gov. 1800 IN NSEC3 1 0 10 44370D9B05A6C1D65CF7B85C4A K6RHUO4USBEIAHQE702HE2E8C0VB7DOC NS
K5GCLHLJDBPPM4HOHCFQ72AGPDHB6IR9.treas.gov. 1800 IN RRSIG NSEC3 7 3 1800 20160423125428 20160422124506 21512 treas.gov. hHmlun8sf/v1B9Bjt6fezDUB/l6nS2KN15doOzGOnrgmgwuMXUBsEFhB RuDl4zpPdN6EcBpMk2I8AuzK3JGatktYbho5BCBbXa6H+Ff9SaiQazjX 6Af7PlebZD9WKD+p4hMz0ELuujji235wLeGL3I61gxHOLBv+7IF2uPCU Hwo=

;; Query time: 70 msec
;; SERVER: 2610:108:4000:100a::249#53(2610:108:4000:100a::249)
;; WHEN: Fri Apr 22 11:18:03 EDT 2016
;; MSG SIZE  rcvd: 346

The problem to see in the DNSViz results is that the NSEC3 record only has the TXT bit set, not the NS bit. So, as clarified in RFC 6840, it should have failed validation. And that means the validator should have considered otcnet.fms.treas.gov still covered by the security of treas.gov and the unsigned responses from it should have failed validation. That's what we saw in BIND, BIND-based products, and Secure64 DNS Cache.

Treasury rolled back the zones with this signing problem to resolve the operational problems, so I don't have a live example that can be used to show this bug. While the signing error was live, however, I discovered that unbound incorrectly validated the improperly signed insecure delegation NSEC3 response.

Or at least the version of unbound deployed at the ODVR did. The unbound instance returned validated insecure responses for otcnet.fms.treas.gov while the BIND instance at ODVR (as well as all other instances of BIND or BIND-based products in production or used in testing) correctly returned a SERVFAIL response instead.

I don't have any suggestion on how to confirm the error. You would have to recreate the NSEC3 record with no NS bit set covering the non-existence of a DS record in an insecure delegation from a secure zone.
Comment 3 Wouter Wijngaards 2016-04-25 17:02:22 CEST
Hi Scott,

So I created a synthetic test case.  And it does not reproduce the problem.
I modified it after your last comments, to have an exact matching NSEC3 record with just a TXT flag in it.  But that does not help.

Unbound neatly makes it bogus with '[0] unbound info: validation failure <foo.www.example.com. A IN>: no signatures from 1.2.3.4'

In this set up, there is
a) an unsigned response for the long name.
b) the intermediate labels, when queried for type DS, get an NSEC3 response with only a TXT flag set.  (And one NSEC3 that matches the queried name).

And this is as far as I can tell exactly what you have, but it still does not reproduce the issue.  I also do not see recent bugfixes that should impact the problem...  You are really giving good response data, but it does not seem to work out.

Best regards, Wouter
Comment 4 Scott Morizot 2016-04-25 18:30:01 CEST
It's not clear to me where you created the accompanying delegation in the synthetic test case. otcnet.fms.treas.gov is a legitimate insecure delegation from treas.gov. When Treasury tried to move to their new signer last weekend, it provided the invalid NSEC3 response, so the insecure delegation should have failed to validate and all responses from otcnet.fms.treas.gov (such as www.otcnet.fms.treas.gov), which were unsigned, should have validated as bogus.

Did your synthetic test case reproduce all the above? (The DNSViz responses in the historical analysis show how the entire delegated structure looked.)

Again, we don't use unbound in production at our organization and I didn't have it set up yet on my test/troubleshooting server. So I don't have any logs or other information. The only thing I can say is that DNS-OARC ODVR's installation of Unbound returned responses (validated as insecure) for names within the otcnet.fms.treas.gov zone. Their BIND version, my BIND installation, and the BIND-based and Secure64 products at US Treasury and IRS all returned SERVFAIL.

If you're certain you recreated the same scenario in your synthetic test case, then I suppose it could be an issue with ODVR's unbound installation. I do know this is an unusual corner case and it uncovered a number of bugs, including one with DNSViz. (Casey had a note in the code for this situation, but hadn't actually coded it yet. He didn't expect to see it in the wild. So during the problem, DNSViz actually reported it as a valid insecure delegation. That made troubleshooting a bit more challenging. Casey fixed DNSviz the next day.) But other large segments of the validating DNS infrastructure also incorrectly validated it as insecure instead of bogus. Since Casey is the only one who has actually fixed the bug so far, I'll avoid mentioning others by name.

I did search before opening this bug to see if it had already been reported and fixed in unbound, but I didn't find any matching bug reports. It seems unlikely, then, that the answer is simply that DNS-OARC is using an older version of unbound. So if you can't recreate it, I guess the question is why the ODVR instance of unbound validated queries from the delegated subzone as insecure instead of bogus while the improperly signed treas.gov zone was being published. (It actually impacted multiple insecure delegations in treas.gov and treasury.gov from what I was told. However, it was this one I was called in to troubleshoot and I don't have a list of the other insecure delegations that would have been impacted.)

Thanks,

Scott