With so much traffic on the global internet day after day, it’s not always easy to spot the occasional irregularity. After all, there are numerous layers of complexity that go into the serving of webpages, with multiple companies, agencies and organizations each playing a role.
That’s why when something does catch our attention, it’s important that the various entities work together to explore the cause and, more importantly, try to identify whether it’s a malicious actor at work, a glitch in the process or maybe even something entirely intentional.
A year ago, under the leadership of the Internet Corporation for Assigned Names and Numbers (ICANN), the internet naming community completed the first-ever rollover of the cryptographic key that plays a critical role in securing internet traffic worldwide. The ultimate success of that endeavor was due in large part to outreach efforts by ICANN and Verisign which, when coupled with the tireless efforts of the global internet measurement community, ensured that this significant event did not disrupt internet name resolution functions for billions of end users.
March 22, 2019 saw the completion of the final important step in the Key Signing Key (KSK) rollover – a process which began about a year and half ago. What may be less well known is that post rollover, and until just a couple days ago, Verisign was receiving a dramatically increasing number of root DNSKEY queries, to the tune of 75 times higher than previously observed, and accounting for ~7 percent of all transactions at the root servers we operate.
On Sept. 27, Internet Corporation of Assigned Names and Numbers (ICANN) announced that the first root zone Key Signing Key (KSK) rollover – originally scheduled to take place on Oct. 11 – will be postponed. Although this was certainly a difficult decision, we fully agree that erring on the side of caution is the best approach to take. In this blog post, I want to explain some of the involvement Verisign has had in KSK rollover preparations, as well as some of the recently available research opportunities which generated data that we shared with ICANN related to this decision.
Every Domain Name System Security Extensions (DNSSEC) validator on the internet requires a Trust Anchor. This is a key, or a hash of a key, that corresponds to the root zone KSK(s). Whenever a KSK rollover occurs, validators need to update their trust anchors to include the new key. The design of DNSSEC includes a mechanism, commonly referred to as RFC 5011, whereby validators can automatically update their trust anchors. Because there has never been an operational root KSK rollover, RFC 5011 has never been tested in production. In assessing rollover preparedness, our folks, as well as others, began to identify and work with the community on correcting some implementation and configuration bugs with RFC 5011.
One missing piece, however, was a way to tell whether or not a population of DNSSEC validating resolvers had successfully updated their trust anchors. That’s why, in late 2015, I began writing an Internet Draft that proposed a way for validators to self-report their trust anchor set. This document, titled “Signaling Trust Anchor Knowledge in DNSSEC,” was adopted by the Internet Engineering Task Force’s (IETF) DNS-OPS working group, refined with some co-authors from Google and ICANN, as well as review from the working group, and published in April of this year as RFC 8145.
At the time that the RFC was published, I thought it was probably too late to have any impact on the 2017 KSK rollover, but would certainly be informative for any subsequent rollovers. However, I was pleased to learn that Internet Systems Consortium (ISC) implemented the draft specification of this protocol in their BIND software in mid-2016, and it was slowly being deployed as people updated their software. NLnet Labs also implemented RFC 8145 in their Unbound software in mid 2017, although the feature was not enabled by default.
The “key tag signals” from these validators are sent to the root name servers. Beginning in May, I began looking at the data sent to Verisign’s A-root and J-root in anticipation of the rollover. Figure 1, shows the number of sources sending signals over time. The red bars represent validators reporting only the old KSK. Green represents validators reporting an updated trust anchor set (i.e., both the old and the new KSK). The small amount of yellow areas on the plot represent sources that sent mixed signals.
As shown in the figure, the new KSK was first published in the root zone on July 11. Some signalers already indicated they had an updated trust anchor before then, however, more importantly, many of them had indicated they had not. These are validators that were either updated manually, or perhaps through a software update. There is a dramatic drop in “Not-Updated” signals beginning on Aug. 10, which corresponds to the end of the RFC 5011 “Hold Down Timer,” or 30 days after publication of the new key. This is good evidence that RFC 5011 worked for many validators.
What’s troubling, however, is the lingering amount of “Not-Updated” signals throughout the remainder of August and September. These validators appear to still have only the old KSK and are not accepting the new KSK into their trust anchor set. These represent six to eight percent of the population providing data, a figure we first shared with ICANN in late August. If the rollover was not postponed, these validators using only the old KSK would fail to resolve any domain names on Oct. 11, until their configurations were corrected.
Fortunately, the KSK Rollover Operational Implementation Plan easily accommodates the necessity to back out or postpone the progress of the rollover. The ability to do so was designed from the start and there is no urgent need to change the key from a cryptographic or security operations point of view. Rather than a zone whose key set is signed by the new KSK on Oct. 11, as ICANN has conveyed, we plan to continue publishing the root zone in its current DNSSEC configuration for the next calendar quarter. ICANN DNS expert and VP of Research, Matt Larson, penned a message to the DNS operations community, available here, which provides more information of the KSK Rollover Project from their perspective.
We were pleased with ICANN’s decision to postpone the KSK rollover in light of this data, which provides a known lower-bound of potential breakage that may occur as a result of the planned KSK rollover. We remain committed to working with ICANN and the community to prepare for these important changes, to help better understand why some validators have not updated their trust anchors, and assist in further community and relying party outreach efforts, all towards helping to minimize negative impacts when the KSK rollover occurs.
A few weeks ago, on Oct. 1, 2016, Verisign successfully doubled the size of the cryptographic key that generates Domain Name System Security Extensions (DNSSEC) signatures for the internet’s DNS root zone. With this change, root zone Domain Name System (DNS) responses can be fully validated using 2048-bit RSA keys. This project involved work by numerous people within Verisign, as well as collaborations with the Internet Corporation for Assigned Names and Numbers (ICANN), Internet Assigned Numbers Authority (IANA) and National Telecommunications and Information Administration (NTIA).
As I mentioned in my previous blog post, the root zone originally used a 1024-bit RSA key for zone signing. In recent years the internet community transitioned away from keys of this size for SSL and there has been pressure to also move away from 1024-bit RSA keys for DNSSEC. Internally, we began discussing the root Zone Signing Key (ZSK) length increase in 2014. However, another important root zone change was looming on the horizon: changing the Key Signing Key (KSK).
A few months ago I published a blog post about Verisign’s plans to increase the strength of the Zone Signing Key (ZSK) for the root zone. I’m pleased to provide this update that we have started the process to pre-publish a 2048-bit ZSK in the root zone for the first time on Sept. 20. Following that, we will publish root zones with the larger key on Oct. 1, 2016.
To help understand how we arrived at this point, let’s take a look back.
One of the most interesting and important changes to the internet’s domain name system (DNS) has been the introduction of the DNS Security Extensions (DNSSEC). These protocol extensions are designed to provide origin authentication for DNS data. In other words, when DNS data is digitally signed using DNSSEC, authenticity can be validated and any modifications detected.
A major milestone was achieved in mid-2010 when Verisign and the Internet Corporation for Assigned Names and Numbers (ICANN), in cooperation with the U.S. Department of Commerce, successfully deployed DNSSEC for the root zone. Following that point in time, it became possible for DNS resolvers and applications to validate signed DNS records using a single root zone trust anchor.
DNSSEC works by forming a chain-of-trust between the root (i.e., the aforementioned trust anchor) and a leaf node. If every node between the root and the leaf is properly signed, the leaf data is validated. However, as is generally the case with digital (and even physical) security, the chain is only as strong as its weakest link.
To strengthen the chain at the top of the DNS, Verisign is working to increase the strength of the root zone’s Zone Signing Key (ZSK), which is currently 1024-bit RSA, and will sign the root zone with 2048-bit RSA keys beginning Oct. 1, 2016.
On Nov. 30 and Dec. 1, 2015, some of the Internet’s Domain Name System (DNS) root name servers received large amounts of anomalous traffic. Last week the root server operators published a report on the incident. In the interest of further transparency, I’d like to take this opportunity to share Verisign’s perspective, including how we identify, handle and react, as necessary, to events such as this.