Supporting DNSSEC Key Signing Ceremonies

By Berry van Halderen and Roland van Rijswijk-Deij

Introduction

Over the past decade, uptake of DNSSEC has grown significantly. The vast majority of top-level domains (TLDs) is now DNSSEC-signed. As you can imagine, the cryptographic keys used by TLDs and the root of the DNS are highly valuable, since if those keys were stolen, an attacker could effectively impersonate the operator of the TLD and manipulate the information for any domain under it. For this reason, many TLD operators closely guard their key material by storing it in a so-called Hardware Security Module (HSM). The same is true for the keys used to sign the root of the DNS, managed by the IANA and Verisign.

Such a setup, in which keys are protected by storing them in an HSM, is even more secure if the most important signing key – the Key Signing Key (KSK) – is in an HSM that is air-gapped. This is, for example, the case for the KSK of the root. But air-gapping also creates a conundrum: you need a special ceremony in which the air-gapped HSM is used to periodically create new signatures that sign records in the zone (typically the DNSKEY set, that also contains the Zone Signing Key (ZSK)). For the root zone, e.g., the IANA organises quarterly Root KSK Ceremonies that are live-streamed so anyone can witness the ceremony. Next to the root zone, there are many TLD operators with similar practices.

Goal of this project

While key signing ceremonies are now deployed in many places in the DNSSEC community, what is lacking is a common approach, especially related to tooling. Many ceremonies are highly custom, both in the process followed and in the tools used. This may be a barrier to more operators implementing good security practices, and a dependence on highly custom tools can make these ceremonies fragile. Therefore, with help of a grant from the NGI0 PET project,  NLnet Labs has built a prototype standardised DNSSEC Key Signing Ceremony, consisting of both documentation to help design a ceremony and tooling to integrate the ceremony in a DNSSEC signing toolchain.

About this blog article

In this long read we discuss the outcome of this project, including a detailed example of how a key ceremony could be conducted using this prototype. We would love to hear your feedback!

Deciding on ceremony requirements

When you want to design a DNSSEC Key Signing Ceremony, there are a few important considerations to keep in mind. First and foremost, you will need to decide whether the ceremony is just a technical process, that is executed by your own staff inside your own organisation, or whether it requires a form of public scrutiny. Example reasons for including explicit public scrutiny in your ceremony are that you want to be transparent to the community of the domain you are the operator for, or that stakeholders in the domain impose this requirement.

As we have learned the hard way in 2020, having public scrutiny makes designing and executing your key ceremony a lot harder. It is therefore important that such a process considers all possible scenario, also one where it is not possible for community representatives to physically attend a ceremony, or a needed quorum of community witnesses cannot be achieved. A good resource from which lessons can be drawn on contigency planning can be found in ICANN's blog on conducting a DNS Root Key Signing Ceremony in the face of COVID-19.

A second major consideration is choosing the system where you generate Zone Signing Keys (ZSKs). In the vast majority of cases, the ZSK needs to be online on the signer system, to deal with re-signing of records or creating new signatures when the zone changes. Only in cases where the zone of a domain rarely changes could you consider storing the ZSK in an air-gapped system.

Assuming the ZSK indeed needs to be online and lives outside of the secure environment that hosts the air-gapped HSM, you have to decide if ZSKs are generated outside the secure environment, by the signer system, or whether they are generated in the air-gapped HSM, as part of the ceremony. Considerations to take into account when choosing include:

• What will work best with the signer system you use? (think of software support, the need to replicate key material among multiple backup locations, ...)
• If your ceremony incorporates public scrutiny, what is the easiest way to allow your community representatives or stakeholders to assert the authenticity of the ZSK?
• Can your HSM securely export keys? And can your signer environment import them?
• If you are considering generating keys outside of the secure air-gapped environment, does you signer system have sufficient entropy to generate secure random numbers for key material?

While these two main considerations are the most important when desiging a new ceremony, there are many more details to consider. For an in-depth discussion of ceremony requirements, we refer to CEREMONY.md, included in our documentation repository on Github.

Automating the ceremony

One of the main goals of our project is to make it easier to automate the ceremony and to facilitate easy integration into a signer environment. To this end, our prototype consists of three core elements:

1. A recipe API, to describe the actions that need to take place during a ceremony.
2. Scripting that can run inside the secure environment and can interface with the HSM using the open standard PKCS #11 API that almost all HSMs support.
3. Integration into the OpenDNSSEC signer software.

For all three elements, we applied these core design principles:

• We avoid control logic in the secure environment; a recipe (specification of what needs to happen during the ceremony) should execute without branching.
• We minimise dependencies for the scripting that runs inside the secure environment.
• We choose a human readable file format based on JSON for the recipe API.

Below, we discuss each of the three core elements in more detail.

Recipe API

Using a "recipe" as a metaphor for what needs to happen inside of the secure environment during a ceremony is a deliberate choice. In the realm of food, a recipe specifies ingredients and amounts, a cooking process and if executed correctly, will lead to a cooked dish. In case of our recipe for the secure environment, the recipe specifies pre-conditions (e.g., keys that need to be present in the HSM), a set of steps to execute and leads to a "cooked" outcome: a set of signed keysets.

Recipes may contain any of the actions described below, actions are always executed in a fixed order (i.e. certain types of actions always come before actions of another type). This prevents the need for control logic. In a nutshell, a recipe may consist of the following actions (executed in the order as listed below):

• haveKey actions are used to check if a key with a certain identifier is present in the HSM in the secure environment; this action can be used to verify the integrity of the environment before the recipe is fully executed.
• deleteKey actions are used to delete retired and obsolete keys from the HSM.
• generateKey actions are used to generate new key pairs with a specified set of parameters (algorithm, key size, ...) and a specified identifier.
• produceSignedKeyset actions trigger creation of a signed DNSKEY set, which includes a specified set of keys and is signed using a key in the HSM in the secure environment. The action also specifies the validity time of the signature.
• importPublicKey actions can be used to import public keys into the HSM, which may be required if ZSKs are generated in the HSM and need to be securely exported for transport to the signer.
• exportKeyPair actions can be used to export a key pair securely from the HSM. This action type is used if ZSKs are generated in the HSM, or can be used to export keys that need to be securely copied to a backup facility.

A detailed specification of the recipe API, and two examples of recipes can be found in RECIPE-API.md in our documentation repository.

Scripting for the secure environment

The approach in this project is very much focused on the recipe.  In order to ensure no complex actions need to be undertaken during the ceremony, all complexity is put into creating the recipe.  This, of course, means that the recipe needs to be prepared and a ready-to-use recipe is available.  During a ceremony we are signing multiple sets of keys for a longer period of time.  Each set needs to become active at a different time in that period.  So management of these sets also needs to be taken care of in that period.

In this project we have created a tool for the generation of the recipe, and a tool for processing the recipe in the secure environment and taking the results into the operational environment.  Even though we've written the tool as one single script, it should be viewed as three different tools, as no assumptions are made that the recipe being processed in the secure environment is actually generated by the same tool or version of the tool.  It is in fact very much desirable not to bring an updated version of the tool into the secure environment unless absolutely necessary.  Since the operation of the tool inside the secure environments is its primary function (as the recipe could even be composed by hand), let us look at the operation of the tool in the secure environment first.

Cooking the recipe

Likely the best approach for software used in the secure environment to set up and possible certify the software in the environment once and then keep it static. The software should normally not be accessed outside of a signing ceremony.  This avoids any unexpected problems from changes or software updates during a ceremony.  In addition, any change in software would require revalidation in order to have the confidence that a ceremony will yield proper results.

It should not be a major problem that software inside the secure environment is only updated infrequently, since it is air-gapped without any input from the outside world other than the recipe. So somewhat out-of-date software does not pose a security risk. The only worry would be that if the software were to be too complex, there is a risk of it becoming rapidly outdated. This is mitigated by keeping the tool that cooks a recipe simple. Also for this reason the recipe has only the bare control logic, to keep dependencies low.

The one thing you want to avoid is manual intervention during the ceremony process in the secure environment. This process should avoid human error. For this reason, it should be possible to perform a dry run of the process beforehand outside the secure environment. This is easily possible as no decisions are made on what to sign when, and there is no notion of the current time in the tool when cooking the recipe.

The finished tool is Python-based, and has very few dependencies.  It requires two input files: the recipe, of course, that is being brought into the secure environment, and a configuration file.  This configuration file specifies environment-specific parameters for the secure environment, primarily how to access the HSM.

Such a configuration file might look like this (note: we sometimes refer to the secure environment as "bunker"):

version: 1
repositories:
  Bunker: &primary
    module: /usr/lib/softhsm/libsofthsm2.so
    label:  Bunker
    pin:    1234

On the command line, the tool accepts options to use an alternate configuration file or to increase verbosity of the tool. The tool itself is called "oks", and the cooking of a recipe can now be as simple as:

$ oks cook

The tool will read the input recipe from the file recipe.json, execute the steps specified in it, and then output the same recipe.json, but now with additional fields added.  This file can now be brought back from the secure environment to the operational environment.

During the execution only a linear pass is needed over the recipe providing feedback on the result:

Recipe Testing generated at 2020-11-27 09:58:07
Recipe step 1: Process key used in migration keys from Bunker to operational environment
Recipe step 2: generateKey
Recipe step 3: Generation key used for next ZSK
Recipe step 4: Export key hex 521c5afa8ce4cc2fde07bd9d40f77b3e encrypted using wrappingkey
Recipe step 5: produceSignedKeyset
Recipe step 6: produceSignedKeyset
Recipe step 7: deleteKey
Recipe completed.

Additional feedback such as the fingerprints of keys may be requested by setting a higher verbosity level. A more in-depth example of a run with input and output of the recipe can be found in the source tree of the toolset.

Producing a signed key set is the most important function of the tool, and the accompanying recipe action is shown as an example below.  The input in the recipe would be for example:

{ actionType: produceSignedKeyset
  actionParams:
  { ownerName: nl
    inception: 2020-11-27 09:59:07
    expiration: 2020-12-27 09:58:07
    ttl: 60
    keyset: [ {
        key: {
          keyType: byRef
          keyID: hex 4556957b8ea06427974a50973d5d0d31
          keyFlags: KSK
          keyAlgo: "8"
        } } {
        key: {
          keyType: byRef
          keyID: hex 521c5afa8ce4cc2fde07bd9d40f77b3e
          keyFlags: ZSK
          keyAlgo: "8"
        } }
    ]
    signedBy: [ {
        key: {
          keyType: byRef
          keyID: hex 4556957b8ea06427974a50973d5d0d31
          keyFlags: KSK
          keyAlgo: "8"
        } }
    ]
  }
}

After cooking the recipe this step in the recipe would be appended with the result:

{ actionType: produceSignedKeyset
  actionParams:
  { ownerName: nl
    inception: 2020-11-27 09:59:07
    expiration: 2020-12-27 09:58:07
    ttl: 60
    keyset: [ {
        ...
  }
  cooked:
  {
      '''
      nl.   60      IN      DNSKEY  257 3 8 AwEAAb5si0v8pvOpYLjSetik1k1OlUszv11x1TKklWC+zVodfLS4hZZ9l5zft88iFwM4sKevRtld4MLbrOSpSfFtGZLHdZuTAfc8UQPime4ARDzLzniM3MJgScHcChKH46B2XG4tfYo1Ju6PPnu924fGsWjbaXiabQ1sSmZx3hQVwBkrNoo3WT0qJsOY14N3ieZF95xKte90sXYWoIr97Axje+M9dXROEMRBQqWfDGYxjGnxze4IgaaFJGkRRkPhYpoBHBW/kPpwtwPJd7fOS/9z3Vwlcfiv86nZOCcwY6psT0aZjYCqjYtskVNE9nL65sTGB3UCBYdu2moJpHg7v1PYqDM= ;{id = 53455 (ksk), size = 256b}
      nl.   60      IN      DNSKEY  256 3 8 AwEAAbmwnNpRAIUFoVe+N4Hv+p8kCS6PPhqq342NpQASEiCLWOR0yv4QMfE75ICXE+pJxHyvv+LlQeBNelPIy7SvX9UWKULhJdYylqo7NSa4tRoaYgCMp0KKnk68+hWj6I1PW6Kd/Kpdpk+AjSecjbKIdtVLQsN8svKOnoasGElfCZc9WDr6mQ8uacZrmhtERt/rYYGnt41yyh6K44xdPpv5X0fXL1fIwYSBxTtlA55ogyfLS8HBkc2Cd/XNrnwq58z+xD1E4r97Ehqwjh3Chd2HMW2s5nbU5zYFhtng8xiNUla873cd0ffLS8os5iqCyLVG6Q2qPHNp2lcqgoZ5lVnn/q0= ;{id = 9148 (zsk), size = 256b}
      nl.   60      IN      RRSIG   DNSKEY 8 1 60 202012270958 20201127095953455 nl C3gUA3lO8JPjzaplzViZ+Rt/ubQCyAA9DMNa7nr6qfQ4FEZ9Syzh+DWDs3fj/RwCyIfOpEdZFA41YHB8b4jqphSRoElR5W1U7gHt8Xrt3ZB1ZxGrpsAuc/sPT9+MmwY8K0XiIV+QMVFtDBspo2tWDwt4q9hBUFV+DDzHEV1q/tPxyUgBhp/CcSgjDWW235IOnoEohG/bvFk/CFRFzpS9xGV0ljso6u6wtNxtFAPU4RfpaNDtHorV/3e+mfjbXZUhuYK+4Ouzk/pSU7UXmDQAhXpD/slzaAhEtYEGPzCCG8p0THTtmKll0hxNeC1sgJQJ8Wavlp33z7jqH5scshoNew==
      '''
  }

For an automatic operation where any observers do not need manual verification of each step this is ideal.  The recipe can be verified beforehand to be correct and with a verification that the right script is used to cook the recipe one can have complete confidence that correct keys are used for the signatures.

Producing a recipe

A recipe can be produced in its entirety beforehand, without the need for any observers of the ceremony to be present.  It is also wise to test the execution of the recipe by having a similar environment available (without the need of a full HSM).  

The tool actually supports pre-producing recipes. To, for example, produce a recipe for the example.com domain, the following command can be executed:

$ oks -c oks.conf produce example.com 2021-12-31 "Keysets for year 2021"

This produces a recipe that when being cooked produces signed key sets from the current date up to the indicated date. For this to work, the tool of course needs to be configured with a  key and signing policy.  This can be specified in the configuration file:

version: 1
repositories:
  Operational:
    module: /usr/lib/softhsm/libsofthsm2.so
    label:  OKS
    pin:    1234
kasp:
  refresh:         P3D
  validity:        P1M
  inceptionoffset: PT3600S
  ttl: 60
  ksk:
    algo: 8
    size: 2048
    lifetime: 1Y
  zsk:
    algo: 8
    size: 1024
    lifetime: 1M
transport:
  key:
    label: recipekey
    size: 2048

The kasp section specifies the key and signing policy.  This is heavily influenced by the OpenDNSSEC KASP configuration. It will specify which type of keys are used and what their lifetime is. Other parameters determine how frequently to produce a signed key set.

When using OpenDNSSEC, these parameters could be directly taken from the kasp.xml policy file.

At the time of writing of this blog, the prototype implementation of the tool has a full set of features to support ceremonies, albeit with limitations on how these ceremonies are structured. For example, only a single KSK roll is supported per ceremony and this will commence immediately. Also a number of other assumptions are made, like performing a pre-publishing non-algorithm rollover, etcetera.  The production of the recipe also assumes the ZSKs are generated inside the secure environment and are exported, preferably using a wrapping key to ensure the keys cannot be retrieved from the recipe while the cooked recipe is transferred from the secure environment to the HSM in the operational environment.

Using the ceremony output

Normally, a successful ceremony will result in a cooked recipe that contains multiple signed keysets, where different signed keysets need to become active over time.  Therefore, a specific signed key set needs to be selected from the recipe at specific times. But over time also new keys may be introduced. The processing of the recipe is therefore split in two:

First all new keys will be imported into the HSM when the consume tool is run on the recipe without arguments.

$ oks consume

If, however, the tool is run with a specific time given on the command line, it will produce the signed key set that would be appropriate for that time.  This makes it suitable to be run on a daily basis for instance:

$ oks consume 202102010000

The signed keyset is produced in a signer configuration file (signconf) which is also inspired on OpenDNSSEC, but can be further processed by other software needed.

Integration in OpenDNSSEC

Some concepts of the tool have been influenced by OpenDNSSEC, like the Key And Signing Policy (KASP) and the signing configuration (signconf). These are, however, hardly unique to OpenDNSSEC, and would be a natural fit for many other signing solutions, as the input files necessary are deliberately kept separate from OpenDNSSEC or can be provided without having OpenDNSSEC.

The produced output is a ready-to-use signer configuration file for the signer of OpenDNSSEC. While this may seem like this ties the tool to OpenDNSSEC, note that OpenDNSSEC was very much explicitly designed such that this signconf file could always have been produced or picked up by a non-OpenDNSSEC solution.  OpenDNSSEC has separate processing for key management and the actual signing operation, called the Enforcer and Signer respectively. There are already many examples where people run either of these tools with a different solution for the other.

If integrated in an OpenDNSSEC environment, this tool replaces the function of the enforcer daemon. After setting up use of the tool, the enforcer should no longer be used, as the enforcer would otherwise control the rolling of keys.  This task is now already prepared when having cooked the recipe.

In fact normally a signer would produce the signed key set by signing the DNSKEY set using the KSK.  This is however now done in the ceremony and the OpenDNSSEC signer daemon should just use the signed key set that has been prepared in the secure environment.  To integrate this functionality with OpenDNSSEC a modification to OpenDNSSEC was necessary.  OpenDNSSEC now supports pre-signed key sets and KSKs, if they are specified in separate fields in the signconf configuration.

Further reading, example runs, and development

The tool set and the ceremony documentation are available in our repository:

• https://github.com/NLnetLabs/dnssec-ceremony-tools/tree/master
• https://github.com/NLnetLabs/dnssec-ceremony-doc/tree/master

And are free to be used. We love to hear from you and any comments you have.

Acknowledgements

This work was made possible by a grant from the NGI Zero PET project, which is part of the European Commission's Next Generation Internet programme.