Overhauling Domain

By Arya K.

Previously, we discussed the massive development our domain library underwent over 2024. However, the project has been around much, much longer than that – the very first commit was made all the way back in 2015! It started out as Martin's side project and grew over the years into something with a lot of potential. It is now a major NLnet Labs project, and we're excited to make it the first choice for any Rust code with DNS related needs.

While the work we did in 2024 added a lot of functionality to domain, it was building on existing foundations. We had to update parts of these foundations in order to support the massive changes, but the majority of the existing codebase held up incredibly well. This allowed us to focus on adding new functionality rather than improving existing code. Now that our goals for 2024 are accomplished, we took a step back and decided to rework those foundations. There were some points of friction we noticed, and we decided to address them in order to make domain easier to use and more efficient.

What does this mean for you, the users of domain? Over the next few months, we're going to rework all of domain's API, making large-scale breaking changes in order to simplify and optimize things. domain will come out of this with the same functionality, but with a better interface, better performance (in both time and memory usage), and in a smaller and more maintainable codebase. This is going to be a difficult transition, but we've chosen to move forward as quickly as possible, as reworking these foundations will only get harder as domain grows.

Step 1: domain::base

The foundations of domain lie in the base module. It provides basic functionality for parsing and building DNS messages. It's strongly connected to the rdata module, which defines all the DNS record types known to domain. Every other module relies on these two in some way.

The biggest point of friction we found in base is octets generics. Every major DNS type, from Message to Nsec3Param, has a generic parameter called Octs. This parameter determines how the memory for each type is stored. A Message<&[u8]> would reference data from a byte slice, while a Message<Vec<u8>> would store data on the heap. This provides the users of domain with a lot of flexibility: they can control how the data they are working with is owned, allowing them to pick the most ergonomic and/or efficient option for their use case.

Unfortunately, this design didn't work out as intended. Users didn't know how to pick an octets type for their needs, leading to most users falling back to Vec<u8> everywhere. This led to unnecessary allocations and copying, even within domain itself. domain also had to provide a lot of boilerplate methods to work with octets types, e.g. converting to and from different octets types. The worst affected were methods that had to be generic over the octets type; they would almost always have to copy data into a new Vec-based allocation, because working with a generic octets type is really hard.

A secondary concern was that domain::base was trying to offer a high-level interface. For example, base::Message provides convenience methods for traversal, e.g. opt() (which finds the OPT record) or canonical_name() (which traverses a CNAME chain). Every invocation of such methods will traverse the whole DNS message; this is not a cheap operation, but the convenience of these methods makes it very easy to overuse them. domain strives to be correct, performant, and easy to use, and we realized that parts of base fell on the wrong side of that trade-off.

Introducing new-base

PR 474, a.k.a new-base, is our first step towards reworking domain. It introduces a brand-new API for the domain::base and domain::rdata modules. It's the culmination of 5 months of effort, with countless iterations and tens of thousands of lines of code written. We're excited to announce that it has been merged today! The new APIs live in their own modules and don't conflict with the existing code, so while new-base is part of the 0.11.0 release, you don't need to rewrite your use of domain::base etc. just yet.

new-base defines the same conceptual types as domain::base, but it applies different principles towards their APIs. It skews away from octets generics, instead preferring concrete byte slice types like &[u8]. This makes it simpler to use and understand. It introduces a new parsing system, inspired by the zerocopy crate, that avoids copying data in many situations. derive macros are provided that eliminate a lot of the boilerplate in parsing from and building into bytes. Altogether, these let new-base provide roughly the same functionality as domain, but with 2-3 times fewer lines of code.

An important facet of new-base is transparency. The API of new-base accounts for the inherent structure and limitations of DNS without hiding them from you. For example, the message parsing API doesn't provide convenience functions for finding specific records in a message; it forces users to explicitly iterate over all records and select the right one. This makes the performance of the code more obvious. new-base is an inherently lower-level API than domain::base, but we think this is the right trade-off so that users (and ourselves) can confidently build efficient software on top of it.

A lot of new-base is built on top of dynamically sized types (DSTs). For example, the new Message type is a concrete type storing [u8]. It can only be handled indirectly (e.g. &Message or Box<Message>), but it naturally guides users towards simply using references to itself. new-base provides a lot of machinery around DSTs, patching over some limitations of Rust along the way; in the future, we hope to factor this code out and make it more generally useful too.

new-base took so much effort because we wanted to make sure we got it right. In order to exercise the API, we also took the time to implement other modules in domain on top of it. For example, PR 482 and PR 491 implement new server and zonefile modules respectively. These are thorough rewrites, not just ports to new-base, because each one had interesting changes to the API we wished to explore. We'll cover them in future articles.

A Roadmap

new-base is the beginning of an umbrella project: domain::new. It has added the modules new::base, new::rdata, and new::edns; these are sufficient to begin porting over the rest of domain. While we're quite happy with the design we've landed on, we'd like to take this opportunity to ask for feedback from the community. domain 0.11.0 includes new-base (gated under the unstable-new feature flag) so that you can try it out yourself; we'd love to hear what you think!

In the short term, we're going to be working on Nameshed, our new primary name server project. This also builds on new-base, and the sister modules we're developing around it. Some of those modules, like new-zonefile, are fairly complete and we hope to merge them soon. As such, you can expect lots of further development in domain::new in 2025 and beyond. Once we're confident that domain::new provides all the same functionality as domain today, we'll consider a breaking-change release that will remove the old domain code entirely, but we'll announce such plans well in advance.

Conclusion

We're really excited about where domain's going. Our goal is to serve you, our community, and in order for domain to succeed in that, we'd love to hear your feedback. You can find all of these changes on GitHub; if you're using domain today or are considering using it in the future, please have a look and tell us if domain::new and its related PRs work for you.