Experimental support for AF_XDP sockets in NSD

By Jannik Peters

We have been developing support for AF_XDP sockets in NSD. They allow handling a higher rate of network packets than through the Linux network stack on supported hardware^[1]—For a list of supported drivers (and by extension hardware) and their minimum Linux kernel version please refer to docs.ebpf.io.

XDP (eXpress Data Path) is a powerful Linux kernel feature that enables fast, programmable packet processing at the earliest point in software, namely at the moment the network driver receives the packet. Using eBPF programs, it can drop, redirect, or modify packets before they reach the rest of the network stack, making it ideal for tasks such as DDoS mitigation, load balancing, or, in our case, enabling high bandwidth and high packet-rate processing of DNS queries in supported user-space software.

While XDP is a pretty exciting technology, building something that uses it isn't particularly straightforward... and neither is debugging it. Throughout this journey I hit several roadblocks. For example, not every network card and driver supports XDP and some not fully, making testing tricky. Even if testing is possible, drawing conclusions from the results proved to be interesting to say the least. For example, when running my initial prototype on an AWS metal machine I was first encouraged, then disappointed, and then confused. At times, I stepped away from the project and focused on other tasks to let things simmer in the back of my head. Let’s just say, my hair color has changed several shades during this process...

Luckily, with servers graciously provided by our friends at DNS Belgium, I was able to test my implementation and expand the support to drivers that don't support the zero-copy mode of AF_XDP sockets (see below in Driver issues).

In the end (with zero-copy disabled) I saw a 1.7x improvement in handled queries per second compared to UDP through the network stack. I expect the improvement to be higher with a driver that fully supports AF_XDP zero-copy mode.

However, because of all the weirdness with driver and hardware support, we would appreciate you trying out this feature over the summer and sharing your experience on the nsd-users mailing list, so that the release of this feature can take place smoothly.

Before heading into the technical aspects of this post, a quick note: Some parts assume basic knowledge about using XDP programs and omit explanations. If you don't know about something mentioned, it's likely not of use for you; but, if you want to learn, feel free to take a look at the references below.

How to enable AF_XDP sockets in NSD

As AF_XDP sockets in NSD aren't released yet, you'll need to use the features/af-xdp branch of the NSD git repository:

git clone https://github.com/NLnetLabs/nsd --branch features/af-xdp
cd nsd
git submodule update --init

NSD requires additional dependencies for AF_XDP sockets, namely libxdp, libbpf, libcap, clang, and llvm^[2]. After installing these dependencies, you can build NSD with AF_XDP socket support:

autoreconf -fi
./configure --enable-xdp # ... and other desired options like --prefix
make -j4
sudo make install

Then XDP can be enabled for a single network interface with the xdp-interface: <ifname> config option (support for multiple interfaces can be considered). By default, NSD loads an included XDP program to redirect UDP traffic to port 53 (regardless of IP address) to NSD's AF_XDP sockets. If you use XDP already, you'll likely want to load the XDP program that's distributed with NSD yourself (see below), or write your own, as NSD skips the xdp-dispatcher when loading the program (so it is able to unload the program even after dropping privileges)^[3].

With XDP configured, UDP traffic to port 53 (and only that and their responses) will be redirected to use the fast-path via AF_XDP sockets to the NSD processes. Any other traffic remains unchanged.

Loading the XDP program yourself

When you use multiple XDP programs utilizing libxdp's xdp-dispatcher, you'd want to load (and later unload) the NSD bundled XDP program yourself. If you don't use the xdp-dispatcher, checkout the next section about Requirements for a custom XDP program.

For that you'll need to set the config option xdp-program-load: no, and if you use a custom bpffs path (e.g. with sudo mount -t bpf bpffs /tmp/nsd/bpf) also set xdp-bpffs-path: <path>. Make sure that you pin the BPF MAP included in the XDP program. For example with xdp-loader you'd use: sudo xdp-loader load -p <bpffs-path> <ifname> <prefix>/share/nsd/xdp-dns-redirect_kern_pinned.o.

Requirements for a custom XDP program

If you want to write your own XDP program, you'll need to include a BPF_MAP_TYPE_XSKMAP with the name xsks_map:

struct {
  __uint(type, BPF_MAP_TYPE_XSKMAP);
  __type(key, __u32);
  __type(value, __u32);
  __uint(max_entries, 64); // max_entries must be >= number of network queues
  __uint(pinning, LIBBPF_PIN_BY_NAME);
} xsks_map SEC(".maps");

Configure NSD with:

xdp-program-load: no
xdp-bpffs-path: <bpffs-path>
xdp-program-path: <your-xdp-program-path>

And start NSD after loading your program and pinning the map into a bpffs.

Known issues/limitations

Driver issues

Some drivers don't support AF_XDP sockets at all (see docs.ebpf.io for a list of supported drivers), some only partially, and some are buggy. In some cases the AF_XDP sockets can still be used, but the ZERO_COPY mode needs to be disabled with xdp-force-copy: yes.

An example for when it is needed is with the i40e driver^[4] as it can't allocate zero-copy buffers and prints the following kernel logs for each netdev queue:

... i40e ... Registered XDP mem model MEM_TYPE_XSK_BUFF_POOL on Rx ring 39
... i40e ... Failed to allocate some buffers on AF_XDP ZC enabled Rx ring 39 (pf_q 39)

Additionally, the AWS ena driver has limitations on the amount of network queues it supports using native XDP with zero-copy AF_XDP, and might even have had a bug leading to some sort of queue saturation (I forgot the details as this was last year...).

Kernel regressions

Sometimes the performance can be very hit or miss when updating the Linux kernel. If you experience low throughput you might want to try a different kernel version. For example, we had half the performance in 6.11.0 compared to 6.8.0 (for both "normal" UDP and via XDP).

No DNSTAP/PROXYv2 support

DNSTAP and PROXYv2 are currently not supported when using XDP, but might be added in the future.

No VLAN support

Currently, NSD's XDP implementation doesn't support VLANs, but will likely do so in the future.

Concluding

Although AF_XDP sockets in NSD are still experimental, we’re eager to hear about your experiences with XDP in NSD on the nsd-users mailing list. This will provide valuable feedback on their practical use and performance across various network interfaces and drivers. With sufficient testing and operational experience, the feature is expected to transition from experimental to a reliable option for high-volume DNS server deployments.

Reference links

Information on eBPF/XDP can be found in the eBPF Introduction and the BPF and XDP Reference Guide.

NSD specific information can be found in the man page nsd.5.conf.