Deckard - a DNS software test harness

Ever wondered about what breaks DNS?


  1. Testing DNS software is the pandora box that we (the developers and operators) always wanted to open, but feared what we might find. Why is that? It's daunting to test all possible aspects of DNS interoperability with the existing standards (and erratas) as there's always a special case. The second thing is the amount of (tedious) work required for each tested server. Here's the latest presentation from Autumn OARC 2015 that raises a lot of questions, to which I'll try to find some answers.
  2. What do we want?

  3. A simple to use tool that tells us repeatably how "good" the server is in terms of standard compliance and recovery from failure. Telling that the server operates well under normal conditions is a good start.
  4. What do we need for that?

  5. In order to create a repeatable environment, we need to be able to - set up model of the Internet (i.e. nameservers that we're going to interact with), communicate with the test subject (i.e. prod it with queries and check answers), and finally manipulate time (i.e. timed failures, server internal events). The last point is important since some DNS events may take a long time to happen (such as root KSK rollover).
  6. How do we do that?

  7. We can either set up a complicated testing environment in a dedicated lab or we can fake everything. There is a socket_wrapper library from the Samba folks which can create an isolated network for the tested subject using library injection. That means the application is free to bind to privileged ports, interact with mocked servers, and ignore what's going on in the real world. That satisfies the emphasised "repeatability". The second tool is a well known libfaketime - a library that alters a perception of time for the tested application.
  8. So instead of complicated lab setup, we can simulate everything on runtime.
  9. Simplifying things

  10. The Unbound from NLNetLabs went in the right direction and described test cases in form of a text file with the description of the test environment (e.g. which NSs exist and what can they answer), and a sequence of interactions with a stub binary. That's great, but it has two shortcomings - it's not easily portable to other servers, and it's not possible to test production binaries. This is why we made Deckard using the techniques mentioned above and mated with the test cases in Unbound.
  11. So we get the easy controlled environment, and we can describe it in simple test files.
  12. Practical example

  13. While we use the test case for Knot DNS Resolver (and building test cases for authoritative), curiosity made us test another resolver implementation - the PowerDNS Recursor. While it's not yet DNSSEC-ready, it passes most of the resolution tests just fine.
  14. Another good example of the tool applicability is testing new RFCs. Take the emerging QNAME minimisation from Stéphane Bortzmeyer and the Akamai's liberal usage of NXDOMAIN for empty non-terminals. If we ask a question - does it break the tested resolver? The tool gives you an answer in a fraction of a second, without the need to set up any specific environment.
  15. Is it any good for the authoritatives?

  16. Yes, although we don't have any test cases yet. The interactions between authoritatives happen over DNS protocol, which we can script in the test cases (zone transfers, notifications, updates). The test can also encode server malfunction, going lame, malformed messages and expiration for example.
  17. Does it sound interesting?

  18. Here's a README and a Git repository. To answer some of the questions posed by the mentioned presentation - yes it's a good idea, we're happy to provide the code, everybody can use it. I'd love to see that it gets the ball rolling and we can focus on enriching the test set instead of the platform.