Home DNS failover, taken too far on purpose

September 2026 — you would rather read this than do it. Good. I did it so you don’t have to.

Every home network has one thing that, when it breaks, makes everything look broken: DNS. The router still routes, the fibre still carries bits, the NAS still serves — and nobody in the house can open a page. If you run an ad-blocking resolver (AdGuard Home, in my case, in Kubernetes), you have also made DNS depend on the most complicated thing you own.

This is the story of a week spent on a question that does not deserve a week: how do I make home DNS failover properly? Not “add a second entry to DHCP” properly. Measured properly.

It got out of hand. There is a BGP session to a router now. There are three AdGuards and an unbound. There is a benchmark tool, which was wrong the first time. There is a table, which was wrong the first time too. Let’s go.

The starting point, and what was actually wrong with it

One AdGuard pod on a three-node cluster, exposed on 10.1.1.236 by MetalLB. DHCP hands every client 10.1.1.236, 10.1.1.1 — AdGuard first, the router’s own resolver second. A MikroTik scheduler script probes AdGuard every five minutes and, if it is dead, rewrites the DHCP option to 10.1.1.1 only.

On paper this is failover. Two things are true in practice that the paper does not show:

1. A client with two DNS servers does not fail over; it waits. A stub resolver asks the first server, waits its timeout (2 s on most systems), then asks the second. It does that for every query while the first is down. Pages load; they load like 2003.
2. A second DNS entry gets real traffic even when the first is fine. When I ran three AdGuards, every one of them saw queries — phones, laptops, a smart TV. The “backup” quietly serves a share of everything.

Neither is an AdGuard problem. They are what DHCP-listed resolvers are.

Three designs, argued into shape

Every design below went through the same treatment: write it down, then write the attack on it — every way it could fail, lie, or cost more than it looks — and only build what survived. Half of this post exists because of the attacks, including the one that caught my own benchmark lying.

Design 1 — the relay. Clients get one address, a stateless DNS forwarder (dnsdist). It holds an ordered list of upstreams — home AdGuard, a second AdGuard on another cluster, the router — health-checks them every two seconds with a real lookup, and sends every query to the first healthy one. It has a packet cache that keeps serving expired answers while no upstream is available. The DHCP script stays, demoted to one job: if the relay itself dies, hand out the router. AdGuard becomes “just software” behind a fixed address, which also let my DR tooling move it between clusters without anyone noticing.

Cost the attack found immediately: AdGuard sees the relay as the client, not your phone. dnsdist forwards the real client in EDNS Client Subnet and AdGuard logs it, but AdGuard’s dashboard books the query to the relay — per-client rules stop working. Fine for me (I had none), noted for you. Cost I found later, measured: AdGuard’s per-client rate limit (100 q/s) now applies to the whole house, and its whitelist did not exempt the relay. Turn the limit off behind a relay.

Design 2 — anycast. Three AdGuards, each announcing the same service address (10.1.9.53/32) to the router over BGP, with BFD for sub-second failure detection and a small health check on each node that withdraws the route when its AdGuard stops resolving — done the boring way, by taking the interface that carries the address down, so the announcement follows the link. The router ECMPs across whoever is healthy. Clients talk to AdGuard directly — identity intact. No relay, no DHCP games, no script. This is how DNS is done at scale, shrunk to a living room.

The catch, found by measurement rather than argument: when all members are gone there is nothing — the route disappears and the address is unreachable. Every other design had the router as an implicit last resort; anycast needs it made explicit (a netwatch that switches on a dst-nat redirect to the router’s resolver while no member answers — MikroTik-native, two lines).

Design 0 — the baseline, kept as the control group.

Two things ran through all of them: adguardhome-sync keeps every AdGuard’s rules, rewrites and filters equal to the one at home (there is no “one config, many instances” mode), and cache_optimistic — AdGuard’s serve-stale — answers from an expired cache entry immediately and refreshes in the background, so an upstream blip is invisible for anything looked up recently.

The lab

Three Incus containers on an Orange Pi 6 Plus (Debian 12, arm64 — after I first launched them as amd64 under emulation without noticing, which would have made every latency number a lie). Each: AdGuard Home + FRR (bgpd, bfdd), announcing 10.1.9.53/32 to a MikroTik CCR2004, which already peered with MetalLB — the “BGP on the router” part was a filter and three connections, not a new religion. A fourth container ran unbound for the resolver comparison. The relay ran on another LAN host. The baseline was the real production AdGuard.

Containers on one host are a functional stand-in for three boxes. They test the mechanics; they do not test the host dying. Say that in your own post too.

How I measured — including the version that was wrong

A small tool plays the phone: a fixed number of queries per second against whatever address the design hands out, a household-like mix — six popular names (cache hits, what most real traffic is), two random names under a real zone (never cached, always walk to the upstream), two .lan names (the path back into the LAN, which fails independently). One CSV line per query. Each scenario: 30 s calm, an action, 60 s, the revert, calm.

The first version sent queries one after another and waited for each answer. That is not what a phone does, and it is a classic benchmark bug (coordinated omission): every 2-second stall also stalled the sending, so during outages the tool quietly ran at 0.5 q/s, and “queries completed” looked like a result when it was an artefact. The tool now sends on schedule whatever happened to the previous query, and the numbers below are from that version. The first table is in the repo, for penance.

The values, and what each is for — because a table full of numbers hides more than it shows if you don’t say which ones matter for which design:

• Lost — queries that got no answer at all. What a game, a video call, a Chromecast feels. Relevant where a design drops (relay, anycast); the baseline does not drop, it stalls.
• Stalled — answered, but only after a full 2-second timeout via the second DHCP entry. The baseline’s whole story is in this column. It “answers 100 %” and it is unusable.
• Felt outage — from the first bad query to the last one’s answer, for the longest stretch of trouble. The one number to compare designs by.
• Worst gap — longest run of consecutive lost queries; the design’s detection time, visible.
• Deliberately not in the table: “answered %” (100 % in the worst rows — the most misleading number in DNS), p50 (identical everywhere; it says the cache works), BGP/BFD timer minutiae. A whole-run p95 also hid a 20-second outage from me in the first version: 15 stalls out of 1300 queries is “p95 = 39 ms”. Percentiles over a quiet run are how you lie to yourself about short outages.

Results

150-second runs, 10 q/s from a wired LAN host (the client’s limits are real: one flow, one timeout model; more below). Steady state first: all three answer cache hits in about a millisecond and misses in ~40 ms; the relay’s cache makes it the fastest (p50 0.4 ms), anycast adds half a millisecond of router hop. Now break things.

Reading it:

• The baseline is a two-second tax on every query for as long as anything is wrong. A restart of the single pod — a normal deploy — is a 21-second outage, because the pod uses strategy: Recreate (it owns a volume). My first draft said “rolling restarts are not outages”; that was the diluted percentile talking. Retracted, with data. A 60-second stop costs 91 seconds, because the pod takes another 25 s to come back. Flapping costs almost the whole run.
• The relay loses a handful of queries per event — the ones in flight during its 2-second check — and nothing else. Its all-dead row: home and worker AdGuards both gone, the router served, 13 queries lost across the two swings, latency unchanged. Flapping: zero lost, because its packet cache answers the 60 % of traffic that is cached without asking anyone.
• Anycast has two different “one node dies” numbers, and the difference is the most useful thing I learned. Killing the container — even with --force — loses nothing: the host kernel tears the network namespace down and closes the BGP socket, the router gets a TCP close and withdraws the route in the same instant. I watched the router’s BFD session and route vanish within the first 250 ms sample. My first draft said BFD did that; it didn’t, TCP did. A frozen box — power gone, kernel dead, cable out — sends no close, and then BFD’s 3 × 300 ms is the detection: 16 queries lost in 0.8 s at 20 q/s, measured by dropping all of the serving node’s traffic while it stayed “up”. That is what BFD buys, and what it costs.
• Anycast’s weak row was all-members-dead: 60 s of nothing, until the router got a seat in the anycast (netwatch → redirect); then 11 lost in 1.1 s. The relay had that last resort built in.
• WAN down with cache_optimistic: exactly the never-cached fifth of the mix failed and nothing else — 124 of 124 misses, 0 of ~1200 cached names. Without serve-stale that row reads “everything fails for a minute”. Keeping the route up during a WAN outage is right: withdrawing would hand the LAN to a router that cannot reach upstream either.
• The last row is a partial failure: one anycast node can still resolve the internet but has lost its way to the router, so it cannot answer local .lan names any more. A node like that looks healthy to every public probe. The first time I ran it, my own health probe said “fine” too — it asked for storage1.lan, which was sitting in the node’s cache, and serve-stale served it happily. The probe now asks for a random .lan name that cannot be cached, and reads NXDOMAIN from the router as “the path works”. With that, the broken node withdrew itself 4 seconds after the break and came back 6 seconds after the repair; the client saw nothing.

Two side quests that turned out to matter more than the topology

Reverse lookups. A friend mentioned that reverse lookups of his own LAN addresses were noticeably slower on AdGuard than on unbound. Measured over the 57 addresses in my lease table, cold pass then warm pass, p50:

unbound’s warm reverse lookup equals its cache path: it caches the answer. AdGuard’s stays at cold speed, and the TTLs tell you why: the router hands out these PTR records with a TTL of 900; unbound returns them counting down (899, 898…); AdGuard returns TTL 300 on every single answer, never decreasing — a fresh forward each time, re-stamped. Forward .lan names are cached normally (their TTL counts down). Reverse lookups of private addresses take a different path inside AdGuard — the “private reverse DNS” resolver (local_ptr_upstreams) — and that path has no cache in front of it.

This is a known, open issue, not a quirk of my setup: AdGuardHome #6950, “Private Reverse DNS Queries aren’t cached, its consequences and more”, with the usual neighbours (#6693, #7586, discussion #4476). The project’s advice amounts to “point the private PTR upstream at something fast”, which is what I have — a router that answers in 0.5 ms — and why the cost here is 1.5 ms per lookup instead of the tens of milliseconds people with a slow upstream report. It is still one round trip per reverse lookup, forever, on the one class of name a home network asks about constantly: its own. Anything that names clients — dashboards, log viewers, AdGuard’s own client list — pays it. That is the difference my friend saw.

Writing the reverse records into AdGuard would fix it (a filter rule per address, 24.1.1.10.in-addr.arpa^$dnsrewrite=NOERROR;PTR;node02.lan., answered locally), but it means mirroring the router’s lease table into every AdGuard to save 1.5 ms. Not here. The honest fix is upstream, or unbound.

Capacity, and the trap inside it. resperf ramping cache-hit traffic to 12 000 q/s from one client: the router’s resolver tops out at ~6 400 q/s, AdGuard in k8s at ~6 800, AdGuard on the Pi at ~10 600, unbound and dnsdist and the anycast beyond 12 000 without loss. All of that is two orders of magnitude above a household’s peak, so capacity is not a differentiator — except that the first run capped every AdGuard at 102 q/s with 66 % lost. That was AdGuard’s per-client rate limit, and its whitelist did not exempt the test source. Behind a relay the whole house is one client. If you run a relay, turn AdGuard’s rate limit off; if you want per-client fairness, it belongs in the relay.

Pros and cons, to bore you properly

The fourth design, not built yet: DNS that follows the device

Everything above is about the LAN. Phones and laptops leave the LAN, and they can speak encrypted DNS (DoT/DoH) on their own. So: give them a name — dns.djieno.com — instead of an address. At home that name resolves to the anycast address and the phone talks to the local AdGuards. Away, it resolves to a small public entry point I run in a data centre (the “edge” — a separate article; think of it as the front door for everything of mine that is reachable from the internet), which forwards to whichever AdGuard is alive. Each device gets its own name (pixel.dns.djieno.com), which AdGuard uses as the device’s identity — so the filtering, the rules and the statistics follow the phone to the caravan and the train.

What it needs: a public certificate for *.dns.djieno.com, TLS on port 853 at both ends (Android only does DoT), and split DNS so the same name means “local” at home. What it does not cover: TVs, plugs, cameras — anything that cannot do encrypted DNS stays on the LAN path. What to measure before believing it: the TLS handshake cost on Wi-Fi and on 5G, and whether any .lan name ever leaks to the internet. Next round.

What I did not measure, and what went wrong

• One wired client, one flow, a fixed 2-second timeout. Real households are dozens of Wi-Fi devices with different stub resolvers (Apple sub-second and parallel, glibc 5 s, Android in between); ECMP is per flow, so one node dying touches a third of the devices, not all — and none if it was killed rather than frozen.
• No router reboot: the router is the LAN; everything is zero for that minute.
• The relay’s WAN-down row is missing (I could not cut the production pod’s upstream from outside); the anycast row stands in.
• The containers share one host: the scenario that matters most for “the family factor” — the box dies — is the one this lab cannot produce. The frozen-box simulation is as close as it gets.
• A Raspberry Pi that had quietly logged the water meter since 2021 was going to host the relay; it turned out to have had no working DNS for four years and did not survive its upgrade reboot. Take the SD image first. I did.
• Vacuous runs that looked perfect, caught before publishing: a stop of a node the client wasn’t even using; an “upstream unreachable” that never applied (no iptables in the containers); a .lan break hidden by a cached answer; a benchmark that throttled itself. Perfect results are the ones to distrust.

What I conclude, after a week of this

1. The DHCP second entry is the enemy. Every design that hands out one address and does the failover behind it beats every design that lets the client “fail over” — by two seconds per query, for as long as the outage lasts. This is the whole post in one sentence.
2. The relay and anycast are closer to each other than either is to the baseline. A handful of lost queries per event versus zero-to-a-dozen; both invisible to a human. The choice between them is about identity (anycast keeps the client address, the relay doesn’t), about what you enjoy running (a list in a config file versus BGP peers), and about independence from the cluster (anycast on real boxes is DNS that survives Kubernetes having a bad day).
3. Serve-stale is the cheapest availability you will ever buy. One setting in AdGuard, three lines in dnsdist, and a WAN outage or an upstream hiccup stops being a DNS outage for anything you looked up in the last hours.
4. Detection is where the design lives. Killed processes announce their own death (TCP close); frozen boxes don’t (BFD, 0.8 s); slow boxes announce nothing at all — none of these designs catches a node that answers in 1.8 s instead of 2 ms, and that is the next gray failure to build a probe for.
5. Measure the tool before the system. My first benchmark produced a clean table that flattered every design, including the one I was replacing. The second table is uglier and true.

What is missing, and what I would explore next

• A gray-failure probe: withdraw a node whose p95 climbs, not only one that fails. dnsdist can do it with a latency-aware policy; anycast needs it in the health script.
• The relay’s WAN-down and frozen-box rows, which need the production AdGuard’s network to be cut from outside its pod — a NetworkPolicy in the scenario runner.
• Many clients, many timeouts. A second client on Wi-Fi with an Apple-style resolver (parallel, sub-second) and a Linux one (5 s) would show how much of the baseline’s pain is the 2-second constant and how anycast’s per-flow ECMP spreads a node failure over a third of the house.
• TCP and large answers through anycast during a flow move (the ECMP rehash breaks TCP sessions; DNS over TCP is rare on a LAN, DNSSEC-sized answers aren’t).
• AdGuard’s blocklists in unbound (RPZ), to see whether the resolver comparison survives once unbound is also doing the filtering.
• DNS that follows the device — the DoT section above — measured, not argued.
• And the one this lab cannot do: three real boxes, one of them unplugged.

If you want to rebuild this

The parts, in the order that gave me the most per hour:

1. Serve-stale everywhere. cache_optimistic: true in AdGuard Home; in dnsdist a packet cache plus setStaleCacheEntriesTTL(600). Free.
2. One address in DHCP. Whatever it points at, hand out one server. Put your failover behind it, not beside it.
3. A relay if you want the smallest change: dnsdist on any always-on Linux box, firstAvailable over your resolvers with a real-lookup health check every 1–2 s, the router last. Twenty lines; the config is in the repo.
4. Anycast if you want DNS that doesn’t care where the resolvers are: FRR (bgpd + bfdd) on each node, the service address on a dummy interface, redistribute connected through a prefix-list so the announcement follows the link, a health script that takes the link down when that node stops resolving an uncacheable local name. On the router: accept only that /32, BFD on, and a netwatch that redirects the address to the router’s own resolver while nobody announces it. Everything — provisioning script, router config, benchmark, scenario runner, all the results — is in dr-blue-green/infra/dns-anycast/ on my GitLab.
5. Three boxes. For a node whose whole job is answering DNS and speaking BGP, the spec is: arm64, 1–2 GB RAM, wired gigabit, mainline-kernel Debian support, ideally eMMC and a metal case, under €50, low power. That points at the NanoPi R3S LTS (RK3566, 2 GB, two gigabit ports, eMMC, metal case, “platinum” Armbian support) rather than the cheaper Orange Pi Zero 3 (H618, community-level Armbian support, Wi-Fi you don’t need, SD card only). Three R3S plus PSUs is roughly the price of one used enterprise switch and draws ~2 W each; the two ports let one of them sit in front of something else later. A Pi 4/5 works too and costs more for nothing this job uses. Avoid anything that boots only from an SD card written by a logging appliance — see the water meter.
6. Sync from one source of truth (adguardhome-sync, five minutes) and one log stream for all instances, or the “which one answered?” question will eat your evenings.

What runs at home now, and what comes next

The relay carried the LAN through the test days; the anycast on the Orange Pi is becoming the main path — DHCP will point at 10.1.9.53, the relay steps back to being the thing a worker cluster is reached through. Three containers today, three boxes when they arrive. cache_optimistic and no rate limit on every AdGuard, one configuration source, all query logs in one place. The measurements above are what the switch rests on.

If you take one thing from this: a second DNS server in DHCP is not failover, it is a two-second tax on every query for as long as the first one is down. Everything else here is about not paying it.