DNS Operations (Pi-hole + Unbound)

Architecture (HA)

                    ┌─ Pi-hole (pi-k3s:53) ──── Unbound (pi-k3s:5335) ──── Root Servers
User Device ───┤
                    └─ Pi-hole-secondary (pi5-worker-1:53) ──── Unbound-secondary (pi5-worker-1:5335) ──── Root Servers

Both paths are independent. Clients may use either Pi-hole instance via DHCP-assigned DNS.

Why Unbound? Full recursive resolution directly to authoritative DNS servers. Better privacy (no single upstream sees all queries), no third-party trust required, DNSSEC validation.

Service Details

Pi-hole (Primary + Secondary)

• Image: pihole/pihole:latest with hostNetwork: true
• Primary: Port 53 on pi-k3s (192.168.1.55)
• Secondary: Port 53 on pi5-worker-1 (192.168.1.56)
• Password: Synced from 1Password via ExternalSecret
• Upstream DNS: Respective Unbound ClusterIP service
• Adlists: Firebog curated lists via ConfigMap (~900k domains)
• DNSSEC: Disabled on Pi-hole (handled by Unbound)
• Caching: Pi-hole maintains its own DNS cache

Unbound (Primary + Secondary)

• Image: madnuttah/unbound:latest (distroless, minimal — no cat/ls/head)
• Port: 5335 (non-privileged) via ClusterIP service
• Type: Recursive resolver with DNSSEC validation
• Node Placement: Pi 5 nodes ONLY via nodeSelector (Pi 3 hardware causes TCP failures)
• Performance: ~21ms uncached, 0-15ms cached

Upstream transport — do NOT set tcp-upstream: yes. It is not a "fall back to TCP if UDP fails" toggle (that fallback is automatic on truncated UDP) — it forces all upstream recursion over TCP, always. We ran it Dec 2025 → Jun 2026 under that misconception (added in 17b857f as generic "home-network resilience"; the AT&T/IPv6 slowness was a separate gateway-side fix — UniFi IPv6 prefix delegation, nothing to do with DNS transport). It caused recurring DNSMASQ_WARN "max concurrent queries (150)" on both Pi-holes at the same instant, plus CONNECTION_ERROR premature TCP drops to Unbound (the Jan 2026 TCP-pool bump 4ed431f only masked the symptom). Removed 2026-06-12. UDP-first + automatic TCP fallback + the default edns-buffer-size: 1232 is the correct fragmentation-safe setup. If UDP ever is proven broken, fix EDNS buffer sizing — don't force TCP.

Configuration

Pi-hole v6 API (Critical)

Pi-hole v6 ignores most environment variables. Configuration is done via REST API in postStart hook:

• POST /api/auth - Get session ID
• PATCH /api/config - Set upstream DNS to Unbound ClusterIP
• POST /api/lists - Add adlists from ConfigMap (batch format)
• POST /api/action/gravity - Update gravity database

Reference: docs/pihole-v6-api.md

madnuttah/unbound Config Quirks (CRITICAL)

The madnuttah/unbound image rewrites the main unbound.conf during its entrypoint. It transforms paths and drops directives it doesn't recognize.

• Compiled config path: /usr/local/unbound/unbound.conf (NOT /opt/unbound/etc/unbound/unbound.conf)
• Our mount: /opt/unbound/etc/unbound/unbound.conf → entrypoint processes this into the actual config
• Dropped directives: domain-insecure and other less common server directives are silently dropped
• conf.d directory: /usr/local/unbound/conf.d/*.conf — mount custom server directives HERE as separate files
• Verify running config: kubectl exec -n pihole deploy/unbound -- unbound-checkconf -o <directive>

To add custom server directives (e.g., domain-insecure):

1. Add the directive to the ConfigMap as a separate data key (e.g., dnssec-exceptions.conf)
2. Mount it at /usr/local/unbound/conf.d/<name>.conf using subPath
3. Do NOT put it in the main unbound.conf — it will be silently dropped

DNSSEC Validation

Unbound validates DNSSEC via:

• auto-trust-anchor-file — root trust anchor
• harden-dnssec-stripped: yes — strict mode, rejects responses that strip DNSSEC

When a domain has broken DNSSEC (DS records published but no valid DNSKEY):

• Unbound returns SERVFAIL
• Unbound logs show: validation failure <domain>: no keys have a DS with algorithm RSASHA256
• Fix: Add domain-insecure: "domain.com" via conf.d mount (see above)

Caching Behavior (CRITICAL for Diagnostics)

Unbound serve-expired configuration:

• serve-expired: yes with serve-expired-ttl: 86400 (24 hours)
• Unbound serves stale cache immediately while refreshing in background
• This means a broken upstream can be masked by stale cache for up to 24 hours

Pi-hole also caches results independently.

Combined effect: test_dns_query (which runs dig inside Pi-hole) can return a successful cached result even when Unbound is actively returning SERVFAIL. A successful test_dns_query does NOT prove the resolution path is healthy. Always use diagnose_dns for troubleshooting.

DNS Resilience

The Pi node uses static DNS (1.1.1.1, 8.8.8.8) configured via NetworkManager.

• Why: Ensures the Pi can pull images (like Pi-hole itself) even if the cluster DNS is down.

IPv6 Blocking (AT&T Routing Issues)

AT&T Fiber has poor IPv6 routing to some CDNs. We selectively block IPv6 for affected domains.

Current blocked domains: See clusters/pi-k3s/pihole/pihole-custom-dns.yaml

IMPORTANT: Test Before Blocking

When a user reports slow/broken connectivity to a service, DO NOT immediately add it to the IPv6 block list. First verify IPv6 is the cause:

# 1. Check if domain returns AAAA records (if no AAAA, IPv6 isn't the issue)
dig AAAA <domain>

# 2. Compare IPv4 vs IPv6 response times from a device on the network
curl -4 -w "IPv4: %{time_total}s\n" -o /dev/null -s https://<domain>
curl -6 -w "IPv6: %{time_total}s\n" -o /dev/null -s https://<domain>

# 3. If IPv6 is significantly slower (2x+) or times out, add to block list

Only add to the block list after confirming IPv6 is the problem. See docs/known-issues.md for details.

DNS Troubleshooting Runbook (MANDATORY)

When a user reports a domain is unreachable, follow ALL steps in order. Do NOT stop after step 1 even if it shows success. Do NOT blame the client's browser or machine until the entire server path is proven clean.

Step 1: Run diagnose_dns MCP Tool (ALWAYS START HERE)

Use diagnose_dns with the reported domain. This single tool tests:

• Pi-hole resolution (may be cached)
• Unbound primary direct resolution (bypasses cache)
• Unbound secondary direct resolution (bypasses cache)
• DNSSEC validation check (if Unbound fails, retries with +cd)
• Unbound pod logs filtered for the domain

If diagnose_dns is unavailable, manually run all substeps:

test_dns_query → check Unbound logs (BOTH pods) → dig @unbound directly

Step 2: Interpret Results

Pi-hole	Unbound Primary	Unbound Secondary	Diagnosis
OK	OK	OK	Resolution path is healthy. Issue is client-side.
OK	FAIL	FAIL	Stale cache masking upstream failure. Check Unbound logs immediately.
OK	OK	FAIL	Secondary Unbound is broken. Client may be using secondary.
FAIL	FAIL	FAIL	Complete DNS failure. Check pod health, network connectivity.
OK (cached)	FAIL + resolves with +cd	FAIL + resolves with +cd	DNSSEC validation failure. Domain has broken DNSSEC.

Step 3: Check Unbound Logs (NEVER SKIP)

Use get_pod_logs for BOTH Unbound pods:

• namespace: pihole, pod: unbound
• namespace: pihole, pod: unbound-secondary

Look for:

• validation failure — DNSSEC issue
• SERVFAIL — upstream failure
• connection timed out — network issue
• TCP connection failed — TCP upstream issue (common on Pi 3)

Step 4: Apply Fix Based on Root Cause

Root Cause	Fix
DNSSEC validation failure	Add domain-insecure via conf.d ConfigMap mount
Unbound timeout/crash	Restart Unbound deployment, check node health
Pi-hole not forwarding	Check Pi-hole upstream DNS config via API
Network issue	Check node connectivity to root servers

Common Issues

1. "Refused": Check if Pi-hole is running and hostNetwork is true.
2. "ServFail": ALWAYS check Unbound logs. Often DNSSEC failure, upstream timeout, or TCP failure on Pi 3 nodes.
3. Adlists not loading: Check postStart hook logs in Pi-hole pod.
4. test_dns_query shows success but client fails: Stale cache. Use diagnose_dns instead.

Pi-hole Custom DNS — GitOps Managed (CRITICAL)

All local DNS records live in clusters/pi-k3s/pihole/pihole-custom-dns.yaml (ConfigMap). This ConfigMap is mounted by both pihole and pihole-secondary; Flux applies it to both instances simultaneously.

Current layout:

• Wildcard *.lab.mtgibbs.dev → 192.168.1.55 (ingress controller)
• Specific overrides below the wildcard (dnsmasq is most-specific-match-wins)

DO NOT add records via the Pi-hole web UI for anything cluster-managed — they will drift from GitOps and will not survive a pod restart. Always edit the ConfigMap and let Flux reconcile.

pi-k3s Master: Static DNS Fallback (CRITICAL — intentional design, DO NOT "fix")

The pi-k3s master node is configured with public DNS (1.1.1.1, 8.8.8.8) via NetworkManager. Worker nodes use Pi-hole.

Why: Bootstrap resilience. Allows pi-k3s to pull container images (including Pi-hole itself) even when the cluster DNS is down. This is intentional. Do not change it.

Side effect: pi-k3s does not resolve Pi-hole local overrides. Pods scheduled on pi-k3s that need a *.lab.mtgibbs.dev hostname get the public wildcard (192.168.1.55) instead of any Pi-hole-specific override. kubelet uses the host's resolv.conf, not CoreDNS.

Workaround for pi-k3s-local DNS names: Add /etc/hosts entries on the pi-k3s host for any local-only name that pods on that node need. These are managed under Flux at clusters/pi-k3s/coredns-custom/pi-k3s-hosts-overrides.yaml (a ConfigMap mounted by a DaemonSet or postStart hook — verify current implementation in the manifest).

Currently overridden: storage.lab.mtgibbs.dev → 192.168.1.61.

Full notes in memory/pi-k3s-dns-fallback.md.

CoreDNS lab.mtgibbs.dev Forwarding (CRITICAL — permanent infrastructure)

CoreDNS default config forwards ALL queries (including lab.mtgibbs.dev) to 1.1.1.1/8.8.8.8. The lab.mtgibbs.dev zone has a public wildcard record — so without a local override, all *.lab.mtgibbs.dev in-cluster DNS resolves to the public wildcard answer rather than the Pi-hole local records.

Fix: A lab.mtgibbs.dev.server block in the coredns-custom ConfigMap forwards local-domain queries to Pi-hole. This is under Flux at clusters/pi-k3s/coredns-custom/. Do NOT remove or edit it via kubectl.

If this block is missing, all *.lab.mtgibbs.dev in-cluster DNS breaks silently.

Full notes in memory/coredns-forward-gotcha.md.

Relevant Files

• clusters/pi-k3s/pihole/pihole-deployment.yaml — Primary Pi-hole
• clusters/pi-k3s/pihole/pihole-secondary-deployment.yaml — Secondary Pi-hole
• clusters/pi-k3s/pihole/unbound-deployment.yaml — Primary Unbound + Service
• clusters/pi-k3s/pihole/unbound-secondary-deployment.yaml — Secondary Unbound + Service
• clusters/pi-k3s/pihole/unbound-configmap.yaml — Unbound config (shared by both, includes conf.d entries)
• clusters/pi-k3s/pihole/pihole-custom-dns.yaml — Local DNS + IPv6 overrides
• clusters/pi-k3s/coredns-custom/ — CoreDNS lab.mtgibbs.dev forwarding + pi-k3s host overrides