What air-gapped actually means: every phone-home in the stack, audited

We sell a box that's supposed to never talk to the internet. Compliance pictures a machine with no cable in it, then asks: how do you know your own software doesn't try anyway?

I couldn't answer that from memory, so we counted. Ten components in the customer bundle could plausibly call out at startup or during a query: three came clean, most needed one environment variable, two needed code, one of which is still sitting in the tree as I write this. Then we took a real DGX Spark, cut its internet with iptables, and ran a retrieval-augmented query while tcpdump watched the uplink.

The ten things that wanted to call home

The audit lives in docs/design/dgx-spark/AIRGAP_PHONE_HOME_AUDIT.md, deliberately boring. The summary:

Number one is the sleeper. HuggingFace libraries treat a cache as a starting point, not an answer: even with a model on disk, it checks the hub for a newer revision through requests, silently, and on a box with a route out succeeds unnoticed for a year.

Number three hurts operationally. Seven files default an OpenAI-compatible base URL to DeepInfra, from agent_api/app/config.py:182 through rag_service/rag_config.py:54 to two fallbacks in rag_service/lightrag/operate.py. Without a base URL set, the client tries DeepInfra and fails on an air-gapped host after a DNS timeout plus a TCP reset, hanging the request 30 seconds or more: still air-gapped since no data leaves, but unusable.

The fix is a block of environment variables the operator copies verbatim:

# ── AIRGAP HARDENING (do not weaken without explicit reason) ─────────────
OPENAI_BASE_URL=http://vllm_llm:8000/v1
LLM_BASE_URL=http://vllm_llm:8000/v1
DEFAULT_LLM_BASE_URL=http://vllm_llm:8000/v1
EMBEDDING_BASE_URL=http://vllm_embedding:8000/v1
RERANK_BASE_URL=http://vllm_reranker:8000/v1/rerank
# OpenAI-compatible clients require a non-empty key even for unauthenticated
# local endpoints. Any non-empty placeholder works; SGLang ignores it.
OPENAI_API_KEY=local-airgap-no-auth
DEEPINFRA_API_KEY=local-airgap-no-auth

USE_LITELLM=false

HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1
HF_DATASETS_OFFLINE=1
DO_NOT_TRACK=1

USAGE_EVENT_ENABLED=false
ANALYTICS_ENABLED=false

Those dummy API keys are load-bearing: the OpenAI client refuses an empty key, even for an unauthenticated endpoint, so a blank value takes the stack down before reaching the local model.

A customer engineer might edit a .env at 2am, so the offline variables live in the compose file too, through an x-airgap-env anchor every model container inherits: a stripped .env can break the LLM base URL, not make the containers probe HuggingFace.

The gaps config could not close

The first is rag_service/lightrag/rerank.py:358: the DeepInfra reranker URL is a hardcoded default argument. I checked again while writing this, and it's still there:

base_url: str = "https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Reranker-4B",

DeepInfra's reranker isn't on an OpenAI-shaped path, so OPENAI_BASE_URL doesn't redirect it. Workarounds: disable reranking on the customer knowledgebase, or set RERANK_BASE_URL, now the bundle's default. The parameterised fix hasn't landed.

The second is the LiteLLM startup probe in agent_api/app/main.py: with no licence loaded, or one without UNMETERED, agent_api probes LiteLLM's readiness endpoint for 150 seconds. With no LiteLLM container on the customer stack, DNS fails in about a second, wasting startup time and log space. USE_LITELLM=false short-circuits it, verified; gating the probe on DNS resolution is the real fix, still queued.

The third isn't ours to fix: a box on a corporate LAN with a route to the internet can still leak if some library we didn't audit talks, and the bundle can't enforce network policy. The options: no egress route on the NIC, an OUTPUT policy dropping everything except the internal LAN, or --ip-masq=false behind a per-network egress firewall.

Proving it, twice

On 23 June 2026 I sat down with a GX10 DGX Spark running the customer stack, Qwen3.6-35B served locally, no DeepInfra in the runtime path, and wrote the two scripts now in customer-bundle/airgap/.

airgap-prove.sh runs a real query through the full pipeline while tcpdump watches the uplink. Everything the BPF filter excludes is a place a false negative could hide:

FILTER='ip and not dst net 10.0.0.0/8 and not dst net 172.16.0.0/12
        and not dst net 192.168.0.0/16 and not dst net 100.64.0.0/10
        and not dst net 169.254.0.0/16 and not dst net 224.0.0.0/4
        and not dst host 255.255.255.255'

With loopback, RFC1918, the CGNAT tailnet range, link-local, multicast and broadcast excluded, anything left in the capture went to the internet during the query. It probes five external endpoints first (huggingface.co, api.deepinfra.com, api.openai.com, api.anthropic.com and 1.1.1.1 on port 443), then checks the container logs to confirm the local model answered.

Level 1, internet plugged in. All five endpoints report reachable, but the query answers anyway, served locally by Qwen3.6, one hit each on the LLM and embedding containers. Certant-attributable external packets: zero.

Level 2, the wire cut. airgap-lockdown.sh --on installs an iptables chain that allows loopback, established connections and every private range, then drops everything else outbound. Re-run the prover and all five endpoints report blocked, 1.1.1.1 included. The same query still answered, in 3 seconds, with citations from the WaterRA knowledgebase. External Certant packets: still zero.

Level 1 is the measurement I care about: a machine with no route out can't demonstrate restraint.

Those surfaces get a walkthrough in the sovereign product video at docs/videos/sovereign_2026-07-20/sovereign_cut_final_captioned_2026-07-20.mp4 (86 MB, linked rather than copied here): data model, documents, a cited answer, governed SQL, then residency controls.

Three things that went wrong

The lockdown chain returns early on ESTABLISHED,RELATED, so it kept both a demo SSH session and my Tailscale session alive: Tailscale's NAT-traversal channel stays warm via STUN on UDP 3478 and DERP on TCP 443, both public IPs landing in the capture.

I taught the prover to classify them: it pulls tailscaled's current IPs from ss -tunp, adds UDP 3478 and Tailscale's control netblock, and reports the split:

EXTERNAL PACKETS: <total>  ·  tailscale-mgmt: <n>  ·  Certant: 0

With no Tailscale, both numbers hit zero. Driven remotely, Certant stays 0 and the rest is a management link you can name.

Cutting a box's network over the same network you're driving it from risks losing the box, so the lockdown installs a deadman: a detached setsid sleeper that runs --off after 300 seconds by default. For remote runs I stacked three: an independent sleeper, the lockdown's --ttl 150, and an explicit --off at the driver's end.

Two runs less than 240 seconds apart produced the third: run one's backup sleeper fired mid-way through run two's probe, flushed the chain, and showed the endpoints reachable during a lockdown test, a false negative from my own safety net. I now kill stale sleep processes before every re-run.

The 26 GB on a USB stick

The delivery model is a USB drive: extract, run ./install.sh, upload a licence file, and the platform is up, no GHCR pull, no ansible controller. The previous customer build was amd64-only and shipped only 3 of the roughly 9 images a working system needs.

The current arm64 artifact, customer-bundle-dgx-qwen36-rc1-arm64, is 26.22 GB compressed, from a CI run that completed 14 of 14 jobs. Our target was 18 GB or under, to fit a 32 GB stick with headroom for weights; it overshoots by 8 GB.

Model weights aren't in it: putting them in would take the bundle from roughly 18 GB to around 80 GB. They ship separately, as a models-${VER}-${ARCH}.tar.zst of roughly 60 GB, built from customer-bundle/models.manifest.yaml: Qwen3.6-35B in NVFP4, its DFlash draft model, Qwen3-Embedding-4B, Qwen3-Reranker-0.6B, GLM-OCR, PP-DocLayoutV3 and gpt-oss-120b. install.sh extracts them into the HuggingFace cache if the archive is there, and carries on without it if not: vLLM fails at the first request and the operator knows they forgot a USB stick.

preflight.sh runs first and refuses to continue on a failure: architecture against the bundle name, a reachable Docker daemon, the nvidia runtime registered, nvidia-smi returning a GPU, 80 GB free on the Docker data root, system RAM (warn under 64 GB, refuse under 32 GB), the CLI tools it needs, and ports 30001, 38003, 35555 and 38100-38102 free. It also curls a public URL, telling the operator which side of the airgap they're on.

Then install.sh verifies the SHA-256 manifest, copies the env template if there's no .env yet and stops for a human to edit it, refuses to continue while any REPLACE_ME remains, loads the images, brings up agent_api, polls /api/license/machine-id, and prints the machine ID to send back to us.

Licensing a machine that cannot phone anyone

We issue a signed JWT bound to that machine ID, the customer uploads the .license file through the admin UI, and the validator checks it locally against a public key that ships in the bundle.

By default the validator refreshes a blacklist daily, and past some grace period treats the licence as invalid, buying time during a managed outage but a slow failure for a customer with no network. An airgap enforcement mode in the JWT skips the remote check and honours only the local expiry, so the runbook is one line: always pass --mode airgap to the licence generator. Issue it in the legacy disabled mode and the daily probe still runs, and still silently fails.

The grace period is documented as observed, never quantified: measuring it needs a 30-day run with the licence server blocked at the firewall, which hasn't happened, so I can't give the number. There's no remote kill switch either, so we ring the customer to revoke an air-gapped licence, which is why we refuse perpetual ones: expiry is required, one year by default, and renewal is the same machine-ID round trip as the original issue.

The checklist someone else can run

Ship software into a place with no network, and this is the shape of the exercise. Ours runs before every release.

1. grep -rE 'api\.(deepinfra|openai|anthropic|huggingface|posthog|segment|sentry)\.' agent_api/ rag_service/ modules/; every hit must be a default .env overrides, or a path that no-ops in a customer build.
2. Diff dependencies since the last audit for new clients (openai, anthropic, posthog, sentry, etc.); add anything new to the audit document with its mitigation.
3. Confirm the .env template still carries the full hardening block and the compose anchor is attached to every model container.
4. Confirm the sandbox network is still declared internal: true.
5. Confirm the licence was issued in airgap mode, not the legacy default.
6. Packet-capture a clean install and a first query; require zero outbound packets to non-private addresses.

Step 6 catches what the other five miss: it's the only one that would have caught the HuggingFace revision check, since nothing in the logs mentions it. Run it with the internet plugged in first.