Public Subdomain Inventory and Cleanup Runbook

A public hostname is production inventory even when nobody remembers creating it. DNS, certificates, hosting projects, reverse-proxy routes, and search results can keep a name visible long after the experiment behind it has ended.

The useful outcome of an inventory is not a large list. It is one accountable decision for every hostname: keep, protect, redirect, or retire. This runbook turns those decisions into observable DNS, HTTP, TLS, and search states.

The inventory is the deliverable

Use one row per hostname and keep these fields together:

Field	What to record
Hostname	The exact public name, including every label
Purpose	What a visitor or client is supposed to find there
Owner	A person or team that can approve changes
DNS	Record type, target, proxy mode, and authoritative provider
Application	Hosting project, origin, route, or service binding
HTTP/TLS	Current status, redirect target, and certificate coverage
Search state	Indexable, noindex, authenticated, removed, or unknown
Desired state	Keep, protect, redirect, or retire
Deadline	A cleanup date or the next ownership review

An entry without an owner is already a cleanup candidate. An entry without a purpose should not remain public merely because its DNS still resolves.

If the hostname itself is unclear, use the public subdomain naming worksheet before deciding that it deserves a permanent place in the namespace.

1. Start from systems of record

Search results are useful evidence, but they are not a complete hostname database. Build the first pass from systems you control:

1. Export the authoritative DNS zone, including delegated NS records and wildcards.
2. List custom domains in each CDN, edge, static-hosting, and application platform.
3. Inspect reverse-proxy and ingress configuration for server_name, host rules, and certificate bindings.
4. Search infrastructure-as-code and deployment configuration for the registered domain.
5. Review certificate inventory for names that may no longer appear in the main DNS dashboard.
6. Compare recent request logs and Search Console data with the resulting list.

Treat a wildcard as one managed routing rule, not as proof that every possible name is a real project. A wildcard creates an effectively unbounded set of DNS answers, so its inventory row needs an application owner, an allowlist policy, and a negative-host test. The wildcard DNS and TLS guide covers that control boundary in detail.

Search queries can reveal leftovers that the operational inventory missed:

site:example.com
site:example.com inurl:stage
site:example.com "hello world"
site:example.com "lorem ipsum"

Use these as discovery hints, then verify every result against the actual DNS and HTTP configuration.

2. Probe DNS, HTTP, and TLS separately

A single browser visit compresses several systems into one impression. Record them separately so the cleanup owner knows where the failure lives.

dig stage.example.com A +short
dig stage.example.com AAAA +short
dig stage.example.com CNAME +short
curl -sS -o /dev/null \
  --connect-timeout 5 \
  -w '%{http_code} %{url_effective}\n' \
  https://stage.example.com/

For a certificate check, send the intended hostname with SNI instead of inspecting only the server IP:

openssl s_client \
  -connect stage.example.com:443 \
  -servername stage.example.com \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Capture at least these outcomes:

• NXDOMAIN: the name does not exist in DNS.
• NOERROR with no requested answer: the name exists but has no record of that type.
• 2xx: inspect the body; a template or error message can still be a soft 404.
• 3xx: confirm the final destination is the closest relevant replacement.
• 4xx: distinguish intentional access control from a retired URL.
• 5xx: treat it as an availability defect, not as a removal method.
• TLS name mismatch or expiry: fix certificate routing before calling the host healthy.

Illustrative cleanup record

The following image demonstrates the shape of a status-and-decision record. It is not output captured from a live environment.

Illustrative example using reserved example domains and IPs.

3. Choose the smallest honest public state

Use the desired user experience to choose the response. Do not use a server error or an unrelated homepage redirect as a substitute for a decision.

Desired state	DNS and HTTP behavior	Search treatment
Keep public	Valid DNS, TLS, and meaningful 200 content	Indexable if it belongs in search; include the canonical URL in a sitemap when it is core content
Keep public but out of search	Valid 200 page	Send a crawlable noindex directive and remove it from the sitemap
Keep private	Valid DNS and TLS with real authentication or network access control	Do not expose a public placeholder page; authentication is stronger than relying on robots.txt
Move	Permanent redirect to the closest equivalent URL	Remove the old URL from the sitemap and keep the redirect long enough to be recrawled
Retire	404 Not Found or 410 Gone	Remove internal links and sitemap entries; allow crawlers to observe the final status

Google documents that a persistent 5xx response initially preserves already indexed URLs while reducing crawl activity. That makes 500, 502, and 503 poor cleanup states. A removed resource should return a stable 404 or 410, not an application crash.

Likewise, robots.txt manages crawling, not guaranteed removal. If a URL is disallowed, a crawler cannot read the page’s noindex directive. For a crawlable public page that should disappear from search, expose noindex until it is processed; for content that is truly gone, return 404 or 410.

4. Apply cleanup in an order that can be verified

For each hostname or URL set:

1. Freeze new publishing. Stop deployments and automated preview creation while the desired state is being established.
2. Remove discovery paths. Delete obsolete internal links, navigation entries, feeds, and sitemap URLs.
3. Make the final response real. Add authentication, a relevant redirect, noindex, 404, or 410 according to the decision table.
4. Let crawlers see that state. Remove any robots.txt rule that would hide a required noindex, redirect, 404, or 410 response.
5. Fix platform bindings. Remove stale custom-domain assignments, routes, certificates, and origins after the public transition is safe.
6. Remove DNS last when appropriate. If an indexed host needs to communicate a 410, keep enough routing in place for crawlers to receive it before deleting the record. For a security incident, prioritize access removal and use Search Console for faster suppression.
7. Re-probe from outside the deployment network. Confirm DNS, TLS, status, final URL, body, and headers.

The Search Console Removals tool can temporarily hide an urgent URL, but Google describes it as a temporary measure. Pair it with a permanent server-side state; otherwise the URL can return after the temporary block expires.

5. Rebuild the sitemap from the keep list

A sitemap should contain canonical URLs that you actually want in search. It is not a historical ledger and it should not include staging hosts, request forms, admin utilities, redirects, or retired pages.

For every submitted URL, verify:

• the URL is absolute and canonical
• it returns a meaningful 200
• it is not marked noindex
• it is internally reachable from the intended site structure
• any lastmod value reflects a real, significant update

Submitting a sitemap is a discovery hint, not a guarantee of indexing. The keep/protect/redirect/retire state still has to be correct when the crawler arrives.

6. Close the runbook with evidence

Cleanup is complete when the inventory can be re-run and produces the intended result, not when a DNS record disappears from a dashboard.

Use this closure checklist:

• every hostname has an owner and purpose, or a completed retirement decision
• intended public hosts return the correct application over valid TLS
• random wildcard hosts fail closed instead of returning a generic 200 or 5xx
• private hosts require authentication
• moved URLs redirect to a relevant replacement without a chain
• retired URLs return 404, 410, or NXDOMAIN according to the recorded plan
• robots.txt, page directives, internal links, and sitemaps agree
• urgent temporary removals also have a permanent cleanup state
• the next inventory date and owner are recorded

Run the inventory after migrations, hosting-provider changes, wildcard launches, and preview-system changes. For a stable namespace, a scheduled quarterly check is usually cheaper than reconstructing ownership after an abandoned host becomes visible again.