# provael.com
#
# CRAWLER POLICY — a decision, not a default (3 Aug 2026).
# This site is deliberately machine-readable: /llms.txt, /llms-full.txt, Markdown twins of
# its pages, and a machine-readable regulatory clock exist so that search and answer engines
# can quote Provael accurately. Being cited is the point. Therefore:
#   search=yes    build indexes, return links and excerpts
#   ai-input=yes  RAG / grounding / answer-engine use is EXPLICITLY welcome
#   ai-train=yes  this file has always allowed the training crawlers (GPTBot, ClaudeBot,
#                 Google-Extended, CCBot, …) and now says so explicitly; flip this one
#                 signal if the operator's preference changes — it is one line.
# HISTORY: until 3 Aug 2026 Cloudflare's managed robots.txt sat ABOVE this file and
# disallowed ClaudeBot, GPTBot and Google-Extended (with ai-train=no), contradicting the
# Allow below — and the per-bot rules won. If a "# BEGIN Cloudflare Managed content" block
# ever appears above this line again, the zone's AI Crawl Control managed-robots.txt setting
# has been re-enabled: turn it off, and check `npm run check:live`, which exists to catch
# exactly this.
# A machine-readable `Content-Signal:` line is deliberately NOT emitted: the content-signals
# spec treats an absent signal as "neither grants nor restricts", which alongside the Allow
# below matches the policy above — and validators (Lighthouse's robots.txt audit among them)
# still reject the directive as unknown syntax. Add the line when the validators catch up.
User-agent: *
Allow: /

# Both are declared on purpose. sitemap-index.xml is canonical; /sitemap.xml is a byte-copy
# alias that exists because crawlers try that path first (see astro.config.mjs), and an
# undeclared alias is a crawl path nothing points at.
Sitemap: https://www.provael.com/sitemap-index.xml
Sitemap: https://www.provael.com/sitemap.xml

# Machine-readable summaries for agents:
#   https://www.provael.com/llms.txt
#   https://www.provael.com/llms-full.txt
#
# Feeds. They were undeclared here until 14 Aug 2026 while five research posts sat behind them —
# the same "a crawl path nothing points at" problem the /sitemap.xml alias above exists to solve.
# They ARE declared as <link rel="alternate"> on every page, but a crawler that reads robots.txt
# before fetching any HTML had no way to find them.
#
# /studies/rss.xml was added on 26 Aug 2026: it had been building and serving since it shipped,
# listed in neither this block, the sitemap's customPages, nor the rel=alternate set — so the one
# feed aimed at a standards-body reader was the one no crawler could discover. Every feed the build
# emits belongs in all three places; if you add a fifth, add it to all three.
#   https://www.provael.com/rss.xml            combined: notes + findings + incidents
#   https://www.provael.com/feed.xml           byte-identical alias of the above
#   https://www.provael.com/notes/rss.xml      research notes only
#   https://www.provael.com/findings/rss.xml   measured findings only
#   https://www.provael.com/incidents/rss.xml  embodied-AI security incident tracker
#   https://www.provael.com/studies/rss.xml    pre-registered studies