How do I check a website's llms.txt file and its robots.txt rules for AI crawlers?

Fairquery · October 18, 2026 · 10 min read

What llms.txt is (and what it isn't)

llms.txt is a proposal by Jeremy Howard (first published September 2024, revised as v2 in August 2026). It suggests a Markdown file at /llms.txt that gives AI agents a short summary of a site and a curated list of links to LLM-friendly pages. v2 also allows a file at a subpath (for example /docs/llms.txt covers /docs/) and recommends rel="alternate" type="text/markdown" and rel="describedby" links so agents can find Markdown versions and the covering llms.txt.

Set your expectations honestly:

• It is a community proposal, not a standard from the IETF or W3C.
• Google Search ignores it. Google's AI optimization guide says Search doesn't use llms.txt files, and that having one won't help or hurt rankings (reported June 2026).
• Chrome's Lighthouse added an experimental llms.txt audit in its Agentic Browsing category (version 13.3), which checks that the file exists and loads without a server error.
• Real traffic is small. An Ahrefs study of 137,000 sites found that 97% of published llms.txt files got no requests at all in May 2026, and that most fetches that did happen came from bots such as GPTBot and coding agents.

So treat llms.txt as a cheap, low-risk aid for coding agents and AI assistants that read your docs, not as an SEO lever. If you publish one, it should at least be valid and point to working pages.

The format in one example

# Project name

> One-paragraph summary with the key facts an agent needs.

Optional notes (paragraphs or lists, no headings).

## Docs

- [Quick start](https://example.com/docs/quickstart.md): install and first request
- [API reference](https://example.com/docs/api.md)

## Optional

- [Changelog](https://example.com/changelog.md)

Order matters: H1, then blockquote, then free text without headings, then H2 sections that each hold a Markdown list of links. The Optional section is a convention for links an agent can skip. Some sites also publish /llms-full.txt with the full text of their docs; that file is a common convention and isn't part of the llms.txt format.

Option 1 (free): check llms.txt and AI-bot rules with Python

Standard library only (Python 3.9+). It fetches /llms.txt, checks the structure against the format above, optionally sends a HEAD request to the first 25 links, looks for /llms-full.txt, and evaluates robots.txt for ten AI user agents:

#!/usr/bin/env python3
"""Check a site's /llms.txt against the llmstxt.org format and its robots.txt rules for AI bots.
Standard library only. Usage: python3 check_llms_txt.py example.com [--links]"""
import re, sys, urllib.request, urllib.robotparser
from urllib.parse import urljoin, urlparse

UA = "llms-txt-check/1.0 (+https://getfairquery.com/)"
AI_BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot",
           "Claude-User", "PerplexityBot", "Google-Extended", "Applebot-Extended", "CCBot"]
LINK_ITEM = re.compile(r"^\s*[-*+]\s+\[([^\]]+)\]\(([^)\s]+)\)(?:\s*:\s*(.*))?$")

def fetch(url, method="GET", limit=1_000_000):
    req = urllib.request.Request(url, method=method, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            body = r.read(limit).decode("utf-8", "replace") if method == "GET" else ""
            return r.status, r.headers.get("Content-Type", ""), r.geturl(), body
    except urllib.error.HTTPError as e:
        return e.code, e.headers.get("Content-Type", ""), url, ""
    except Exception as e:
        return None, "", url, str(e)

def check_llms(text):
    issues, lines = [], text.lstrip("\ufeff").splitlines()
    content = [l for l in lines if l.strip()]
    h1 = [l for l in lines if re.match(r"^#\s+\S", l)]
    if not content or not re.match(r"^#\s+\S", content[0]):
        issues.append("first line is not an H1 (# Name), the only required element")
    if len(h1) > 1:
        issues.append(f"{len(h1)} H1 headings (expected 1)")
    if len(content) < 2 or not content[1].startswith(">"):
        issues.append("no blockquote summary (> ...) right after the H1 (recommended)")
    sections, links, bad_items, current = [], [], 0, None
    for l in lines:
        if l.startswith("## "):
            current = l[3:].strip(); sections.append(current); continue
        if current and re.match(r"^\s*[-*+]\s", l):
            m = LINK_ITEM.match(l)
            if m: links.append(m.group(2))
            else: bad_items += 1
    if not sections:
        issues.append("no H2 sections with file lists")
    if bad_items:
        issues.append(f"{bad_items} list items in H2 sections are not '- [name](url): notes'")
    if re.search(r"^#{3,}\s", text, re.M):
        issues.append("H3+ headings found (the format uses only H1 and H2)")
    return {"title": h1[0][2:].strip() if h1 else None, "sections": sections,
            "links": links, "issues": issues}

def main(site, check_links=False):
    base = site if site.startswith("http") else "https://" + site
    origin = "{0.scheme}://{0.netloc}".format(urlparse(base))
    status, ctype, final, body = fetch(origin + "/llms.txt")
    print(f"llms.txt: HTTP {status}, {ctype or '-'}, {len(body.encode())} bytes, {final}")
    if status == 200 and "html" not in ctype.lower() and not body.lstrip().lower().startswith("<"):
        r = check_llms(body)
        print(f"  title: {r['title']}")
        print(f"  sections ({len(r['sections'])}): {', '.join(r['sections'][:8])}")
        print(f"  links: {len(r['links'])}, of which relative: {sum(not u.startswith('http') for u in r['links'])}")
        print("  issues: " + ("; ".join(r["issues"]) if r["issues"] else "none"))
        if check_links:
            broken = []
            for u in r["links"][:25]:
                s = fetch(urljoin(final, u), "HEAD")[0]
                if s is None or s >= 400: broken.append(f"{u} ({s})")
            print(f"  first {min(25, len(r['links']))} links checked, broken: {broken or 'none'}")
    else:
        print("  no valid llms.txt (missing, error or an HTML page)")
    s_full = fetch(origin + "/llms-full.txt", "HEAD")[0]
    print(f"llms-full.txt: HTTP {s_full}")
    rp = urllib.robotparser.RobotFileParser(origin + "/robots.txt")
    rs, _, _, rbody = fetch(origin + "/robots.txt")
    rp.parse(rbody.splitlines() if rs == 200 else [])
    print(f"robots.txt: HTTP {rs}")
    for bot in AI_BOTS:
        home, llms = rp.can_fetch(bot, origin + "/"), rp.can_fetch(bot, origin + "/llms.txt")
        print(f"  {bot:<18} /: {'allowed' if home else 'blocked'}   /llms.txt: {'allowed' if llms else 'blocked'}")

if __name__ == "__main__":
    main(sys.argv[1], "--links" in sys.argv)

Real output for our own site (October 18, 2026, python3 check_llms_txt.py fairquery.github.io --links):

llms.txt: HTTP 200, text/plain; charset=utf-8, 8355 bytes, https://fairquery.github.io/llms.txt
  title: Fairquery
  sections (5): Posts, Tools, Where to get them, About and security, Optional
  links: 35, of which relative: 0
  issues: 1 list items in H2 sections are not '- [name](url): notes'
  first 25 links checked, broken: ['https://www.npmjs.com/package/@fairquery/agent-trust-mcp (403)']
llms-full.txt: HTTP 200
robots.txt: HTTP 200
  GPTBot             /: allowed   /llms.txt: allowed
  OAI-SearchBot      /: allowed   /llms.txt: allowed
  ChatGPT-User       /: allowed   /llms.txt: allowed
  ClaudeBot          /: allowed   /llms.txt: allowed
  Claude-SearchBot   /: allowed   /llms.txt: allowed
  Claude-User        /: allowed   /llms.txt: allowed
  PerplexityBot      /: allowed   /llms.txt: allowed
  Google-Extended    /: allowed   /llms.txt: allowed
  Applebot-Extended  /: allowed   /llms.txt: allowed
  CCBot              /: allowed   /llms.txt: allowed

Two findings, both useful. First, our file had one non-link list item (- Contact: fairquery@proton.me) inside an H2 section; strict parsers skip such lines, so we turned it into a mailto: link in this update. Second, npm answered the link check with HTTP 403. That is bot protection rejecting a script, not a broken page, so treat 401/403/429 from big platforms as "check by hand" rather than "broken".

For comparison, the proposal's own site passes cleanly:

llms.txt: HTTP 200, text/plain; charset=utf-8, 637 bytes, https://llmstxt.org/llms.txt
  title: llms.txt
  sections (1): Docs
  links: 3, of which relative: 0
  issues: none
llms-full.txt: HTTP 404
robots.txt: HTTP 200
  GPTBot             /: allowed   /llms.txt: allowed
  OAI-SearchBot      /: allowed   /llms.txt: allowed
  ChatGPT-User       /: allowed   /llms.txt: allowed
  ClaudeBot          /: allowed   /llms.txt: allowed
  Claude-SearchBot   /: allowed   /llms.txt: allowed
  Claude-User        /: allowed   /llms.txt: allowed
  PerplexityBot      /: allowed   /llms.txt: allowed
  Google-Extended    /: allowed   /llms.txt: allowed
  Applebot-Extended  /: allowed   /llms.txt: allowed
  CCBot              /: allowed   /llms.txt: allowed

Large documentation sites often bend the format. Stripe's docs file (first lines of the output) is about 98 KB with 468 links, H3 subheadings and no blockquote summary:

llms.txt: HTTP 200, text/markdown; charset=utf-8, 97564 bytes, https://docs.stripe.com/llms.txt
  title: Stripe Documentation
  sections (27): Docs, Payment Methods, Checkout, Payments, Authorization Boost, Link, Architecture and Dashboard, Instructions for Large Language Model Agents: Best Practices for integrating Stripe
  links: 468, of which relative: 0
  issues: no blockquote summary (> ...) right after the H1 (recommended); 5 list items in H2 sections are not '- [name](url): notes'; H3+ headings found (the format uses only H1 and H2)
llms-full.txt: HTTP 404

That works fine for an LLM reading it, but a tool that parses the format strictly will miss the parts that deviate. A file that large also takes a meaningful share of an agent's context window, which is the problem llms.txt was meant to avoid; keep the file short and put detail behind the links.

Limits of the script: Python's urllib.robotparser applies the first matching rule rather than the longest match that RFC 9309 specifies, and it doesn't support * or $ wildcards in paths. It's fine for the common "block or allow the whole site per bot" case; for complex rule sets, test with a full RFC 9309 parser.

Which AI user agents to check in robots.txt

Blocking a training crawler and allowing a search or user-triggered agent is a common middle ground. Whatever you choose, make sure /llms.txt itself isn't blocked for the agents you want to read it. Operators change their crawler lists from time to time, so check their documentation before you rely on a name.

Option 2 (API or MCP): check many sites at once

The Fairquery Website Tech, SEO & llms.txt Checker (fairquery/website-tech-seo-ai-readiness) checks a list of URLs and returns one row per site: whether llms.txt and llms-full.txt exist (HTML soft-404 pages are not counted), the llms.txt title and link count, robots.txt status for 13 AI user agents, the sitemap, the detected tech stack, security headers, an SEO score and an AI-readiness score. Input:

{"urls": ["fairquery.github.io", "llmstxt.org"]}

Real output for our site, abridged to the AI-readiness fields (we ran the Actor's published source code locally on October 18, 2026):

{
  "source_url": "https://fairquery.github.io/",
  "checked_at": "2026-10-18T02:28:21.936Z",
  "status_code": 200,
  "robots_txt_found": true,
  "ai_bots": {
    "GPTBot": "allowed",
    "ChatGPT-User": "allowed",
    "OAI-SearchBot": "allowed",
    "ClaudeBot": "allowed",
    "Claude-User": "allowed",
    "anthropic-ai": "allowed",
    "PerplexityBot": "allowed",
    "Google-Extended": "allowed",
    "CCBot": "allowed",
    "Bytespider": "allowed",
    "Applebot-Extended": "allowed",
    "Meta-ExternalAgent": "allowed",
    "Amazonbot": "allowed"
  },
  "sitemap_found": true,
  "sitemap_url_count": 14,
  "llms_txt_found": true,
  "llms_txt_url": "https://fairquery.github.io/llms.txt",
  "llms_txt_title": "Fairquery",
  "llms_txt_links": 35,
  "llms_full_txt_found": true,
  "seo_score": 100,
  "ai_readiness_score": 100
}

llmstxt.org got an AI-readiness score of 80 in the same run (it has no llms-full.txt, among other things). The AI-readiness score is our own heuristic, not a Google or OpenAI metric: llms.txt counts 25 of 100 points, AI-bot access 20, JSON-LD 15, text content 15, sitemap 10, meta description 10 and llms-full.txt 5, with a penalty for noindex. It does not validate the llms.txt structure line by line the way the script above does; use the script for that.

Pricing: $0.0015 per site checked ($1.50 per 1,000 sites) plus Apify's $0.00005 Actor-start fee; failed fetches are free. AI agents can call it through the Apify MCP server (https://mcp.apify.com/?tools=fetch-actor-details,fairquery/website-tech-seo-ai-readiness) or through the stdio MCP server in the GitHub repo. To watch a list of sites over time, the free Fairquery n8n templates include an SEO and AI-crawler monitor.

Checklist for a good llms.txt

• Served at /llms.txt with HTTP 200 and a text/plain or text/markdown content type; a missing file should return 404, not an HTML page with status 200.
• H1 first, then a one-paragraph blockquote summary that states what the site or product is in plain words.
• H2 sections with - [name](url): notes items only; put contact details and other plain facts in the free-text part above the first H2.
• Links point to pages that load without JavaScript, ideally Markdown versions; absolute URLs avoid ambiguity.
• Short enough to fit easily into a context window; move secondary links to ## Optional.
• Not blocked in robots.txt for the agents you care about; there is no submission step.
• Kept in sync with the site: regenerate it in your build, as this blog does.

FAQ

How do I check if a website has an llms.txt file?

Request https://<domain>/llms.txt. A valid file returns HTTP 200 as text/plain or text/markdown and starts with an H1 line (# Name). An HTML page or a 404 means there is no usable file. The free Python script in this guide checks this, the structure, the links and robots.txt in one run.

What makes an llms.txt file valid?

Per the llmstxt.org proposal: an H1 with the site or project name (the only required part), then an optional blockquote summary, optional free text without headings, and H2 sections whose lists contain Markdown links in the form - [name](url): notes. An ## Optional section is a convention for secondary links.

Does llms.txt help SEO or Google rankings?

No. Google Search says it does not use llms.txt files and that having one neither helps nor hurts rankings. It can help AI agents and coding assistants that read it, and Chrome Lighthouse has an experimental llms.txt audit, but measured traffic to llms.txt files is low.

Is llms-full.txt required?

No. llms-full.txt (the full text of a site's docs in one file) is a common convention, not part of the llms.txt format. Publish it only if your content is small enough to be useful in one file.

Which AI crawlers should I check in robots.txt?

At least GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Google-Extended and Applebot-Extended are control tokens for AI model use, not separate crawlers.

How much does the Fairquery Website Tech, SEO & llms.txt Checker cost?

$0.0015 per site checked ($1.50 per 1,000 sites) on Apify, pay per event, plus Apify's $0.00005 Actor-start fee. Failed fetches are free. The Python script in this guide is free.

Can AI agents use it?

Yes. Add the Apify MCP server scoped to fairquery/website-tech-seo-ai-readiness to Claude Desktop, Cursor or VS Code, or run the stdio MCP server from the GitHub repo.

Source and docs: github.com/fairquery/website-tech-seo-ai-readiness (MIT) · Apify Store · this site's llms.txt · all Fairquery tools