How do I check what my website and GitHub repos expose to AI crawlers and the open web?

Fairquery · October 18, 2026 · 9 min read

What "AI exposure" means in practice

AI crawlers and the people building on top of language models read the same open web as everyone else. Nobody can certify a site as "AI-safe", and this post won't pretend to. The useful question is narrower: what is already public, and is any of it a mistake? In practice the same few things come up: a key committed to a public repo years ago, a .env or .git directory deployed by accident, a robots.txt nobody wrote with AI crawlers in mind, and missing email-authentication records that make your domain easy to spoof.

What to check

A free, minimal check in Python

Run it only on a domain you own. It requests each path once (no brute forcing), counts a file as exposed only if the response also looks like that file (many sites answer 200 for every URL), skips paths your robots.txt disallows, stops on a 403 or 429, prints which AI crawlers robots.txt allows, and looks up SPF and DMARC. It does not read GitHub, mask anything or score anything.

#!/usr/bin/env python3
"""Minimal, passive public-exposure check for a domain YOU OWN.
Requests each well-known path once (no brute forcing), reads robots.txt rules for AI crawlers,
and looks up SPF/DMARC. No GitHub scan, no masking, no scoring.
Usage: pip install dnspython && python3 exposure-free.py example.com"""
import re, sys, urllib.error, urllib.request, urllib.robotparser
import dns.resolver

UA = "ExposureCheck/1.0 (+mailto:you@example.com)"
# path -> content signature; a 200 alone is not enough (many sites answer 200 for everything)
PATHS = {
    "/.env": r"(?m)^\s*[A-Z0-9_]+\s*=",
    "/.git/config": r"\[core\]|repositoryformatversion",
    "/config.json": r"(?i)api[_-]?key|secret|password|token",
    "/backup.sql": r"(?i)CREATE TABLE|INSERT INTO",
    "/phpinfo.php": r"PHP Version",
    "/openapi.json": r'"(openapi|swagger)"\s*:',
    "/swagger.json": r'"(openapi|swagger)"\s*:',
}
AI_BOTS = ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot", "PerplexityBot", "Bytespider"]

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=10) as r:
            return r.status, r.read(65536).decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        return e.code, ""
    except Exception:
        return None, ""

def txt(name):
    try:
        return " ".join(b.decode() for r in dns.resolver.resolve(name, "TXT") for b in r.strings)
    except Exception:
        return ""

def main(domain):
    base = f"https://{domain}"
    rp = urllib.robotparser.RobotFileParser()
    status, body = fetch(base + "/robots.txt")
    rp.parse(body.splitlines() if status == 200 else [])
    print(f"# {domain}")
    for path, sig in PATHS.items():
        if not rp.can_fetch(UA, base + path):
            print(f"  skip (robots.txt): {path}")
            continue
        status, body = fetch(base + path)
        if status in (403, 429):
            print(f"  {path}: HTTP {status}, stopping requests to this host")
            break
        exposed = status == 200 and re.search(sig, body)
        print(f"  {'EXPOSED' if exposed else 'ok':7} {path} ({status})")
    print("  AI crawlers allowed on / by robots.txt:")
    for bot in AI_BOTS:
        print(f"    {bot}: {'allowed' if rp.can_fetch(bot, base + '/') else 'blocked'}")
    print(f"  SPF:   {'present' if 'v=spf1' in txt(domain) else 'MISSING'}")
    print(f"  DMARC: {'present' if 'v=DMARC1' in txt('_dmarc.' + domain) else 'MISSING'}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: exposure-free.py <domain-you-own>")
    main(sys.argv[1])

Output for our own domain on 2026-10-18 (the SPF and DMARC lines are left out because the sandbox we ran it in can't do DNS TXT lookups; on a normal machine they print present or MISSING):

# getfairquery.com
  ok      /.env (404)
  ok      /.git/config (404)
  ok      /config.json (404)
  ok      /backup.sql (404)
  ok      /phpinfo.php (404)
  ok      /openapi.json (404)
  ok      /swagger.json (404)
  AI crawlers allowed on / by robots.txt:
    GPTBot: allowed
    ClaudeBot: allowed
    Google-Extended: allowed
    CCBot: allowed
    PerplexityBot: allowed
    Bytespider: allowed

The full scan with the Fairquery Actor

The Actor does the same website checks more thoroughly (15 fixed paths, source maps referenced by your homepage, security headers, contact data, security.txt, 18 AI crawlers split into training and search bots) and adds the GitHub part: it lists your org's public repos, reads up to 40 likely files per repo (config, env, CI, infrastructure, source) and matches rules for AWS, GitHub, Slack, Google, Stripe, SendGrid, Twilio, OpenAI, Anthropic and npm tokens, JWTs, private keys, database URLs and high-entropy assignments.

Secrets are masked at the source: you get the rule, file, line and a preview such as AK********LE (len 20), never the full value, plus a likely_valid / likely_test flag. Tick the authorization box, give it your domains and org:

{
  "authorizedToScan": true,
  "domains": [
    "getfairquery.com"
  ],
  "githubOrgs": [
    "fairquery"
  ],
  "maxRepos": 50,
  "minSeverity": "info"
}

Rows from the verification run on the public build (2026-10-18, our own domain and the fairquery GitHub org; run KF83rd80zNUKoM4oy). The only "secrets" in our 15 repos are AWS's documented example key and test strings in our own test code, correctly flagged low and likely_test:

{
  "findings (3 of 11)": [
    {
      "category": "ai_policy",
      "kind": "ai_training_crawlers",
      "severity": "medium",
      "title": "13 AI training crawler(s) are allowed by robots.txt",
      "location": "https://getfairquery.com/robots.txt",
      "recommendation": "If you do not want your content used for AI model training, add Disallow rules for GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and similar, and consider a TDM Reservation / noai meta tag. This is a policy signal, not an enforcement mechanism."
    },
    {
      "category": "headers",
      "kind": "missing_security_headers",
      "severity": "medium",
      "title": "5 security header(s) missing",
      "location": "https://getfairquery.com",
      "recommendation": "Add HSTS: Strict-Transport-Security: max-age=31536000; includeSubDomains. Add a Content-Security-Policy to limit script/style sources. Add X-Content-Type-Options: nosniff. Add X-Frame-Options: DENY or a CSP frame-ancestors directive to prevent clickjacking. Add a Referrer-Policy (e.g. strict-origin-when-cross-origin)."
    },
    {
      "category": "secrets",
      "kind": "aws-access-key-id",
      "severity": "low",
      "title": "AWS Access Key ID in public repo fairquery/llm-data-guard-pii-secrets",
      "location": "https://github.com/fairquery/llm-data-guard-pii-secrets/blob/main/.actor/README.md#L66",
      "evidence": "aws-access-key-id: AK********LE (len 20)",
      "confidence": "likely_test",
      "recommendation": "Revoke/rotate the credential if real, remove it from the code, purge it from git history (e.g. git filter-repo), and load secrets from environment variables or a secrets manager instead."
    }
  ],
  "summary (OUTPUT, abridged)": {
    "overall_score": 61,
    "grade": "D",
    "category_scores": {
      "secrets": 96,
      "exposed_files": 100,
      "ai_policy": 92,
      "data_exposure": 97,
      "headers": 92,
      "email_auth": 84
    },
    "findings_total": 11,
    "findings_by_severity": {
      "info": 1,
      "low": 6,
      "medium": 4,
      "high": 0,
      "critical": 0
    },
    "events": {
      "domain-scanned": 1,
      "finding-reported": 11,
      "repo-scanned": 15
    },
    "event_charges_usd_at_list_price": 0.1184,
    "duration_secs": 144.6,
    "github_orgs": [
      {
        "org": "fairquery",
        "repos_found": 15,
        "repos_scanned": 15,
        "findings": 5
      }
    ]
  }
}

That run scored 61/100 (D), mostly from four medium findings: 13 AI training crawlers allowed, missing security headers, and no SPF or DMARC record. The last two are worth a second look: our DNS zone changed that day (we had just moved to a custom domain), and public resolvers already returned both records while the platform's resolver did not. DNS checks report what the resolver sees at scan time, so re-run before you act on them. Some of the other findings are deliberate. We want AI assistants to read and cite our guides, so our robots.txt allows those crawlers on purpose, and GitHub Pages does not let us set custom headers. The point of the report is to make those choices visible, not to push every score to 100. It cost $0.1184: one domain, 15 repos, 11 findings.

Passive by design

• No port scans, logins, exploits, fuzzing or brute forcing. Only GET requests to public URLs, the public GitHub API and DNS lookups.
• robots.txt is checked for every website request, including the well-known-file checks; the full path list is in the README.
• Honest user agent with a contact address, one request at a time per host, and a 403 or 429 stops all requests to that host for the run.
• Private and internal addresses are refused (SSRF protection, ports 80 and 443 only).
• Without the authorization checkbox the run stops before contacting any target and nothing is charged.

What it costs, compared with security platforms

Vendor prices from their pricing pages as of 2026-10-18. These platforms do much more: continuous monitoring, private repositories, alerting and remediation workflows. This Actor doesn't replace them. It is a cheap snapshot for small teams that want to check their public footprint once, or on a schedule.

FAQ

What is AI exposure?

Mostly ordinary public exposure seen through an AI lens: what crawlers, model builders and automated scanners can already read about you. That includes secrets committed to public repos, sensitive files your web server serves by accident, your robots.txt rules for AI crawlers, contact data on your pages, and missing SPF/DMARC records.

How do I block AI crawlers from training on my site?

Add Disallow rules for training crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider in robots.txt, and optionally a noai or TDM-Reservation signal. These are policy signals that well-behaved crawlers follow, not technical enforcement. Search-and-answer bots such as OAI-SearchBot or PerplexityBot can be allowed separately.

Will the scanner show me the leaked secret?

No. Findings are masked at the source: you get the rule, file, line and a preview with only the first and last two characters and the length, plus a likely_valid or likely_test flag. Open the file on GitHub to see the value, then rotate it.

Is the scan safe to run against my production site?

It is passive: only GET requests to public URLs, the public GitHub API and DNS lookups, one request at a time per host, robots.txt respected, each well-known path requested at most once, and a 403 or 429 stops requests to that host. No port scans, logins, exploits, fuzzing or brute forcing.

Can I scan a site I don't own?

Only with explicit permission. The Actor requires an authorization checkbox; without it the run stops before contacting any target and nothing is charged. You are responsible for having that authorization.

Does a clean report mean my site is secure?

No. It means nothing was found in what was checked, in public, at scan time. It is not a penetration test, an audit or a compliance certification, and secret detection is heuristic.

How much does it cost?

Pay per event on Apify: $0.039 per domain scanned, $0.0039 per public repo scanned and $0.0019 per finding reported, with no start fee. Our own domain plus 15 repos with 11 findings cost $0.1184.

Run it: Apify Store · Source: GitHub (MIT) · all Fairquery tools