# Fairquery > Fairquery builds data tools for developers and AI agents: sanctions screening (OFAC, EU, UK, UN), SEC EDGAR data, PII/secret redaction, email validation, website tech/SEO/AI-readiness checks, company contact enrichment and OSV dependency vulnerability checks. Most tools are Apify Actors with a local stdio MCP server; the Agent Trust MCP server bundles several of them. ## Posts - [How do I check what my website and GitHub repos expose to AI crawlers and the open web?](https://getfairquery.com/posts/check-website-ai-exposure-scan/): Check your own domain and public GitHub repos for AI-era exposure: committed secrets (masked), exposed .env/.git/backup files, which AI crawlers may train on your site, security headers and SPF/DMARC. A free Python check and a passive, scored scan at $0.039 per domain. - [How do I track changes to competitors' pricing, terms and careers pages without false alarms?](https://getfairquery.com/posts/track-competitor-pricing-terms-changes/): Track competitor pricing, terms/privacy, careers and changelog pages by meaning, not pixels: what to compare per page type, a free Python change check, A/B-noise filtering, and a scheduled Actor at $0.00049 per page checked. - [How do I check a website's llms.txt file and its robots.txt rules for AI crawlers?](https://getfairquery.com/posts/check-llms-txt-file-ai-crawlers-robots-txt/): Check that /llms.txt follows the llmstxt.org format (H1, summary, link lists), that its links resolve and that robots.txt doesn't block GPTBot, ClaudeBot or PerplexityBot: a free Python checker, real outputs, and a bulk API at $0.0015 per site. - [How do I use the OSV API in Python to check packages for vulnerabilities (query, batch, pagination)?](https://getfairquery.com/posts/osv-api-python-query-batch-vulnerabilities/): Query OSV.dev from Python with the standard library: /v1/query, /v1/querybatch with pagination, /v1/vulns details, alias de-duplication, severity traps and a CI exit code. Real outputs; or finished rows at $0.0005 per package + $0.0015 per vulnerability. - [How do I get live job listings straight from company ATS boards (Greenhouse, Lever, Ashby) via API?](https://getfairquery.com/posts/live-job-listings-api-greenhouse-lever-ashby/): Read jobs straight from employers' own Greenhouse, Lever and Ashby boards: the free public APIs, a 25-line Python example, ghost-job signals (age, long-open, reposts) and a feed at $0.99 per 1,000 jobs. Real outputs. - [How do I get SEC EDGAR company facts (XBRL financials) and filings by ticker via API?](https://getfairquery.com/posts/sec-edgar-api-company-facts-xbrl-filings-by-ticker/): Get XBRL company facts (revenue, net income, EPS, assets) and 10-K/10-Q/8-K filings from SEC EDGAR by ticker, CIK or name: free Python against the official SEC APIs, two data traps, and an Apify Actor at $0.0015/company + $0.001/filing + $0.0002/fact. Real outputs. - [How do I screen names against OFAC, EU, UK and UN sanctions lists with an API?](https://getfairquery.com/posts/how-to-screen-names-ofac-eu-uk-un-sanctions-api/): Screen people and companies against OFAC SDN/Consolidated, EU, UK and UN sanctions lists via API: fuzzy matching, aliases, country and DOB checks, real output, $3 per 1,000 names on Apify or RapidAPI plans. - [How do I redact PII and secrets before sending data to an LLM?](https://getfairquery.com/posts/redact-pii-secrets-before-sending-data-to-llm/): Detect and redact emails, phones, card numbers, SSNs, IBANs and API keys before text reaches an LLM: free local MCP tools (pii_scan_text, pii_redact_text) or an API at $0.0015/doc + $0.0002/finding. Real outputs. - [How do I find a company's contact email, phone and social profiles from its domain?](https://getfairquery.com/posts/find-company-contact-email-phone-socials-from-domain/): Turn a domain into public business contacts: role emails validated for MX, E.164 phones, social profiles, company name and address, robots.txt respected. Real output, $0.003 per domain. - [How do I check npm and PyPI dependencies for vulnerabilities in CI with OSV?](https://getfairquery.com/posts/check-npm-pypi-dependencies-vulnerabilities-ci-osv/): Gate CI on vulnerable npm/PyPI dependencies with OSV.dev: a free querybatch gate, free MCP lookups, and a GitHub Actions example with CVSS, CVE IDs and fixed versions at $0.0005/package + $0.0015/vulnerability. Real outputs. - [How do I give my AI agent trust and safety tools via MCP?](https://getfairquery.com/posts/give-ai-agent-trust-safety-tools-mcp-server/): Add PII/secret redaction, email checks, sanctions screening and OSV vulnerability lookups to Claude Desktop, Cursor or VS Code with one MCP server: 12 tools, 7 free. Config snippets and real outputs. - [How to validate email addresses via API: MX, disposable and typo checks](https://getfairquery.com/posts/validate-email-addresses-api-mx-disposable-typo/): Validate email addresses without sending mail: syntax, MX and null MX, disposable, role and free-provider flags, typo suggestions, SPF/DMARC. Free options and an API at $0.00079 per email. ## Tools - [AI Exposure Scanner](https://apify.com/fairquery/ai-exposure-scanner): Passive, scored scan of your own domains and public GitHub repos: masked leaked secrets, exposed .env/.git/backup files, AI-crawler robots.txt policy, security headers and SPF/DMARC, each with a fix. Authorization required. Apify Actor. Source: https://github.com/fairquery/ai-exposure-scanner - [Competitor Change Tracker](https://apify.com/fairquery/competitor-change-tracker): Track competitors’ public pricing, terms/privacy, careers, changelog and sitemap pages on a schedule — only meaningful changes, with plain-English summaries, severity and Slack/Discord alerts. Apify Actor. Source: https://github.com/fairquery/competitor-change-tracker - [Company Jobs Feed (Greenhouse, Lever, Ashby)](https://apify.com/fairquery/company-jobs-feed): Live jobs read from employers' own ATS boards via their public APIs, with age, long-open and repost signals, history and filters. Apify Actor + stdio MCP server (4 free local tools). Source: https://github.com/fairquery/company-jobs-feed - [Sanctions & Watchlist Screener](https://apify.com/fairquery/sanctions-watchlist-screener): Fuzzy-screen people and company names against OFAC SDN/Consolidated, EU, UK and UN sanctions lists from the official sources (at most 6 h old, publication dates shown). Apify Actor + stdio MCP server. Source: https://github.com/fairquery/sanctions-watchlist-screener - [SEC EDGAR Company Facts & Filings](https://apify.com/fairquery/edgar-company-facts-filings): SEC EDGAR filings and XBRL financial facts by ticker, CIK or company name. Apify Actor + MCP. Source: https://github.com/fairquery/edgar-company-facts-filings - [LLM Data Guard (PII & secrets)](https://apify.com/fairquery/llm-data-guard-pii-secrets): Detect and redact PII and secrets (API keys, tokens, private keys) before text reaches an LLM. Apify Actor + stdio MCP server. Source: https://github.com/fairquery/llm-data-guard-pii-secrets - [Website Contact & Company Enricher](https://apify.com/fairquery/website-contact-company-enricher): Domain to public business contacts: role emails, phones, socials, company name, address. Respects robots.txt. Apify Actor + MCP. Source: https://github.com/fairquery/website-contact-company-enricher - [OSV Dependency Vulnerability Check](https://apify.com/fairquery/osv-dependency-vulnerability-check): Check npm/PyPI/Go/Maven dependencies and lockfiles against OSV.dev: CVEs, CVSS, fixed versions. Apify Actor + MCP. Source: https://github.com/fairquery/osv-dependency-vulnerability-check - [Website Tech, SEO & AI-Readiness](https://apify.com/fairquery/website-tech-seo-ai-readiness): Website tech stack, SEO score, robots.txt AI-bot rules, sitemap and llms.txt check. Apify Actor + MCP. Source: https://github.com/fairquery/website-tech-seo-ai-readiness - [Email Validator (MX, disposable, typo check)](https://apify.com/fairquery/email-validator-mx-disposable): Validate emails without sending mail: syntax, MX, disposable/role/free-provider flags, typo fixes, SPF/DMARC. Apify Actor + MCP. Source: https://github.com/fairquery/email-validator-mx-disposable - [Agent Trust & Data Safety MCP server](https://github.com/fairquery/fairquery-agent-trust-mcp): One MCP server with 12 tools for AI agents: PII/secret redaction, email checks, sanctions screening and OSV vulnerability lookups. 7 free local tools; paid tools use your own Apify token. Source: https://github.com/fairquery/fairquery-agent-trust-mcp - [Fairquery OSV for VS Code](https://github.com/fairquery/fairquery-vscode-osv): VS Code extension that flags vulnerable dependencies using the free OSV.dev API. Source: https://github.com/fairquery/fairquery-vscode-osv - [n8n templates](https://github.com/fairquery/n8n-templates): Free n8n workflows for Fairquery Apify Actors: OSV CI gate, signup email gate, SEO and AI-crawler monitor. Source: https://github.com/fairquery/n8n-templates ## Where to get them - [Apify Store](https://apify.com/fairquery) - [GitHub](https://github.com/fairquery) - [RapidAPI: Sanctions Screening (OFAC, EU, UK, UN)](https://rapidapi.com/fairquery/api/sanctions-screening-ofac-eu-uk-un) - [RapidAPI: Email Validator (MX, disposable and typo check)](https://rapidapi.com/Fairquery/api/email-validator-mx-disposable-and-typo-check) - [RapidAPI: Website Intelligence (tech stack, SEO and contacts)](https://rapidapi.com/Fairquery/api/website-intelligence-tech-stack-seo-and-contacts) - [npm: @fairquery/agent-trust-mcp](https://www.npmjs.com/package/@fairquery/agent-trust-mcp) - [Glama: Fairquery Agent Trust MCP](https://glama.ai/mcp/servers/fairquery/fairquery-agent-trust-mcp) - [Gumroad](https://fairquery.gumroad.com) ## About and security - [About Fairquery](https://getfairquery.com/about/): what Fairquery is, its principles (precision, transparent pay-per-result pricing, minimal data by design, AI-agent friendly) and contact. - [Security & Privacy](https://getfairquery.com/security/): data handling per tool and how to report a vulnerability. - [Terms of Service](https://getfairquery.com/terms/): plain-language terms of use; platform terms (Apify, RapidAPI, Gumroad) govern billing and refunds. - [Privacy Notice](https://getfairquery.com/privacy/): what Fairquery collects, how it is used and shared, cookies (none on the blog), retention and how to make requests. - [security.txt](https://getfairquery.com/.well-known/security.txt) - [Contact](mailto:fairquery@proton.me): support, questions and security reports (fairquery@proton.me) ## Optional - [Full text for LLMs](https://getfairquery.com/llms-full.txt) - [Tools page](https://getfairquery.com/tools/) - [Atom feed](https://getfairquery.com/feed.xml) --- # How do I check what my website and GitHub repos expose to AI crawlers and the open web? URL: https://getfairquery.com/posts/check-website-ai-exposure-scan/ Published: 2026-10-18 Short answer: "AI exposure" is mostly plain public exposure. Check four things on the domains and GitHub orgs you own: secrets committed to public repos, sensitive files your web server hands out (/.env, /.git/config, backups, API specs), which AI crawlers your robots.txt lets train on your content, and basic hygiene (security headers, SPF and DMARC). The free Python script below covers the website part in under a minute. For a masked, scored report that also reads your public GitHub repos, the Fairquery AI Exposure Scanner runs on Apify for $0.039 per domain, $0.0039 per public repo and $0.0019 per finding. It is passive, and it only runs after you confirm you own the targets or are allowed to scan them. ## What "AI exposure" means in practice AI crawlers and the people building on top of language models read the same open web as everyone else. Nobody can certify a site as "AI-safe", and this post won't pretend to. The useful question is narrower: what is already public, and is any of it a mistake? In practice the same few things come up: a key committed to a public repo years ago, a .env or .git directory deployed by accident, a robots.txt nobody wrote with AI crawlers in mind, and missing email-authentication records that make your domain easy to spoof. ## What to check Area | Check | Why it matters | Public GitHub repos | API keys, tokens, private keys and database URLs in committed files | Anyone can read and clone a public repo, including automated scanners | Web server | /.env, /.git/config, backups, /config.json, phpinfo, source maps, OpenAPI/Swagger files | One misplaced file can expose credentials or your whole source tree | AI-crawler policy | robots.txt rules per AI bot, ai.txt/llms.txt, noai/TDM meta tags | Decide on purpose which bots may train on your content and which may only search and answer | Public contact data | Emails and phone numbers on the homepage, security.txt | Harvested addresses feed spam and phishing; security.txt tells researchers where to report problems | Hygiene | HSTS, CSP, X-Frame-Options and friends; SPF and DMARC records | Cheap fixes against clickjacking and email spoofing | ## A free, minimal check in Python Run it only on a domain you own. It requests each path once (no brute forcing), counts a file as exposed only if the response also looks like that file (many sites answer 200 for every URL), skips paths your robots.txt disallows, stops on a 403 or 429, prints which AI crawlers robots.txt allows, and looks up SPF and DMARC. It does not read GitHub, mask anything or score anything. ``` #!/usr/bin/env python3 """Minimal, passive public-exposure check for a domain YOU OWN. Requests each well-known path once (no brute forcing), reads robots.txt rules for AI crawlers, and looks up SPF/DMARC. No GitHub scan, no masking, no scoring. Usage: pip install dnspython && python3 exposure-free.py example.com""" import re, sys, urllib.error, urllib.request, urllib.robotparser import dns.resolver UA = "ExposureCheck/1.0 (+mailto:you@example.com)" # path -> content signature; a 200 alone is not enough (many sites answer 200 for everything) PATHS = { "/.env": r"(?m)^\s*[A-Z0-9_]+\s*=", "/.git/config": r"\[core\]|repositoryformatversion", "/config.json": r"(?i)api[_-]?key|secret|password|token", "/backup.sql": r"(?i)CREATE TABLE|INSERT INTO", "/phpinfo.php": r"PHP Version", "/openapi.json": r'"(openapi|swagger)"\s*:', "/swagger.json": r'"(openapi|swagger)"\s*:', } AI_BOTS = ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot", "PerplexityBot", "Bytespider"] def fetch(url): req = urllib.request.Request(url, headers={"User-Agent": UA}) try: with urllib.request.urlopen(req, timeout=10) as r: return r.status, r.read(65536).decode("utf-8", "replace") except urllib.error.HTTPError as e: return e.code, "" except Exception: return None, "" def txt(name): try: return " ".join(b.decode() for r in dns.resolver.resolve(name, "TXT") for b in r.strings) except Exception: return "" def main(domain): base = f"https://{domain}" rp = urllib.robotparser.RobotFileParser() status, body = fetch(base + "/robots.txt") rp.parse(body.splitlines() if status == 200 else []) print(f"# {domain}") for path, sig in PATHS.items(): if not rp.can_fetch(UA, base + path): print(f" skip (robots.txt): {path}") continue status, body = fetch(base + path) if status in (403, 429): print(f" {path}: HTTP {status}, stopping requests to this host") break exposed = status == 200 and re.search(sig, body) print(f" {'EXPOSED' if exposed else 'ok':7} {path} ({status})") print(" AI crawlers allowed on / by robots.txt:") for bot in AI_BOTS: print(f" {bot}: {'allowed' if rp.can_fetch(bot, base + '/') else 'blocked'}") print(f" SPF: {'present' if 'v=spf1' in txt(domain) else 'MISSING'}") print(f" DMARC: {'present' if 'v=DMARC1' in txt('_dmarc.' + domain) else 'MISSING'}") if __name__ == "__main__": if len(sys.argv) != 2: sys.exit("usage: exposure-free.py ") main(sys.argv[1]) ``` Output for our own domain on 2026-10-18 (the SPF and DMARC lines are left out because the sandbox we ran it in can't do DNS TXT lookups; on a normal machine they print present or MISSING): ``` # getfairquery.com ok /.env (404) ok /.git/config (404) ok /config.json (404) ok /backup.sql (404) ok /phpinfo.php (404) ok /openapi.json (404) ok /swagger.json (404) AI crawlers allowed on / by robots.txt: GPTBot: allowed ClaudeBot: allowed Google-Extended: allowed CCBot: allowed PerplexityBot: allowed Bytespider: allowed ``` ## The full scan with the Fairquery Actor The Actor does the same website checks more thoroughly (15 fixed paths, source maps referenced by your homepage, security headers, contact data, security.txt, 18 AI crawlers split into training and search bots) and adds the GitHub part: it lists your org's public repos, reads up to 40 likely files per repo (config, env, CI, infrastructure, source) and matches rules for AWS, GitHub, Slack, Google, Stripe, SendGrid, Twilio, OpenAI, Anthropic and npm tokens, JWTs, private keys, database URLs and high-entropy assignments. Secrets are masked at the source: you get the rule, file, line and a preview such as AK********LE (len 20), never the full value, plus a likely_valid / likely_test flag. Tick the authorization box, give it your domains and org: ``` { "authorizedToScan": true, "domains": [ "getfairquery.com" ], "githubOrgs": [ "fairquery" ], "maxRepos": 50, "minSeverity": "info" } ``` Rows from the verification run on the public build (2026-10-18, our own domain and the fairquery GitHub org; run KF83rd80zNUKoM4oy). The only "secrets" in our 15 repos are AWS's documented example key and test strings in our own test code, correctly flagged low and likely_test: ``` { "findings (3 of 11)": [ { "category": "ai_policy", "kind": "ai_training_crawlers", "severity": "medium", "title": "13 AI training crawler(s) are allowed by robots.txt", "location": "https://getfairquery.com/robots.txt", "recommendation": "If you do not want your content used for AI model training, add Disallow rules for GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and similar, and consider a TDM Reservation / noai meta tag. This is a policy signal, not an enforcement mechanism." }, { "category": "headers", "kind": "missing_security_headers", "severity": "medium", "title": "5 security header(s) missing", "location": "https://getfairquery.com", "recommendation": "Add HSTS: Strict-Transport-Security: max-age=31536000; includeSubDomains. Add a Content-Security-Policy to limit script/style sources. Add X-Content-Type-Options: nosniff. Add X-Frame-Options: DENY or a CSP frame-ancestors directive to prevent clickjacking. Add a Referrer-Policy (e.g. strict-origin-when-cross-origin)." }, { "category": "secrets", "kind": "aws-access-key-id", "severity": "low", "title": "AWS Access Key ID in public repo fairquery/llm-data-guard-pii-secrets", "location": "https://github.com/fairquery/llm-data-guard-pii-secrets/blob/main/.actor/README.md#L66", "evidence": "aws-access-key-id: AK********LE (len 20)", "confidence": "likely_test", "recommendation": "Revoke/rotate the credential if real, remove it from the code, purge it from git history (e.g. git filter-repo), and load secrets from environment variables or a secrets manager instead." } ], "summary (OUTPUT, abridged)": { "overall_score": 61, "grade": "D", "category_scores": { "secrets": 96, "exposed_files": 100, "ai_policy": 92, "data_exposure": 97, "headers": 92, "email_auth": 84 }, "findings_total": 11, "findings_by_severity": { "info": 1, "low": 6, "medium": 4, "high": 0, "critical": 0 }, "events": { "domain-scanned": 1, "finding-reported": 11, "repo-scanned": 15 }, "event_charges_usd_at_list_price": 0.1184, "duration_secs": 144.6, "github_orgs": [ { "org": "fairquery", "repos_found": 15, "repos_scanned": 15, "findings": 5 } ] } } ``` That run scored 61/100 (D), mostly from four medium findings: 13 AI training crawlers allowed, missing security headers, and no SPF or DMARC record. The last two are worth a second look: our DNS zone changed that day (we had just moved to a custom domain), and public resolvers already returned both records while the platform's resolver did not. DNS checks report what the resolver sees at scan time, so re-run before you act on them. Some of the other findings are deliberate. We want AI assistants to read and cite our guides, so our robots.txt allows those crawlers on purpose, and GitHub Pages does not let us set custom headers. The point of the report is to make those choices visible, not to push every score to 100. It cost $0.1184: one domain, 15 repos, 11 findings. ## Passive by design - No port scans, logins, exploits, fuzzing or brute forcing. Only GET requests to public URLs, the public GitHub API and DNS lookups. - robots.txt is checked for every website request, including the well-known-file checks; the full path list is in the README. - Honest user agent with a contact address, one request at a time per host, and a 403 or 429 stops all requests to that host for the run. - Private and internal addresses are refused (SSRF protection, ports 80 and 443 only). - Without the authorization checkbox the run stops before contacting any target and nothing is charged. ## What it costs, compared with security platforms Option | Price | Shape | GitGuardian | Free up to 25 developers, then $18 per active developer per month | Continuous secret monitoring, including private code | UpGuard BreachSight | $250–$2,000 per month by company size | Continuous attack-surface monitoring | Fairquery AI Exposure Scanner | ≈ $0.05 for one domain, ≈ $0.12 for a domain plus 15 repos | On-demand passive snapshot of public exposure, pay per event, no contract | Vendor prices from their pricing pages as of 2026-10-18. These platforms do much more: continuous monitoring, private repositories, alerting and remediation workflows. This Actor doesn't replace them. It is a cheap snapshot for small teams that want to check their public footprint once, or on a schedule. ## FAQ ### What is AI exposure? Mostly ordinary public exposure seen through an AI lens: what crawlers, model builders and automated scanners can already read about you. That includes secrets committed to public repos, sensitive files your web server serves by accident, your robots.txt rules for AI crawlers, contact data on your pages, and missing SPF/DMARC records. ### How do I block AI crawlers from training on my site? Add Disallow rules for training crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider in robots.txt, and optionally a noai or TDM-Reservation signal. These are policy signals that well-behaved crawlers follow, not technical enforcement. Search-and-answer bots such as OAI-SearchBot or PerplexityBot can be allowed separately. ### Will the scanner show me the leaked secret? No. Findings are masked at the source: you get the rule, file, line and a preview with only the first and last two characters and the length, plus a likely_valid or likely_test flag. Open the file on GitHub to see the value, then rotate it. ### Is the scan safe to run against my production site? It is passive: only GET requests to public URLs, the public GitHub API and DNS lookups, one request at a time per host, robots.txt respected, each well-known path requested at most once, and a 403 or 429 stops requests to that host. No port scans, logins, exploits, fuzzing or brute forcing. ### Can I scan a site I don't own? Only with explicit permission. The Actor requires an authorization checkbox; without it the run stops before contacting any target and nothing is charged. You are responsible for having that authorization. ### Does a clean report mean my site is secure? No. It means nothing was found in what was checked, in public, at scan time. It is not a penetration test, an audit or a compliance certification, and secret detection is heuristic. ### How much does it cost? Pay per event on Apify: $0.039 per domain scanned, $0.0039 per public repo scanned and $0.0019 per finding reported, with no start fee. Our own domain plus 15 repos with 11 findings cost $0.1184. Limitations. A clean report is not proof of safety, not a penetration test and not a compliance certification. It covers only what was public and checked at scan time. Secret detection is heuristic: it can miss formats it has no rule for, files outside the 40 candidates per repo and anything in git history, and it can flag harmless strings, so verify every finding. Only public repos and the homepage are read. Fairquery is not affiliated with GitHub, GitGuardian, UpGuard or any AI company mentioned. Run it: Apify Store · Source: GitHub (MIT) · all Fairquery tools --- # How do I track changes to competitors' pricing, terms and careers pages without false alarms? URL: https://getfairquery.com/posts/track-competitor-pricing-terms-changes/ Published: 2026-10-18 Short answer: fetch each competitor page on a schedule, strip everything that changes on every visit (scripts, nav, cookie banners, dates, tokens), and compare what is left by page type: plans and prices on pricing pages, sections on terms and privacy pages, job ids on Greenhouse, Lever or Ashby boards, URLs in the sitemap. A 30-line Python script below tells you whether a page changed. To know what changed (which plan's price moved, which legal section was rewritten, which roles opened) with alerts to Slack, the Fairquery Competitor Change Tracker runs on an Apify schedule for $0.00049 per page checked plus $0.0049 per change reported. ## Why screenshot and raw-text diffs are noisy on competitor pages Pricing and legal pages look static, but their HTML is not. Every request can carry a new CSRF token, a "last updated 3 minutes ago" label, a rotating testimonial, a cookie banner, an A/B-tested headline or a cache-busting query string. Visual and plain-text monitors report all of that as a change. After a week of false alarms people stop reading the alerts, and the one real price increase goes unnoticed. What you actually want to know is narrower: did a plan's price, currency or billing period change, was a plan added or removed, which section of the Terms changed and how, which roles opened or closed, which product pages appeared? That needs a comparison that understands the page type. ## What to compare on each page type Page | Compare | Ignore | Pricing | Plan name → price, currency, period (month/year), per-user unit, "billed annually" | Badges ("Most popular"), button labels, reordered cards, strikethrough prices | Terms / privacy | Sections by heading; added/removed sentences; renamed sections | "Last updated" dates, moved sentences, whitespace | Careers | Job ids from the public Greenhouse, Lever or Ashby board API; departments | Page layout, testimonials, view counters | Changelog / blog | New entry headings and links | Relative dates ("2 days ago") | Product pages | URLs in sitemap.xml (added/removed, product-like paths first) | Mass churn in blog archives, lastmod timestamps | ## A free, minimal change check in Python This script respects robots.txt, identifies itself, drops scripts, nav and footer, removes dates and times, and stores a hash per URL. Run it from cron; it prints baseline, unchanged or CHANGED. It tells you that something changed, not what, and it will still trip on A/B tests and rotating content. ``` # Minimal "did this page's text change?" check: polite, robots-aware, standard library only. import hashlib, json, os, re, sys, urllib.error, urllib.request, urllib.robotparser from html.parser import HTMLParser from urllib.parse import urlsplit UA = "MyChangeCheck/0.1 (+mailto:you@example.com)" url = sys.argv[1] # e.g. https://basecamp.com/about/policies/terms base = "{0.scheme}://{0.netloc}".format(urlsplit(url)) rp = urllib.robotparser.RobotFileParser() try: # fetch robots.txt with our own user agent; a 4xx means no rules rp.parse(urllib.request.urlopen(urllib.request.Request(base + "/robots.txt", headers={"User-Agent": UA}), timeout=20).read().decode("utf-8", "replace").splitlines()) except urllib.error.HTTPError as e: if e.code >= 500: sys.exit("robots.txt unavailable; not fetching") rp.parse([]) if not rp.can_fetch(UA, url): sys.exit("robots.txt disallows " + url) class Text(HTMLParser): skip, out = 0, [] def handle_starttag(self, t, a): self.skip += t in ("script", "style", "nav", "footer") def handle_endtag(self, t): self.skip -= t in ("script", "style", "nav", "footer") and self.skip > 0 def handle_data(self, d): if not self.skip and d.strip(): self.out.append(d.strip()) html = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": UA}), timeout=20).read().decode("utf-8", "replace") p = Text(); p.feed(html) text = re.sub(r"\b(19|20)\d\d-\d\d-\d\d\b|\b\d{1,2}:\d\d\b", "", "\n".join(p.out)) # drop dates/times digest = hashlib.sha256(text.encode()).hexdigest() state = json.load(open("state.json")) if os.path.exists("state.json") else {} print("baseline" if url not in state else ("unchanged" if state[url] == digest else "CHANGED")) state[url] = digest; json.dump(state, open("state.json", "w")) ``` ## Semantic tracking with the Fairquery Actor The Actor does the type-aware comparison for you. Give it pages (or just competitor domains plus the page types to find), schedule it, and each run returns only the changes, each with a plain-English summary, a severity and the details. The first run stores a baseline in a named key-value store in your own Apify account. ``` { "watchlistName": "fq-test-main", "pages": [ { "url": "https://basecamp.com/pricing", "type": "pricing" }, { "url": "https://linear.app/pricing", "type": "pricing" }, { "url": "https://tailscale.com/pricing", "type": "pricing" }, { "url": "https://zapier.com/pricing", "type": "pricing" }, { "url": "https://about.gitlab.com/pricing/", "type": "pricing" }, { "url": "https://github.com/pricing", "type": "pricing" }, { "url": "https://www.notion.com/pricing", "type": "pricing" }, { "url": "https://basecamp.com/about/policies/terms", "type": "terms" }, { "url": "https://github.com/site/terms", "type": "terms" }, { "url": "https://linear.app/terms", "type": "terms" }, { "url": "https://linear.app/privacy", "type": "privacy" }, { "url": "https://tailscale.com/terms", "type": "terms" }, { "url": "https://slack.com/terms-of-service", "type": "terms" }, { "url": "https://jobs.ashbyhq.com/linear", "type": "careers" }, { "url": "https://tailscale.com/careers", "type": "careers" }, { "url": "https://www.figma.com/careers/", "type": "careers" }, { "url": "https://linear.app/changelog", "type": "changelog" }, { "url": "https://posthog.com/changelog", "type": "changelog" }, { "url": "https://tailscale.com/", "type": "product" }, { "url": "https://job-boards.greenhouse.io/anthropic", "type": "careers" }, { "url": "https://jobs.lever.co/palantir", "type": "careers" }, { "url": "https://slack.com/pricing", "type": "pricing" }, { "url": "https://www.dropbox.com/plans", "type": "pricing" }, { "url": "https://www.amazon.com/gp/cart/view.html", "type": "generic" } ], "careersKeywords": [ "AI", "machine learning", "enterprise" ], "watchKeywords": [ "AI", "SOC 2" ], "minSeverity": "low", "includeDiff": true } ``` Rows from our test runs on 2026-10-18. The old values were simulated by editing the stored baseline, so the live page then differed from it; the current values are the real ones from Basecamp, Linear and Linear's Ashby board: ``` [ { "schema_version": "1.0.0", "watchlist": "fq-test-main", "checked_at": "2026-10-18T02:15:41.208Z", "url": "https://linear.app/pricing", "page_type": "pricing", "label": null, "domain": "linear.app", "type": "change", "final_url": "https://linear.app/pricing", "severity": "high", "change_kind": "price_changed", "summary": "\"Business\" price changed from $14/month per user to $16/month per user (up 14.3%).", "details": { "plan": "Business", "from": { "amount": 14, "currency": "USD", "period": "month", "billing": null, "unit": "user", "raw": "$16" }, "to": { "amount": 16, "currency": "USD", "period": "month", "billing": null, "unit": "user", "raw": "$16" }, "change_pct": 14.3 }, "previous_checked_at": "2026-10-18T02:10:52.136Z" }, { "schema_version": "1.0.0", "watchlist": "fq-test-main", "checked_at": "2026-10-18T02:15:41.208Z", "url": "https://basecamp.com/pricing", "page_type": "pricing", "label": null, "domain": "basecamp.com", "type": "change", "final_url": "https://basecamp.com/pricing", "severity": "high", "change_kind": "price_changed", "summary": "\"Pro\" price changed from $79/month to $99/month (up 25.3%).", "details": { "plan": "Pro", "from": { "amount": 79, "currency": "USD", "period": "month", "billing": null, "unit": null, "raw": "$79" }, "to": { "amount": 99, "currency": "USD", "period": "month", "billing": null, "unit": null, "raw": "$99" }, "change_pct": 25.3 }, "previous_checked_at": "2026-10-18T02:10:52.136Z" }, { "schema_version": "1.0.0", "watchlist": "fq-test-main", "checked_at": "2026-10-18T02:15:41.208Z", "url": "https://linear.app/terms", "page_type": "terms", "label": null, "domain": "linear.app", "type": "change", "final_url": "https://linear.app/terms", "severity": "high", "change_kind": "legal_text_changed", "summary": "Terms of service: 2 sections changed: \"Terms of Service\" (modified: +1/−1 sentences), \"Linear’s Ownership\" (added). Added: “(“Linear”).”", "details": { "sections_changed": 2, "changed_chars": 872, "sections": [ { "heading": "Terms of Service", "change": "modified", "added": [ "(“Linear”)." ], "removed": [ "Customer may terminate this Agreement at any time with thirty days written notice and will receive a pro-rata refund of prepaid fees." ], "more_added": 0, "more_removed": 0 }, { "heading": "Linear’s Ownership", "change": "added", "added": [ "Linear owns the Service, Software, Linear Content, Documentation, and anything else provided by Linear to Customer (collectively the “Linear Materials”)." ], "removed": [], "more_added": 0, "more_removed": 0 } ] }, "previous_checked_at": "2026-10-18T02:10:52.136Z" }, { "schema_version": "1.0.0", "watchlist": "fq-test-main", "checked_at": "2026-10-18T02:15:41.208Z", "url": "https://jobs.ashbyhq.com/linear", "page_type": "careers", "label": null, "domain": "jobs.ashbyhq.com", "type": "change", "final_url": "https://jobs.ashbyhq.com/linear", "severity": "low", "change_kind": "jobs_changed", "summary": "3 new roles (3 Product); 1 role closed (1 Engineering). Open roles: 27 → 29 (+7%). Examples: Senior / Staff Product Engineer, Product Engineer, Senior / Staff Product Engineer.", "details": { "open_before": 27, "open_after": 29, "opened": [ { "id": "069c4628-88d7-4e4d-b393-c996fc7f3076", "title": "Senior / Staff Product Engineer", "department": "Product", "location": "Europe" } ], "closed": [ { "id": "sim-closed-1", "title": "Senior Simulated Engineer", "department": "Engineering", "location": "North America" } ], "keyword_matches": [] }, "previous_checked_at": "2026-10-18T02:10:52.136Z" } ] ``` Two noise filters matter most in practice. First, a change is confirmed with a second read before it is reported: in our tests Slack's pricing page served two variants within seconds, and the confirmation read filtered it out. Second, the Actor remembers recent versions, so a page that alternates between two versions across runs alerts once and then only as a low-severity "switched back" item. Immediate reruns over 23 real pages (pricing, terms, careers, changelogs, a 1,281-URL sitemap) reported zero changes. ## Politeness rules we follow - Public, no-login pages only. No cookies or sessions, no CAPTCHA solving, no proxies to get around blocks. - robots.txt (RFC 9309) is checked for every page and redirect; disallowed pages are skipped and reported. Crawl-delay is honored. - Honest user agent (FairqueryChangeTracker/1.0 with a contact address), one request at a time per host, at least 1.5 s apart, conditional requests (ETag / If-Modified-Since). - Plain HTTP first. A headless browser is opt-in, only for JavaScript-rendered pricing pages, and still respects robots.txt. - A 403 or 429 stops all requests to that host for the rest of the run. ## Track a competitor's pricing and hiring together Price and headcount move together. A competitor that raises prices while opening a dozen enterprise sales and solutions-engineering roles is moving upmarket; one that freezes hiring and quietly discounts is defending share. Watching both at once turns a single data point into a trend you can act on. The tracker already reads a careers page that runs on Greenhouse, Lever or Ashby through that board's public API and reports new and closed roles by department, so a pricing change and a hiring change land in the same run, with the same severity and the same webhook alert. For deeper hiring intelligence — every open role across many companies, how long each has been open, repost and ghost-job signals, and keyword filters such as "AI" or "enterprise" — pair it with the Fairquery Company Jobs Feed (guide). Run the tracker on the pages you want watched for change, and the Jobs Feed when you want the full, filterable list of who is hiring and for what. ## Cost compared with change-monitoring services Option | 20 pages checked hourly (14,400 checks/month) | Output | Visualping Business | $100–140/month (20K checks) | Visual/text diff alerts | ChangeTower Essential | $36/month (100 pages hourly) | Visual/text/HTML diff alerts | Distill.io Starter | $15/month (30K checks) | Text diff alerts | Fairquery Competitor Change Tracker | ≈ $7 + $0.0049 per change | Plan/price, legal-section, job and sitemap changes with summaries and severity | Prices from the vendors' pricing pages on 2026-10-18. Freshness is set by your schedule in every case; none of these, including ours, sees a change made and reverted between two checks. ## FAQ ### Why do website change monitors send so many false alerts? Most compare screenshots or raw text, so rotating tokens, timestamps, cookie banners, testimonials and A/B-tested headlines all count as changes. Comparing by page type (plans and prices, legal sections, job ids, sitemap URLs) and masking dates and tokens removes most of that noise. ### How can I detect a competitor's price change automatically? Fetch the pricing page on a schedule, extract each plan's name, price, currency and billing period, and compare with the previous run. The Fairquery Competitor Change Tracker does this and reports changes such as a Pro plan price moving from $79/month to $99/month (up 25.3%), plus plans added or removed. ### Can I see exactly what changed in a Terms of Service or Privacy Policy? Yes. The Actor splits legal pages into sections by heading and reports which sections were added, removed, renamed or modified, with the removed and added sentences. Edits about liability, arbitration, data sharing, AI training, fees or termination are marked high severity. ### Is it real-time? No. Each run is a check, so freshness equals your schedule (for example hourly or daily with an Apify Schedule). A change made and reverted between two runs is not seen. ### Does it respect robots.txt? Yes. robots.txt is checked for every page and redirect, disallowed pages are skipped and reported, Crawl-delay is honored, and a 403 or 429 stops requests to that host for the run. It reads public pages only: no login, cookies, CAPTCHA solving or proxies. ### Can it track hiring at competitors? Yes. If a careers page uses Greenhouse, Lever or Ashby, the Actor reads that board's public API and reports new and closed roles by department, with optional keyword counts such as AI or enterprise. For the full, filterable list of who is hiring, pair it with the Fairquery Company Jobs Feed. ### How much does it cost? Pay-per-event pricing: $0.00049 per page checked and $0.0049 per change reported, $0.0059 per page rendered with the optional headless browser. Twenty pages checked hourly is about $7 a month plus changes. Limitations. Plan parsing is heuristic and works best on server-rendered pricing pages; usage-based calculators can produce odd plan names, but text changes are still caught. Prices can vary by country and A/B test, and runs use Apify datacenter IPs. Fairquery is not affiliated with Visualping, Distill, ChangeTower or any site mentioned. Docs: Apify Store · all Fairquery tools --- # How do I check a website's llms.txt file and its robots.txt rules for AI crawlers? URL: https://getfairquery.com/posts/check-llms-txt-file-ai-crawlers-robots-txt/ Published: 2026-10-18 Short answer: fetch https://your-site/llms.txt and check four things: it returns HTTP 200 as plain text or Markdown (not an HTML error page), its first line is an H1 with the site or project name (the only required element), it has a > blockquote summary, and its H2 sections contain link lists in the form - [name](url): notes. Then check that the links resolve and that robots.txt doesn't block the AI agents you want to read the file. The 80-line Python script below does all of that for free. To check many sites at once, together with tech stack, SEO and sitemap data, the Fairquery Website Tech, SEO & llms.txt Checker returns one JSON row per site for $0.0015 per site. ## What llms.txt is (and what it isn't) llms.txt is a proposal by Jeremy Howard (first published September 2024, revised as v2 in August 2026). It suggests a Markdown file at /llms.txt that gives AI agents a short summary of a site and a curated list of links to LLM-friendly pages. v2 also allows a file at a subpath (for example /docs/llms.txt covers /docs/) and recommends rel="alternate" type="text/markdown" and rel="describedby" links so agents can find Markdown versions and the covering llms.txt. Set your expectations honestly: - It is a community proposal, not a standard from the IETF or W3C. - Google Search ignores it. Google's AI optimization guide says Search doesn't use llms.txt files, and that having one won't help or hurt rankings (reported June 2026). - Chrome's Lighthouse added an experimental llms.txt audit in its Agentic Browsing category (version 13.3), which checks that the file exists and loads without a server error. - Real traffic is small. An Ahrefs study of 137,000 sites found that 97% of published llms.txt files got no requests at all in May 2026, and that most fetches that did happen came from bots such as GPTBot and coding agents. So treat llms.txt as a cheap, low-risk aid for coding agents and AI assistants that read your docs, not as an SEO lever. If you publish one, it should at least be valid and point to working pages. ## The format in one example ``` # Project name > One-paragraph summary with the key facts an agent needs. Optional notes (paragraphs or lists, no headings). ## Docs - [Quick start](https://example.com/docs/quickstart.md): install and first request - [API reference](https://example.com/docs/api.md) ## Optional - [Changelog](https://example.com/changelog.md) ``` Order matters: H1, then blockquote, then free text without headings, then H2 sections that each hold a Markdown list of links. The Optional section is a convention for links an agent can skip. Some sites also publish /llms-full.txt with the full text of their docs; that file is a common convention and isn't part of the llms.txt format. ## Option 1 (free): check llms.txt and AI-bot rules with Python Standard library only (Python 3.9+). It fetches /llms.txt, checks the structure against the format above, optionally sends a HEAD request to the first 25 links, looks for /llms-full.txt, and evaluates robots.txt for ten AI user agents: ``` #!/usr/bin/env python3 """Check a site's /llms.txt against the llmstxt.org format and its robots.txt rules for AI bots. Standard library only. Usage: python3 check_llms_txt.py example.com [--links]""" import re, sys, urllib.request, urllib.robotparser from urllib.parse import urljoin, urlparse UA = "llms-txt-check/1.0 (+https://getfairquery.com/)" AI_BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Google-Extended", "Applebot-Extended", "CCBot"] LINK_ITEM = re.compile(r"^\s*[-*+]\s+\[([^\]]+)\]\(([^)\s]+)\)(?:\s*:\s*(.*))?$") def fetch(url, method="GET", limit=1_000_000): req = urllib.request.Request(url, method=method, headers={"User-Agent": UA}) try: with urllib.request.urlopen(req, timeout=15) as r: body = r.read(limit).decode("utf-8", "replace") if method == "GET" else "" return r.status, r.headers.get("Content-Type", ""), r.geturl(), body except urllib.error.HTTPError as e: return e.code, e.headers.get("Content-Type", ""), url, "" except Exception as e: return None, "", url, str(e) def check_llms(text): issues, lines = [], text.lstrip("\ufeff").splitlines() content = [l for l in lines if l.strip()] h1 = [l for l in lines if re.match(r"^#\s+\S", l)] if not content or not re.match(r"^#\s+\S", content[0]): issues.append("first line is not an H1 (# Name), the only required element") if len(h1) > 1: issues.append(f"{len(h1)} H1 headings (expected 1)") if len(content) < 2 or not content[1].startswith(">"): issues.append("no blockquote summary (> ...) right after the H1 (recommended)") sections, links, bad_items, current = [], [], 0, None for l in lines: if l.startswith("## "): current = l[3:].strip(); sections.append(current); continue if current and re.match(r"^\s*[-*+]\s", l): m = LINK_ITEM.match(l) if m: links.append(m.group(2)) else: bad_items += 1 if not sections: issues.append("no H2 sections with file lists") if bad_items: issues.append(f"{bad_items} list items in H2 sections are not '- [name](url): notes'") if re.search(r"^#{3,}\s", text, re.M): issues.append("H3+ headings found (the format uses only H1 and H2)") return {"title": h1[0][2:].strip() if h1 else None, "sections": sections, "links": links, "issues": issues} def main(site, check_links=False): base = site if site.startswith("http") else "https://" + site origin = "{0.scheme}://{0.netloc}".format(urlparse(base)) status, ctype, final, body = fetch(origin + "/llms.txt") print(f"llms.txt: HTTP {status}, {ctype or '-'}, {len(body.encode())} bytes, {final}") if status == 200 and "html" not in ctype.lower() and not body.lstrip().lower().startswith("<"): r = check_llms(body) print(f" title: {r['title']}") print(f" sections ({len(r['sections'])}): {', '.join(r['sections'][:8])}") print(f" links: {len(r['links'])}, of which relative: {sum(not u.startswith('http') for u in r['links'])}") print(" issues: " + ("; ".join(r["issues"]) if r["issues"] else "none")) if check_links: broken = [] for u in r["links"][:25]: s = fetch(urljoin(final, u), "HEAD")[0] if s is None or s >= 400: broken.append(f"{u} ({s})") print(f" first {min(25, len(r['links']))} links checked, broken: {broken or 'none'}") else: print(" no valid llms.txt (missing, error or an HTML page)") s_full = fetch(origin + "/llms-full.txt", "HEAD")[0] print(f"llms-full.txt: HTTP {s_full}") rp = urllib.robotparser.RobotFileParser(origin + "/robots.txt") rs, _, _, rbody = fetch(origin + "/robots.txt") rp.parse(rbody.splitlines() if rs == 200 else []) print(f"robots.txt: HTTP {rs}") for bot in AI_BOTS: home, llms = rp.can_fetch(bot, origin + "/"), rp.can_fetch(bot, origin + "/llms.txt") print(f" {bot:<18} /: {'allowed' if home else 'blocked'} /llms.txt: {'allowed' if llms else 'blocked'}") if __name__ == "__main__": main(sys.argv[1], "--links" in sys.argv) ``` Real output for our own site (October 18, 2026, python3 check_llms_txt.py fairquery.github.io --links): ``` llms.txt: HTTP 200, text/plain; charset=utf-8, 8355 bytes, https://fairquery.github.io/llms.txt title: Fairquery sections (5): Posts, Tools, Where to get them, About and security, Optional links: 35, of which relative: 0 issues: 1 list items in H2 sections are not '- [name](url): notes' first 25 links checked, broken: ['https://www.npmjs.com/package/@fairquery/agent-trust-mcp (403)'] llms-full.txt: HTTP 200 robots.txt: HTTP 200 GPTBot /: allowed /llms.txt: allowed OAI-SearchBot /: allowed /llms.txt: allowed ChatGPT-User /: allowed /llms.txt: allowed ClaudeBot /: allowed /llms.txt: allowed Claude-SearchBot /: allowed /llms.txt: allowed Claude-User /: allowed /llms.txt: allowed PerplexityBot /: allowed /llms.txt: allowed Google-Extended /: allowed /llms.txt: allowed Applebot-Extended /: allowed /llms.txt: allowed CCBot /: allowed /llms.txt: allowed ``` Two findings, both useful. First, our file had one non-link list item (- Contact: fairquery@proton.me) inside an H2 section; strict parsers skip such lines, so we turned it into a mailto: link in this update. Second, npm answered the link check with HTTP 403. That is bot protection rejecting a script, not a broken page, so treat 401/403/429 from big platforms as "check by hand" rather than "broken". For comparison, the proposal's own site passes cleanly: ``` llms.txt: HTTP 200, text/plain; charset=utf-8, 637 bytes, https://llmstxt.org/llms.txt title: llms.txt sections (1): Docs links: 3, of which relative: 0 issues: none llms-full.txt: HTTP 404 robots.txt: HTTP 200 GPTBot /: allowed /llms.txt: allowed OAI-SearchBot /: allowed /llms.txt: allowed ChatGPT-User /: allowed /llms.txt: allowed ClaudeBot /: allowed /llms.txt: allowed Claude-SearchBot /: allowed /llms.txt: allowed Claude-User /: allowed /llms.txt: allowed PerplexityBot /: allowed /llms.txt: allowed Google-Extended /: allowed /llms.txt: allowed Applebot-Extended /: allowed /llms.txt: allowed CCBot /: allowed /llms.txt: allowed ``` Large documentation sites often bend the format. Stripe's docs file (first lines of the output) is about 98 KB with 468 links, H3 subheadings and no blockquote summary: ``` llms.txt: HTTP 200, text/markdown; charset=utf-8, 97564 bytes, https://docs.stripe.com/llms.txt title: Stripe Documentation sections (27): Docs, Payment Methods, Checkout, Payments, Authorization Boost, Link, Architecture and Dashboard, Instructions for Large Language Model Agents: Best Practices for integrating Stripe links: 468, of which relative: 0 issues: no blockquote summary (> ...) right after the H1 (recommended); 5 list items in H2 sections are not '- [name](url): notes'; H3+ headings found (the format uses only H1 and H2) llms-full.txt: HTTP 404 ``` That works fine for an LLM reading it, but a tool that parses the format strictly will miss the parts that deviate. A file that large also takes a meaningful share of an agent's context window, which is the problem llms.txt was meant to avoid; keep the file short and put detail behind the links. Limits of the script: Python's urllib.robotparser applies the first matching rule rather than the longest match that RFC 9309 specifies, and it doesn't support * or $ wildcards in paths. It's fine for the common "block or allow the whole site per bot" case; for complex rule sets, test with a full RFC 9309 parser. ## Which AI user agents to check in robots.txt User agent | Operator | What it is used for (per the operator's docs) | GPTBot | OpenAI | Crawling for model training | OAI-SearchBot | OpenAI | Search results in ChatGPT | ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to | ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | Training, search indexing and user-requested fetches | PerplexityBot | Perplexity | Indexing for Perplexity answers | Google-Extended | Google | A robots.txt control token (not a separate crawler) for use of content in Gemini models; it doesn't affect Google Search | Applebot-Extended | Apple | A control token for use of content in Apple's AI models | CCBot | Common Crawl | Open web crawl that many AI datasets are built from | Blocking a training crawler and allowing a search or user-triggered agent is a common middle ground. Whatever you choose, make sure /llms.txt itself isn't blocked for the agents you want to read it. Operators change their crawler lists from time to time, so check their documentation before you rely on a name. ## Option 2 (API or MCP): check many sites at once The Fairquery Website Tech, SEO & llms.txt Checker (fairquery/website-tech-seo-ai-readiness) checks a list of URLs and returns one row per site: whether llms.txt and llms-full.txt exist (HTML soft-404 pages are not counted), the llms.txt title and link count, robots.txt status for 13 AI user agents, the sitemap, the detected tech stack, security headers, an SEO score and an AI-readiness score. Input: ``` {"urls": ["fairquery.github.io", "llmstxt.org"]} ``` Real output for our site, abridged to the AI-readiness fields (we ran the Actor's published source code locally on October 18, 2026): ``` { "source_url": "https://fairquery.github.io/", "checked_at": "2026-10-18T02:28:21.936Z", "status_code": 200, "robots_txt_found": true, "ai_bots": { "GPTBot": "allowed", "ChatGPT-User": "allowed", "OAI-SearchBot": "allowed", "ClaudeBot": "allowed", "Claude-User": "allowed", "anthropic-ai": "allowed", "PerplexityBot": "allowed", "Google-Extended": "allowed", "CCBot": "allowed", "Bytespider": "allowed", "Applebot-Extended": "allowed", "Meta-ExternalAgent": "allowed", "Amazonbot": "allowed" }, "sitemap_found": true, "sitemap_url_count": 14, "llms_txt_found": true, "llms_txt_url": "https://fairquery.github.io/llms.txt", "llms_txt_title": "Fairquery", "llms_txt_links": 35, "llms_full_txt_found": true, "seo_score": 100, "ai_readiness_score": 100 } ``` llmstxt.org got an AI-readiness score of 80 in the same run (it has no llms-full.txt, among other things). The AI-readiness score is our own heuristic, not a Google or OpenAI metric: llms.txt counts 25 of 100 points, AI-bot access 20, JSON-LD 15, text content 15, sitemap 10, meta description 10 and llms-full.txt 5, with a penalty for noindex. It does not validate the llms.txt structure line by line the way the script above does; use the script for that. Pricing: $0.0015 per site checked ($1.50 per 1,000 sites) plus Apify's $0.00005 Actor-start fee; failed fetches are free. AI agents can call it through the Apify MCP server (https://mcp.apify.com/?tools=fetch-actor-details,fairquery/website-tech-seo-ai-readiness) or through the stdio MCP server in the GitHub repo. To watch a list of sites over time, the free Fairquery n8n templates include an SEO and AI-crawler monitor. ## Checklist for a good llms.txt - Served at /llms.txt with HTTP 200 and a text/plain or text/markdown content type; a missing file should return 404, not an HTML page with status 200. - H1 first, then a one-paragraph blockquote summary that states what the site or product is in plain words. - H2 sections with - [name](url): notes items only; put contact details and other plain facts in the free-text part above the first H2. - Links point to pages that load without JavaScript, ideally Markdown versions; absolute URLs avoid ambiguity. - Short enough to fit easily into a context window; move secondary links to ## Optional. - Not blocked in robots.txt for the agents you care about; there is no submission step. - Kept in sync with the site: regenerate it in your build, as this blog does. ## FAQ ### How do I check if a website has an llms.txt file? Request https:///llms.txt. A valid file returns HTTP 200 as text/plain or text/markdown and starts with an H1 line (# Name). An HTML page or a 404 means there is no usable file. The free Python script in this guide checks this, the structure, the links and robots.txt in one run. ### What makes an llms.txt file valid? Per the llmstxt.org proposal: an H1 with the site or project name (the only required part), then an optional blockquote summary, optional free text without headings, and H2 sections whose lists contain Markdown links in the form - [name](url): notes. An ## Optional section is a convention for secondary links. ### Does llms.txt help SEO or Google rankings? No. Google Search says it does not use llms.txt files and that having one neither helps nor hurts rankings. It can help AI agents and coding assistants that read it, and Chrome Lighthouse has an experimental llms.txt audit, but measured traffic to llms.txt files is low. ### Is llms-full.txt required? No. llms-full.txt (the full text of a site's docs in one file) is a common convention, not part of the llms.txt format. Publish it only if your content is small enough to be useful in one file. ### Which AI crawlers should I check in robots.txt? At least GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Google-Extended and Applebot-Extended are control tokens for AI model use, not separate crawlers. ### How much does the Fairquery Website Tech, SEO & llms.txt Checker cost? $0.0015 per site checked ($1.50 per 1,000 sites) on Apify, pay per event, plus Apify's $0.00005 Actor-start fee. Failed fetches are free. The Python script in this guide is free. ### Can AI agents use it? Yes. Add the Apify MCP server scoped to fairquery/website-tech-seo-ai-readiness to Claude Desktop, Cursor or VS Code, or run the stdio MCP server from the GitHub repo. Source and docs: github.com/fairquery/website-tech-seo-ai-readiness (MIT) · Apify Store · this site's llms.txt · all Fairquery tools --- # How do I use the OSV API in Python to check packages for vulnerabilities (query, batch, pagination)? URL: https://getfairquery.com/posts/osv-api-python-query-batch-vulnerabilities/ Published: 2026-10-18 Short answer: OSV.dev has a free JSON API with no key and, per its docs, no rate limits. In Python, POST one package and exact version to https://api.osv.dev/v1/query to get full advisories, or send many at once to /v1/querybatch, which returns only advisory IDs, then fetch details from GET /v1/vulns/{id}. Handle three traps: follow next_page_token, de-duplicate aliases (one CVE often appears as both a GHSA- and a PYSEC- record), and decide which severity field you trust. The standard-library script below does all of it. If you'd rather get finished rows with computed CVSS scores, a severity filter and a recommended upgrade, the Fairquery OSV Dependency Vulnerability Check costs $0.0005 per package and $0.0015 per vulnerability row. ## The three OSV endpoints you need Endpoint | Returns | Use it for | POST /v1/query | Full advisory objects for one package version (or one commit) | A single lookup | POST /v1/querybatch | Only id and modified per advisory, results in the same order as your queries | Many packages in one request | GET /v1/vulns/{id} | One full advisory: summary, aliases, severity, affected ranges and fixed versions | Details after a batch query | Facts from the OSV API docs worth knowing before you write code: - Rate limits: "Currently there are no limits on the API." Responses are capped at 32 MiB over HTTP/1.1 (no cap over HTTP/2). Be polite anyway: a few parallel requests are plenty. - Exact versions only. Send "version": "2.25.0", not ">=2.0". OSV does not resolve ranges, so a range gives you a wrong answer, not an error. Read versions from a lockfile or pip freeze. - Case-sensitive ecosystems: PyPI, npm, Go, Maven, crates.io, RubyGems and so on. pypi doesn't match. You can use a package URL instead ({"package": {"purl": "pkg:pypi/jinja2@2.11.2"}}), but don't put the version in both the purl and version; that returns 400. - Pagination: /v1/query pages after 1,000 results or 20 seconds; /v1/querybatch pages when one query has more than 1,000 results or the whole batch more than 3,000. Only the affected queries get a next_page_token, and you re-send just those. ## Option 1 (free): one package with /v1/query ``` import json, urllib.request def osv(path, payload=None): req = urllib.request.Request( "https://api.osv.dev/v1/" + path, data=json.dumps(payload).encode() if payload is not None else None, headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req, timeout=30) as r: return json.load(r) res = osv("query", {"package": {"ecosystem": "PyPI", "name": "jinja2"}, "version": "2.11.2"}) for v in res.get("vulns", []): print(v["id"], v.get("aliases", []), v.get("summary", "")[:60]) ``` Real output (October 18, 2026, about 0.2 seconds): ``` GHSA-cpwx-vrp4-4pq7 ['CVE-2025-27516', 'PYSEC-2026-1471'] Jinja2 vulnerable to sandbox breakout through attr filter se GHSA-g3rq-g295-4j3m ['CVE-2020-28493', 'PYSEC-2021-66', 'SNYK-PYTHON-JINJA2-1012994'] Regular Expression Denial of Service (ReDoS) in Jinja2 GHSA-h5c8-rqwp-cp95 ['CVE-2024-22195', 'PYSEC-2026-1473'] Jinja vulnerable to HTML attribute injection when passing us GHSA-h75v-3vvj-5mfj ['CVE-2024-34064', 'PYSEC-2026-1474'] Jinja vulnerable to HTML attribute injection when passing us GHSA-q2x7-8rv6-6q7h ['CVE-2024-56326', 'PYSEC-2026-1475'] Jinja has a sandbox breakout through indirect reference to f PYSEC-2021-66 ['CVE-2020-28493', 'GHSA-g3rq-g295-4j3m', 'SNYK-PYTHON-JINJA2-1012994'] PYSEC-2026-1471 ['CVE-2025-27516', 'GHSA-cpwx-vrp4-4pq7'] Jinja2 vulnerable to sandbox breakout through attr filter se PYSEC-2026-1473 ['CVE-2024-22195', 'GHSA-h5c8-rqwp-cp95'] Jinja vulnerable to HTML attribute injection when passing us PYSEC-2026-1474 ['CVE-2024-34064', 'GHSA-h75v-3vvj-5mfj'] Jinja vulnerable to HTML attribute injection when passing us PYSEC-2026-1475 ['CVE-2024-56326', 'GHSA-q2x7-8rv6-6q7h'] Jinja has a sandbox breakout through indirect reference to f ``` Ten records, but only five vulnerabilities: each CVE appears once as a GitHub advisory (GHSA-…) and once as a PyPA advisory (PYSEC-…), and each lists the other in aliases. Count records naively and you double your findings. ## Option 1b (free): a whole requirements file with querybatch This script reads exact pins from requirements.txt, sends them to querybatch in chunks of 1,000, follows next_page_token per query, fetches each unique advisory once with 8 parallel requests, collapses aliases (preferring the GHSA record), prints the CVE, the GHSA severity label and the fixed versions, and exits with code 1 if anything was found, so you can use it as a CI gate: ``` """Check pinned packages against OSV.dev: querybatch + pagination + details + alias de-dup. Standard library only (Python 3.9+). Usage: python3 osv_batch.py requirements.txt""" import json, sys, urllib.request from concurrent.futures import ThreadPoolExecutor API = "https://api.osv.dev/v1/" def osv(path, payload=None): req = urllib.request.Request(API + path, headers={"Content-Type": "application/json"}, data=json.dumps(payload).encode() if payload is not None else None) with urllib.request.urlopen(req, timeout=60) as r: return json.load(r) def read_pins(path, ecosystem="PyPI"): pins = [] for line in open(path): line = line.split("#")[0].strip() if "==" in line: # only exact pins: OSV needs a concrete version name, version = line.split("==", 1) pins.append({"package": {"ecosystem": ecosystem, "name": name.strip()}, "version": version.strip()}) elif line: print(f"skipped (not pinned with ==): {line}", file=sys.stderr) return pins def query_batch(queries): """Return one list of vuln IDs per query, following next_page_token where present.""" ids = [[] for _ in queries] for start in range(0, len(queries), 1000): # keep batches at 1,000 queries todo = {i: dict(q) for i, q in enumerate(queries[start:start + 1000], start)} while todo: order = list(todo) res = osv("querybatch", {"queries": [todo[i] for i in order]})["results"] nxt = {} for i, r in zip(order, res): ids[i] += [v["id"] for v in r.get("vulns", [])] if r.get("next_page_token"): nxt[i] = {**todo[i], "page_token": r["next_page_token"]} todo = nxt return ids def fixed_versions(vuln, name): out = set() for aff in vuln.get("affected", []): if aff.get("package", {}).get("name", "").lower() != name.lower(): continue for rng in aff.get("ranges", []): out.update(e["fixed"] for e in rng.get("events", []) if "fixed" in e) return sorted(out) def main(path): pins = read_pins(path) ids = query_batch(pins) unique = sorted({i for row in ids for i in row}) with ThreadPoolExecutor(max_workers=8) as pool: # be polite: a few parallel requests details = dict(zip(unique, pool.map(lambda i: osv("vulns/" + i), unique))) findings = 0 for pin, row in zip(pins, ids): name, version = pin["package"]["name"], pin["version"] seen, groups = set(), [] for vid in sorted(row, key=lambda x: (not x.startswith("GHSA-"), x)): # prefer GHSA records v = details[vid] group = {vid, *v.get("aliases", [])} if group & seen: continue # same advisory under another ID (e.g. PYSEC-... vs GHSA-...) seen |= group groups.append(v) print(f"{name}=={version}: {len(groups)} advisories ({len(row)} OSV IDs)") for v in groups: cve = next((a for a in v.get("aliases", []) if a.startswith("CVE-")), "-") sev = v.get("database_specific", {}).get("severity", "n/a") fix = ", ".join(fixed_versions(v, name)) or "none listed" print(f" {v['id']:<20} {cve:<15} {sev:<9} fixed: {fix}") findings += len(groups) return 1 if findings else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1])) ``` Input file: ``` requests==2.25.0 jinja2==2.11.2 urllib3==2.2.3 flask>=2.0 ``` Real output (October 18, 2026; 1 batch request plus 34 detail requests, under 1 second end to end in our runs): ``` skipped (not pinned with ==): flask>=2.0 requests==2.25.0: 4 advisories (8 OSV IDs) GHSA-9hjg-9r4m-mvj7 CVE-2024-47081 MODERATE fixed: 2.32.4 GHSA-9wx4-h78v-vm56 CVE-2024-35195 MODERATE fixed: 2.32.0 GHSA-gc5v-m9x4-r6x2 CVE-2026-25645 MODERATE fixed: 2.33.0 GHSA-j8r2-6x86-q33q CVE-2023-32681 MODERATE fixed: 2.31.0 jinja2==2.11.2: 5 advisories (10 OSV IDs) GHSA-cpwx-vrp4-4pq7 CVE-2025-27516 MODERATE fixed: 3.1.6 GHSA-g3rq-g295-4j3m CVE-2020-28493 MODERATE fixed: 2.11.3 GHSA-h5c8-rqwp-cp95 CVE-2024-22195 MODERATE fixed: 3.1.3 GHSA-h75v-3vvj-5mfj CVE-2024-34064 MODERATE fixed: 3.1.4 GHSA-q2x7-8rv6-6q7h CVE-2024-56326 MODERATE fixed: 3.1.5 urllib3==2.2.3: 8 advisories (16 OSV IDs) GHSA-2xpw-w6gg-jr37 CVE-2025-66471 HIGH fixed: 2.6.0 GHSA-38jv-5279-wg99 CVE-2026-21441 HIGH fixed: 2.6.3 GHSA-48p4-8xcf-vxj5 CVE-2025-50182 MODERATE fixed: 2.5.0 GHSA-8988-9cw3-xx77 CVE-2026-97687 HIGH fixed: 2.8.0 GHSA-gm62-xv2j-4w53 CVE-2025-66418 HIGH fixed: 2.6.0 GHSA-pq67-6m6q-mj2v CVE-2025-50181 MODERATE fixed: 2.5.0 GHSA-qccp-gfcp-xxvc CVE-2026-44431 HIGH fixed: 2.7.0 GHSA-vxq7-64xx-v4gw CVE-2026-97689 HIGH fixed: 2.8.0 exit=1 ``` Three things to read from it. The unpinned flask>=2.0 line is skipped with a warning rather than guessed. Raw OSV IDs (34) collapse to 17 distinct advisories. And "fixed" is a list per advisory: for packages with several release lines (Django, for example, lists fixes such as 4.2.28, 5.2.11, 6.0.2) pick the fix on your own line, or the highest one if you can upgrade freely. ## Trap: which severity do you trust? OSV records can carry several severity signals: a severity array with CVSS vectors (CVSS_V3, CVSS_V4) and, for GitHub advisories, a database_specific.severity label. They don't always agree. In the run above, GitHub labels GHSA-q2x7-8rv6-6q7h (Jinja sandbox breakout) as MODERATE, but its CVSS 3.1 vector AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H computes to a base score of 7.8, which is HIGH. For GHSA-qccp-gfcp-xxvc (urllib3) it's the other way round: the label is HIGH while its CVSS 3.1 vector scores 5.3 (MEDIUM). OSV gives you vectors, not scores, so computing a number needs a CVSS calculator (for example the cvss package on PyPI). Pick one rule, document it, and apply it consistently. A CI gate on "HIGH or above" will pass or fail different advisories depending on the field you read. ## Option 2 (API, MCP or CI): finished rows The Fairquery OSV Dependency Vulnerability Check (fairquery/osv-dependency-vulnerability-check) wraps the same endpoints. It parses requirements.txt, package.json, package-lock.json (v1 to v3) and go.mod, computes the CVSS v3 base score when a v3 vector exists and otherwise falls back to the database label, filters by minSeverity, de-duplicates aliases and recommends an upgrade version. Input for the same three pinned packages: ``` {"manifestText": "requests==2.25.0\njinja2==2.11.2\nurllib3==2.2.3\n", "minSeverity": "HIGH"} ``` We ran the Actor's published source locally on October 18, 2026. It made 35 OSV requests and returned 3 package rows and 6 vulnerability rows (17 alias duplicates removed, 11 advisories below HIGH filtered out). One vulnerability row, abridged: ``` { "type": "vulnerability", "id": "GHSA-q2x7-8rv6-6q7h", "ecosystem": "PyPI", "package_name": "jinja2", "package_version": "2.11.2", "aliases": [ "CVE-2024-56326", "PYSEC-2026-1475" ], "cve_ids": [ "CVE-2024-56326" ], "summary": "Jinja has a sandbox breakout through indirect reference to format method", "severity": "HIGH", "severity_source": "cvss", "cvss_score": 7.8, "cvss_vector": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", "cvss_version": "3.1", "fixed_versions": [ "3.1.5" ], "osv_url": "https://osv.dev/vulnerability/GHSA-q2x7-8rv6-6q7h" } ``` Note severity_source: "cvss": this row is HIGH because of the computed 7.8, even though GitHub's label says MODERATE, which is exactly the disagreement described above. The same run would cost 3 × $0.0005 + 6 × $0.0015 = $0.0105, plus Apify's $0.00005 Actor-start fee. For a GitHub Actions job that fails the build on HIGH or CRITICAL, and for the free osv_query_packages MCP tool, see how to check npm and PyPI dependencies for vulnerabilities in CI with OSV. ## Which option should you use? Option | Good for | Trade-offs | Your own Python against the OSV API (above) | Free, no dependencies, full control over severity rules | You maintain manifest parsing, alias handling, CVSS scoring and pagination | Google's OSV-Scanner CLI | Free, official, scans lockfiles, SBOMs and container images locally | A binary to install in CI; not a hosted API or MCP tool | Fairquery OSV Actor | One API or MCP call from agents, n8n or CI; manifest parsing, computed CVSS, severity filter, recommended version as JSON rows, pay per row | Paid per package and vulnerability; needs an Apify account; same OSV data, no reachability analysis | ## FAQ ### Is the OSV API free and does it have rate limits? It is free and needs no API key. The OSV docs state that there are currently no limits on the API; responses are capped at 32 MiB over HTTP/1.1 and uncapped over HTTP/2. Keep parallel requests modest anyway. ### What is the difference between /v1/query and /v1/querybatch? /v1/query returns full advisory objects for one package version or commit. /v1/querybatch takes many queries and returns only the advisory id and modified time for each, in input order; fetch details with GET /v1/vulns/{id}. ### How does OSV pagination work? A response that has more results includes next_page_token. Re-send the same query with page_token set to it until no token comes back. In querybatch only some queries may be paginated, so re-send just those. ### Why does OSV return the same vulnerability twice? Several databases publish the same issue: a GitHub advisory (GHSA-...) and a PyPA advisory (PYSEC-...) for one CVE, each listing the other in aliases. Group records by their aliases before counting. ### Can I query OSV with a version range like >=2.0? No. OSV matches exact versions. Send concrete versions from a lockfile or pip freeze; a range will not give a correct answer. ### How much does the Fairquery OSV Actor cost? $0.0005 per package checked plus $0.0015 per vulnerability row emitted after the severity filter and alias de-duplication, plus Apify's $0.00005 Actor-start fee. Error and warning rows are free. The Python scripts in this guide are free. No result is not proof of safety. An empty response means OSV has no matching advisory for that exact version. No database is complete, and none of these options checks whether the vulnerable code is reachable in your application. Source and docs: github.com/fairquery/osv-dependency-vulnerability-check (MIT) · Apify Store · OSV API docs · all Fairquery tools --- # How do I get live job listings straight from company ATS boards (Greenhouse, Lever, Ashby) via API? URL: https://getfairquery.com/posts/live-job-listings-api-greenhouse-lever-ashby/ Published: 2026-10-17 Short answer: most tech and many enterprise employers publish their openings through an applicant-tracking system (ATS) that has a free, public, documented job-board API: Greenhouse (boards-api.greenhouse.io/v1/boards/{company}/jobs), Lever (api.lever.co/v0/postings/{company}) and Ashby (api.ashbyhq.com/posting-api/job-board/{company}). No key is needed. Reading jobs there gives you listings that are currently published on the employer's own hiring system, not copies on an aggregator. That is not proof the role is being filled, so check age and reposts too. For many companies, history and repost signals in one call, the Fairquery Company Jobs Feed returns normalized JSON rows for $0.99 per 1,000 jobs + $0.0005 per board scanned, and its MCP server has four free local tools. ## Why read jobs straight from the company's ATS? Job aggregators and scraped job-board copies lag behind the source. A role can be filled and removed from the employer's ATS while copies of it stay up elsewhere. If you are building a job alert, a hiring-signal feed, a sales trigger ("company X opened 12 data roles") or an agent that answers "who is hiring Rust engineers in Berlin right now?", the employer's own ATS board is the closest public source of truth: if the job is listed there now, the employer is publishing it now. It is not a ghost-job detector on its own, though. Greenhouse itself reports that 18–22% of jobs posted on its platform in a given quarter are ghost jobs, so "published on the ATS" and "actively hiring" are different things. What you can measure from public data is how old a posting is, whether it has been open unusually long, and whether the same role was closed and posted again. Those are the signals this guide uses. ## The three public job-board APIs ATS | Endpoint (no key) | What you get | Docs | Greenhouse | https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true | All published jobs with title, location, departments, offices, first_published, updated_at, HTML description. Per-job: /jobs/{id} (404 when closed) | Job Board API | Lever | https://api.lever.co/v0/postings/{company}?mode=json | Postings with team, department, location, workplaceType, createdAt, optional salary range. Per-job: /{id} | Postings API | Ashby | https://api.ashbyhq.com/posting-api/job-board/{name}?includeCompensation=true | Jobs with department, location, remote flag, publishedAt, compensation summary. No per-job endpoint | Job Postings API | The {company} part is the board slug from the company's careers links (boards.greenhouse.io/stripe, jobs.lever.co/palantir and jobs.ashbyhq.com/ramp). Unknown slugs return 404. What we left out and why. SmartRecruiters has a public Posting API, but the robots.txt on its API host allows only LinkedIn's crawler, so we don't call it. Workday has no public job API. Recruitee's Careers Site API needs a token from the employer. Workable documents its public endpoints only for an account's own careers page. LinkedIn is not used at all. We stick to APIs that are documented for outside use. ## Option 1 (free): three ATS APIs in 25 lines of Python Standard library only. It reads Stripe (Greenhouse), Palantir (Lever) and Ramp (Ashby), and prints engineering roles with their age: ``` # Free: read live jobs straight from three ATS job-board APIs (Python standard library only). import json, urllib.request from datetime import datetime, timezone UA = {"User-Agent": "my-job-research/1.0 (you@example.com)"} def get(url): with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=15) as r: return json.load(r) now = datetime.now(timezone.utc) def age(iso): return (now - datetime.fromisoformat(iso.replace("Z", "+00:00"))).days if iso else None gh = get("https://boards-api.greenhouse.io/v1/boards/stripe/jobs")["jobs"] # Greenhouse lv = get("https://api.lever.co/v0/postings/palantir?mode=json") # Lever ab = get("https://api.ashbyhq.com/posting-api/job-board/ramp")["jobs"] # Ashby rows = [("greenhouse", j["title"], j["location"]["name"], j.get("first_published")) for j in gh] rows += [("lever", j["text"], j["categories"].get("location"), datetime.fromtimestamp(j["createdAt"] / 1000, timezone.utc).isoformat()) for j in lv] rows += [("ashby", j["title"], j["location"], j.get("publishedAt")) for j in ab if j.get("isListed", True)] print(len(gh), "Stripe /", len(lv), "Palantir /", len(ab), "Ramp jobs") for ats, title, loc, posted in [r for r in rows if "engineer" in r[1].lower()][:4]: print(f"{ats:10} {age(posted):>4}d {title[:48]:48} {loc}") ``` Real output (October 17, 2026, about 1 second for the three requests): ``` 732 Stripe / 310 Palantir / 167 Ramp jobs greenhouse 38d Abuse Research Engineer Remote from the US greenhouse 10d AI Engineer Chicago greenhouse 18d Android BSP Engineer Taipei, Taiwan greenhouse 5d Android Engineer, Terminal - CA Toronto, CA ``` That works for a handful of boards. At scale you run into the parts that take the time: - Three schemas. Dates are ISO strings on Greenhouse and Ashby and epoch milliseconds on Lever. Remote is a flag, a workplaceType or free text depending on the system. Countries are structured on Lever and Ashby and free text on Greenhouse. - Finding the board. You need the slug, and many companies link to it only from a careers page. - Change tracking. "New since yesterday", "closed", and "reposted" need stored history per board. A repost usually comes back under a new job id, so id-based diffs miss it. - Politeness. Lever's robots.txt asks for a 1-second crawl delay, very large Greenhouse boards can exceed 40 MB with descriptions, and any 403 or 429 means you should stop, not retry harder. ## Ghost jobs and dead listings: the signals you can measure Signal | How it is computed | What it does not prove | verified_open_at | Time the job was read from the employer's ATS (or re-checked on its per-job endpoint) | That the role is still being filled | posted_at, age_days | The ATS publish date (null when the ATS doesn't give one, never guessed); age falls back to first-seen and says so in age_basis | Some employers refresh dates when they edit a posting | long_open | age_days above a threshold you set (default 60) | Evergreen and hard-to-fill roles are legitimately open for months | reposted, repost_of | Same company + normalized title + location as a job that closed in the last 90 days, under a new id | Needs at least two tracked runs | change, first_seen_at, last_seen_at | Per-board history: new, updated, unchanged, closed | Closures are only reported when the whole board was read | Example from our test runs: Hightouch's Ashby board had one listing, a "Backend Engineer, Platform" role with posted_at 2021-03-16, so age_days 2041 and long_open: true. That is very likely an evergreen posting, which is exactly what the flag is for. ## Option 2 (API or MCP): one feed across companies The Fairquery Company Jobs Feed Actor (fairquery/company-jobs-feed) calls only the three documented API hosts above. You give it company names, domains, careers URLs or exact boards; it finds the board (careers-page links with robots.txt respected, then at most six slug guesses), reads it, normalizes every job into one schema, applies your filters, de-duplicates, and keeps per-board history in a key-value store in your own Apify account. We ran it on October 17, 2026 on Stripe's Greenhouse board, filtered to engineering roles in San Francisco or remote, with descriptions and a per-job re-check for the first 10: ``` { "boards": [ "greenhouse:stripe" ], "titleIncludes": [ "engineer" ], "locations": [ "remote", "san francisco" ], "includeDescription": "text", "confirmOpen": "detail", "maxOpenChecks": 10 } ``` Real output, abridged to the first Stripe job, one Ramp job from a second run (remote: "only", hasSalary: true) and the Stripe board row: ``` [ { "type": "job", "job_key": "greenhouse:stripe:8172503", "company": "Stripe", "title": "Abuse Research Engineer", "department": "8611 Security Analytics", "locations": [ "Remote from the US" ], "countries": [ "US" ], "remote": true, "workplace_type": "remote", "posted_at": "2026-09-09T14:52:09.000Z", "age_days": 38, "age_basis": "posted_at", "long_open": false, "verified_open_at": "2026-10-17T22:54:04.083Z", "first_seen_at": "2026-10-17T22:54:03.320Z", "change": "new", "reposted": false, "open_check": "detail_open", "compensation": null, "apply_url": "https://stripe.com/jobs/search?gh_jid=8172503", "fingerprint": "d7127148fd018f2c" }, { "type": "job", "job_key": "ashby:ramp:34413f8d-26bf-4bbc-8ade-eb309a0e2245", "company": "Ramp", "title": "Security Engineer, Cloud", "department": "Engineering", "locations": [ "New York, NY (HQ)", "Remote (Canada)", "Remote (US)", "Miami, FL" ], "countries": [ "US", "CA" ], "remote": true, "workplace_type": "hybrid", "posted_at": "2026-04-07T17:12:35.753Z", "age_days": 193, "age_basis": "posted_at", "long_open": true, "verified_open_at": "2026-10-17T22:54:03.924Z", "first_seen_at": "2026-10-17T22:54:03.924Z", "change": "new", "reposted": false, "open_check": "listed", "compensation": { "summary": "$211.4K – $290.6K • Offers Equity", "min": 211400, "max": 290600, "currency": "USD", "interval": "1 YEAR" }, "apply_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application", "fingerprint": "646e3148231ba20d" }, { "type": "board", "ats": "greenhouse", "board": "stripe", "company": "Stripe", "jobs_total": 732, "jobs_matched": 32, "jobs_emitted": 32, "complete": true, "new": 732, "updated": 0, "closed": 0, "reposted": 0, "requests": 1, "checked_at": "2026-10-17T22:54:03.320Z" } ] ``` The Stripe run took 3.1 seconds, read 732 jobs in one request, matched 32 and re-checked 10 on Greenhouse's per-job endpoint (open_check: "detail_open"). It cost 32 × $0.00099 + 1 × $0.0005 + 10 × $0.00029 = $0.035. ### "What's new since yesterday?" with onlyNew Give each saved search its own stateStoreName and set onlyNew: true. The first run is the baseline; later runs return only new or changed jobs, plus free closed rows. Re-running Stripe and Ramp a few minutes later returned no job rows, only the two board rows ($0.0005 each): ``` { "type": "board", "ats": "greenhouse", "board": "stripe", "jobs_total": 732, "jobs_emitted": 0, "new": 0, "updated": 0, "closed": 0, "reposted": 0, "checked_at": "2026-10-17T23:04:29.186Z" } ``` ### Company discovery Input {"companies": ["gem.com", "vanta.com", "sentry.io"]}: Sentry's board was found from links on sentry.io's own careers page (confidence: "high"), Vanta's by a slug probe ("medium": the slug exists but could in principle be another company with the same name), and gem.com returned a free NO_BOARD_FOUND row because it doesn't use one of the three systems publicly: ``` [ { "type": "company", "input": "sentry.io", "company": null, "domain": "sentry.io", "boards": [ { "ats": "ashby", "slug": "sentry", "via": "careers_page", "confidence": "high" } ], "careers_pages_fetched": [ "https://sentry.io/welcome/", "https://sentry.io/careers/" ], "probes": 0, "cached": false, "source_url": "https://sentry.io/welcome/", "checked_at": "2026-10-18T00:11:58.387Z" }, { "type": "error", "checked_at": "2026-10-18T00:11:58.570Z", "code": "NO_BOARD_FOUND", "message": "No Greenhouse, Lever or Ashby board found for \"gem.com\" (3 careers pages, 3 API probes).", "retryable": false, "input": "gem.com", "source_url": null }, { "type": "company", "input": "vanta.com", "company": null, "domain": "vanta.com", "boards": [ { "ats": "ashby", "slug": "vanta", "via": "slug_probe", "confidence": "medium" } ], "careers_pages_fetched": [ "https://www.vanta.com/", "https://www.vanta.com/company/careers" ], "probes": 3, "cached": false, "source_url": "https://www.vanta.com/", "checked_at": "2026-10-18T00:12:01.280Z" } ] ``` ### From Python, curl or an AI agent With the official apify-client package, as a daily US engineering monitor with a spend cap: ``` # pip install apify-client (set APIFY_TOKEN) import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("fairquery/company-jobs-feed").call(run_input={ "companies": ["stripe.com", "ramp.com", "https://jobs.lever.co/palantir"], "titleIncludes": ["engineer"], "countries": ["US"], "postedWithinDays": 30, "stateStoreName": "my-us-engineering-search", "onlyNew": True, # from the 2nd run on: only new or changed jobs (+ free closed rows) "maxJobs": 2000, # spend cap: at most 2,000 billed job rows }) for row in client.dataset(run["defaultDatasetId"]).iterate_items(): if row["type"] == "job": print(row["company"], row["title"], row["age_days"], row["long_open"], row["apply_url"]) ``` Or one synchronous HTTP call that returns the rows directly (Palantir's Lever board, US engineering jobs posted in the last 180 days; our test run read 310 Palantir postings in one request and matched 36): ``` curl -s -X POST "https://api.apify.com/v2/acts/fairquery~company-jobs-feed/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \ -d '{"companies": ["https://jobs.lever.co/palantir"], "countries": ["US"], "titleIncludes": ["engineer"], "postedWithinDays": 180}' ``` For agents there are two routes. The Apify MCP server scoped to this Actor (https://mcp.apify.com/?tools=fairquery/company-jobs-feed) works in Claude Desktop, Cursor and VS Code. Or run the stdio MCP server @fairquery/company-jobs-feed-mcp (source in the GitHub repo): jobs_get_board, jobs_find_board, jobs_check_open and jobs_list_seed_boards run free on your machine with no token, and jobs_search_companies runs the Actor with your APIFY_TOKEN. ``` { "mcpServers": { "fairquery-jobs-feed": { "command": "npx", "args": ["-y", "@fairquery/company-jobs-feed-mcp"], "env": { "APIFY_TOKEN": "only needed for the paid jobs_search_companies tool" } } } } ``` Then ask, for example: "Which remote engineering jobs does Ramp have open right now, with salary ranges? Skip anything open for more than 90 days." ## Filters - titleIncludes / titleExcludes (whole words, so engineer doesn't match "Engineering Manager" unless you ask for it), or a safe titleRegex - locations (substring), countries (ISO codes, structured data where the ATS has it), remote: any / only / exclude - companyFilter, departments, employmentTypes, hasSalary - postedWithinDays and maxAgeDays; excludeStale drops long-open and reposted rows - dedupe: exact (one row per job id; the same role on several boards is grouped by a fingerprint and marked duplicate_of, never silently dropped) or role (one board's rows with the same title and department become one row listing all locations; merged rows are free) ## Pricing Option | Price | The ATS APIs directly (your own code) | Free | Fairquery MCP server, local tools (jobs_get_board, jobs_find_board, jobs_check_open, jobs_list_seed_boards) | Free | Fairquery Company Jobs Feed (Apify, pay per event) | $0.99 per 1,000 jobs delivered ($0.00099 each), $0.0005 per board scanned, $0.00029 per optional per-job re-check, $0.0019 per company found by discovery. Error, closed and merged rows, cached and starter-list lookups, and runs with empty input are free. | Examples: every open job at Stripe, Palantir and Ramp (1,209 in our unfiltered test) costs 1,209 × $0.00099 + 3 × $0.0005 = about $1.20. A daily monitor of 50 companies with onlyNew that finds 20 new jobs a day costs 50 × $0.0005 + 20 × $0.00099 = about $0.045 per day. maxJobs and Apify's "max cost per run" cap every run; the Actor stops cleanly at the cap and charges nothing above it. ## How it compares Option | Good for | Trade-offs | ATS APIs directly | Free and official; ideal for a few known boards | Three schemas, slug discovery, history, repost detection and rate-limit handling are yours to build | Job aggregator APIs and scraped job-board datasets | Broad coverage across employers and ATSs, including ones without public APIs | Second-hand copies that can lag behind or outlive the original posting; terms vary by source | Fairquery Company Jobs Feed | Jobs read live from employers' own Greenhouse, Lever and Ashby boards; one schema; age, long-open and repost signals; history; MCP tools; pay per job | Only these three systems (no SmartRecruiters, Workday, Workable, Recruitee or LinkedIn); reposts need two or more runs; slug-probe matches are medium confidence | ## FAQ ### Is there a free API for live job listings? Yes, per employer. Greenhouse (boards-api.greenhouse.io), Lever (api.lever.co/v0/postings) and Ashby (api.ashbyhq.com/posting-api/job-board) publish documented job-board APIs that need no key. You need each company's board slug, and you get only that company's published jobs. ### How do I find a company's Greenhouse, Lever or Ashby board? Look for links on its careers page such as boards.greenhouse.io/, jobs.lever.co/ or jobs.ashbyhq.com/. The Fairquery jobs_find_board tool (free) and the Company Jobs Feed Actor do this for you: careers-page links first (robots.txt respected), then at most six slug guesses against the APIs. ### Does this filter out ghost jobs? No tool can prove hiring intent from public data, and Greenhouse reports that 18 to 22% of jobs on its own platform in a given quarter are ghost jobs. What the feed gives you is verified_open_at (published on the employer's ATS right now), posted_at and age_days, a long_open flag, and repost detection across runs, so you can filter out the listings most likely to be stale. ### How fresh is the data? Every run reads the employer's ATS live; nothing is served from a cache of job data. verified_open_at is the time of that read. With confirmOpen the Actor also re-checks each job on its per-job endpoint (Greenhouse, Lever) for $0.00029 per job. ### Which ATS systems are supported? Greenhouse, Lever and Ashby. SmartRecruiters is left out because the robots.txt on its API host allows only LinkedIn's crawler. Workday has no public job API, Recruitee's careers API needs an employer token, and Workable documents its public endpoints only for an account's own careers page, so they are not included. LinkedIn is not used. ### How much does the Fairquery Company Jobs Feed cost? $0.99 per 1,000 jobs delivered, $0.0005 per board scanned, $0.00029 per optional per-job re-check and $0.0019 per company found by discovery, pay per event on Apify. Error, closed and merged rows and empty-input runs are free. The four local MCP tools are free. ### Can AI agents use it? Yes. Add the Apify MCP server scoped to fairquery/company-jobs-feed, or run the stdio MCP server from the GitHub repo: jobs_get_board, jobs_find_board, jobs_check_open and jobs_list_seed_boards run free on your machine, and jobs_search_companies runs the Actor with your Apify token. ### Is it allowed to use these APIs? They are public job-board APIs that the ATS vendors document for publishing an employer's jobs. The Fairquery Actor calls only those documented hosts, without login, at most two concurrent requests per host, honoring Lever's crawl delay, and it stops calling a host on the first 403 or 429. Job content belongs to the employers; check that your use (for example republishing) fits their terms. About the data. Jobs belong to the employers and are returned as published, with light normalization. Age, remote, country, long-open and repost fields are signals, labeled as such, not guarantees that a role is being hired for. Fairquery is not affiliated with Greenhouse, Lever or Ashby. Source and docs: github.com/fairquery/company-jobs-feed (MIT) · Apify Store · all Fairquery tools --- # How do I get SEC EDGAR company facts (XBRL financials) and filings by ticker via API? URL: https://getfairquery.com/posts/sec-edgar-api-company-facts-xbrl-filings-by-ticker/ Published: 2026-10-17 Short answer: the SEC publishes free JSON APIs with no key: map a ticker to its 10-digit CIK with company_tickers.json, then read data.sec.gov/api/xbrl/companyfacts/CIK##########.json for XBRL financials (revenue, net income, EPS, assets) and data.sec.gov/submissions/CIK##########.json for the filing list (10-K, 10-Q, 8-K). Send a User-Agent with your contact details and stay under 10 requests per second. Watch two traps: values are repeated across filings, and concepts change over time (Apple's Revenues stops in 2018). If you want clean rows instead, the Fairquery SEC EDGAR Actor resolves tickers, CIKs or names and returns filings and the latest facts as JSON for $0.0015 per company, $0.001 per filing row and $0.0002 per fact row. ## The free SEC EDGAR APIs you need What | Endpoint | Notes | Ticker → CIK | https://www.sec.gov/files/company_tickers.json | One file with every listed ticker, its CIK and company name. Pad the CIK to 10 digits for the other endpoints. | Filings and company metadata | https://data.sec.gov/submissions/CIK##########.json | Name, SIC code, exchanges, fiscal year end and the recent filings (form, dates, accession number, primary document). | All XBRL facts of one company | https://data.sec.gov/api/xbrl/companyfacts/CIK##########.json | Every reported value of every concept (us-gaap, dei), by unit, with period start/end, form, fiscal period and filing date. | One concept of one company | https://data.sec.gov/api/xbrl/companyconcept/CIK##########/us-gaap/.json | Smaller download when you need a single line item. | One concept across companies | https://data.sec.gov/api/xbrl/frames/us-gaap//USD/CY2025Q2.json | One value per company for a calendar period, useful for screens and comparisons. | No API key or account is needed. The SEC's fair-access rules ask every client to declare a User-Agent that identifies you (for example a name and email) and to stay at or below 10 requests per second. Requests without a proper User-Agent are often refused. ## Option 1 (free): SEC company facts in Python Standard library only. It looks up Apple's CIK and prints the three most recent revenue values: ``` import json, urllib.request UA = {"User-Agent": "Your Name you@example.com"} # SEC asks every client to identify itself def get(url): with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r: return json.load(r) # 1. Ticker -> 10-digit CIK tickers = get("https://www.sec.gov/files/company_tickers.json") cik = next(f"{t['cik_str']:010d}" for t in tickers.values() if t["ticker"] == "AAPL") # 2. All XBRL facts for the company, then the latest values of one concept facts = get(f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json") rows = facts["facts"]["us-gaap"]["RevenueFromContractWithCustomerExcludingAssessedTax"]["units"]["USD"] rows = sorted(rows, key=lambda f: (f["end"], f["filed"]), reverse=True) for f in rows[:3]: print(f["start"], f["end"], f["val"], f["form"], f["fp"], f["filed"], f.get("frame", "")) ``` Real output (October 17, 2026, about 0.4 s for both requests). Columns: period start, period end, value in USD, form, fiscal period, filing date, frame: ``` 2025-09-28 2026-06-27 364357000000 10-Q Q3 2026-07-31 2026-03-29 2026-06-27 109417000000 10-Q Q3 2026-07-31 CY2026Q2 2025-09-28 2026-03-28 254940000000 10-Q Q2 2026-05-01 ``` The first row is the nine-month year-to-date figure, the second is the single quarter (April to June 2026), and the third is the previous quarter's year-to-date value. To build a clean quarterly or annual series you have to tell those apart, and handle the two traps below. ## Trap 1: the same value appears many times Each 10-Q and 10-K repeats earlier periods as comparatives, and companyfacts lists the value once per filing that reported it. If you sort only by filing date and take the top rows, you get last year's comparatives from the newest 10-Q instead of the current quarter. (We found exactly this in our own Actor and fixed it; see below.) Two ways out: - Sort by period end first, then by filed, and keep one value per (start, end) pair, from the latest filing so restatements win. - Or use the frame field. The SEC sets it (for example CY2026Q2) on one value per company and calendar period, so filtering to facts that have a frame gives one value per period. Year-to-date values that don't match a calendar quarter or year have no frame. ## Trap 2: the concept you ask for may have been retired Companies choose XBRL concepts, and the choices change. Apple reported revenue as Revenues until fiscal 2018; its filings since January 2019 (when the ASC 606 revenue standard took effect for it) use RevenueFromContractWithCustomerExcludingAssessedTax. Ask for Revenues and you silently get 2018 data. In practice, request both concepts (and check the period dates), or look at which concepts a company actually uses in its latest filing. The same applies to other line items, such as net income attributable to the parent versus including non-controlling interests. ## Option 2 (API or MCP): filings and facts as clean rows The Fairquery SEC EDGAR Company Facts & Filings Actor (fairquery/edgar-company-facts-filings) calls the same official endpoints, throttled to at most 5 requests per second with a declared User-Agent. It takes tickers (AAPL), CIKs (320193) or name fragments (nvidia), and emits one row per company, filing, fact, optional text excerpt or error. For facts it sorts by period end, then filing date, and returns a re-reported value only once, so maxFactsPerMetric: 2 gives you the two most recent periods. We ran it on October 17, 2026 with this input, asking for both revenue concepts on purpose: ``` { "companies": [ "AAPL" ], "mode": "both", "formTypes": [ "10-K" ], "maxFilingsPerCompany": 2, "metrics": [ "Revenues", "RevenueFromContractWithCustomerExcludingAssessedTax", "NetIncomeLoss", "EarningsPerShareDiluted" ], "maxFactsPerMetric": 2 } ``` Real output (run time about 2 seconds; 11 rows: 1 company, 2 filings, 8 facts). Abridged to the company row, the latest 10-K and the revenue facts: ``` [ { "schema_version": "1.0.0", "type": "company", "source_url": "https://data.sec.gov/submissions/CIK0000320193.json", "scraped_at": "2026-10-17T20:09:52.368Z", "query": "AAPL", "cik": "0000320193", "ticker": "AAPL", "name": "Apple Inc.", "entity_type": "operating", "sic": "3571", "sic_description": "Electronic Computers", "exchanges": [ "Nasdaq" ], "ein": "942404110", "fiscal_year_end": "0926" }, { "schema_version": "1.0.0", "type": "filing", "source_url": "https://data.sec.gov/submissions/CIK0000320193.json", "scraped_at": "2026-10-17T20:09:52.368Z", "cik": "0000320193", "ticker": "AAPL", "name": "Apple Inc.", "form": "10-K", "filing_date": "2025-10-31", "report_date": "2025-09-27", "accession_number": "0000320193-25-000079", "primary_document": "aapl-20250927.htm", "primary_doc_description": "10-K", "document_url": "https://www.sec.gov/Archives/edgar/data/320193/000032019325000079/aapl-20250927.htm", "filing_url": "https://www.sec.gov/Archives/edgar/data/320193/000032019325000079/", "is_xbrl": true, "is_inline_xbrl": true, "size": 9392337 }, { "schema_version": "1.0.0", "type": "fact", "source_url": "https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json", "scraped_at": "2026-10-17T20:09:52.659Z", "cik": "0000320193", "ticker": "AAPL", "name": "Apple Inc.", "taxonomy": "us-gaap", "concept": "RevenueFromContractWithCustomerExcludingAssessedTax", "label": "Revenue from Contract with Customer, Excluding Assessed Tax", "unit": "USD", "value": 109417000000, "end": "2026-06-27", "start": "2026-03-29", "filed": "2026-07-31", "form": "10-Q", "frame": "CY2026Q2", "fy": 2026, "fp": "Q3", "accession_number": "0000320193-26-000020" }, { "schema_version": "1.0.0", "type": "fact", "source_url": "https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json", "scraped_at": "2026-10-17T20:09:52.659Z", "cik": "0000320193", "ticker": "AAPL", "name": "Apple Inc.", "taxonomy": "us-gaap", "concept": "RevenueFromContractWithCustomerExcludingAssessedTax", "label": "Revenue from Contract with Customer, Excluding Assessed Tax", "unit": "USD", "value": 364357000000, "end": "2026-06-27", "start": "2025-09-28", "filed": "2026-07-31", "form": "10-Q", "fy": 2026, "fp": "Q3", "accession_number": "0000320193-26-000020" }, { "schema_version": "1.0.0", "type": "fact", "source_url": "https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json", "scraped_at": "2026-10-17T20:09:52.659Z", "cik": "0000320193", "ticker": "AAPL", "name": "Apple Inc.", "taxonomy": "us-gaap", "concept": "Revenues", "label": "Revenues", "unit": "USD", "value": 62900000000, "end": "2018-09-29", "start": "2018-07-01", "filed": "2018-11-05", "form": "10-K", "frame": "CY2018Q3", "fy": 2018, "fp": "FY", "accession_number": "0000320193-18-000145" } ] ``` Note the last row: Revenues for Apple ends in September 2018, while RevenueFromContractWithCustomerExcludingAssessedTax gives the quarter to June 27, 2026 ($109.4 billion) and the nine months to that date ($364.4 billion). The same run returned NetIncomeLoss (29,789,000,000 for the quarter) and diluted EPS (2.02). Every row carries source_url, the accession number and the filing date, so an agent can cite where a number came from. Run summary: 3 SEC requests, 0 errors. That run costs 1 × $0.0015 + 2 × $0.001 + 8 × $0.0002 = $0.0051, plus Apify's $0.00005 Actor-start fee. ### Name lookup and error rows Input {"companies": ["nvidia", "ZZNOTREAL"], "mode": "filings", "formTypes": ["8-K"], "maxFilingsPerCompany": 3} resolved the name to NVIDIA CORP (CIK 0001045810) and returned a machine-readable error for the unknown query. Real output, abridged to the company row, the newest 8-K and the error: ``` [ { "schema_version": "1.0.0", "type": "company", "source_url": "https://data.sec.gov/submissions/CIK0001045810.json", "scraped_at": "2026-10-17T20:10:05.758Z", "query": "nvidia", "cik": "0001045810", "ticker": "NVDA", "name": "NVIDIA CORP", "entity_type": "operating", "sic": "3674", "sic_description": "Semiconductors & Related Devices", "exchanges": [ "Nasdaq" ], "ein": "943177549", "fiscal_year_end": "0131" }, { "schema_version": "1.0.0", "type": "filing", "source_url": "https://data.sec.gov/submissions/CIK0001045810.json", "scraped_at": "2026-10-17T20:10:05.758Z", "cik": "0001045810", "ticker": "NVDA", "name": "NVIDIA CORP", "form": "8-K", "filing_date": "2026-09-03", "report_date": "2026-09-02", "accession_number": "0001045810-26-000078", "primary_document": "nvda-20260902.htm", "primary_doc_description": "8-K", "document_url": "https://www.sec.gov/Archives/edgar/data/1045810/000104581026000078/nvda-20260902.htm", "filing_url": "https://www.sec.gov/Archives/edgar/data/1045810/000104581026000078/", "is_xbrl": true, "is_inline_xbrl": true, "size": 146499 }, { "schema_version": "1.0.0", "type": "error", "source_url": "https://data.sec.gov/", "scraped_at": "2026-10-17T20:10:05.929Z", "code": "COMPANY_NOT_FOUND", "message": "Could not resolve company query: ZZNOTREAL", "retryable": false, "query": "ZZNOTREAL" } ] ``` Error rows are never charged. Other codes are INPUT_EMPTY, RESOLVE_FAILED, SUBMISSIONS_FETCH_FAILED, FACTS_FETCH_FAILED and EXCERPT_FETCH_FAILED. ### From Python, curl or an AI agent With the official apify-client package (v3; set APIFY_TOKEN): ``` import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("fairquery/edgar-company-facts-filings").call(run_input={ "companies": ["AAPL"], "mode": "both", "formTypes": ["10-K"], "maxFilingsPerCompany": 2, "metrics": ["Revenues", "RevenueFromContractWithCustomerExcludingAssessedTax", "NetIncomeLoss", "EarningsPerShareDiluted"], "maxFactsPerMetric": 2, }) for row in client.dataset(run.default_dataset_id).iterate_items(): if row["type"] == "fact": print(row["concept"], row["start"], row["end"], row["value"], row["form"], row.get("frame", "")) ``` Or one synchronous HTTP call that returns the rows directly: ``` curl -s -X POST "https://api.apify.com/v2/acts/fairquery~edgar-company-facts-filings/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \ -d '{"companies": ["nvidia"], "mode": "filings", "formTypes": ["8-K"], "maxFilingsPerCompany": 1}' ``` In our tests a one-company run took about 2 to 14 seconds end to end, mostly container start-up; the SEC requests themselves took under half a second. For agents, add the Apify MCP server scoped to this Actor (https://mcp.apify.com/?tools=fetch-actor-details,fairquery/edgar-company-facts-filings) to Claude Desktop, Cursor or VS Code, then ask for example: "Get Apple's last two 10-K filings and its latest revenue, net income and diluted EPS from SEC EDGAR." A local stdio MCP server is also in the GitHub repo. ## Input options worth knowing - mode: filings, facts or both. A company row is always emitted when the query resolves. - formTypes (default 10-K, 10-Q, 8-K; [] for all) and since (YYYY-MM-DD) filter filings; maxFilingsPerCompany caps them (default 5). - metrics: exact XBRL concept names, case-insensitive. An empty list uses a curated set of 10 us-gaap concepts that includes both revenue concepts. - includeTextExcerpt: downloads each kept filing's primary document and returns a plain-text excerpt of up to excerptMaxChars characters ($0.002 per excerpt). ## Pricing Option | Price | SEC EDGAR APIs directly (your own code) | Free | Fairquery SEC EDGAR Actor (Apify, pay per event) | $0.0015 per company resolved, $0.001 per filing row, $0.0002 per fact row, $0.002 per text excerpt, plus Apify's $0.00005 Actor-start event. Error rows are free. | Example: 100 companies with 5 filings and 16 fact rows each cost 100 × $0.0015 + 500 × $0.001 + 1,600 × $0.0002 = $0.97. ## How it compares with alternatives Option | Good for | Trade-offs | SEC APIs directly | Free, official, complete for what they cover | You handle CIK lookup, throttling, User-Agent, duplicate values, concept changes and joining filings to facts | Open-source Python libraries (for example edgartools) | Rich parsing of filings and financial statements in Python notebooks and scripts | A Python dependency to run and maintain; not a hosted API or MCP tool | Commercial SEC data APIs | Broader feature sets such as full-text search and parsers for many form types | Account and subscription plans | Fairquery SEC EDGAR Actor | One API or MCP call from agents, n8n or scripts; ticker/CIK/name lookup, filings with document URLs and de-duplicated latest facts as JSON rows, pay per row | Recent filings and companyfacts only: no real-time quotes, no full-text search, no Form 4 parsing, no frames (cross-company) queries, US SEC registrants only | ## FAQ ### Is there a free SEC EDGAR API? Yes. The SEC serves free JSON at data.sec.gov with no API key: submissions (company metadata and filings), companyfacts, companyconcept and frames (XBRL financial data). Declare a User-Agent with your contact details and stay at or below 10 requests per second. ### How do I get a company's CIK from its ticker? Download https://www.sec.gov/files/company_tickers.json, find the ticker, and left-pad cik_str to 10 digits (Apple is 0000320193). The Fairquery EDGAR Actor also accepts tickers, CIKs or company name fragments directly. ### Why is revenue missing or years old for some companies? XBRL concepts change. Apple's Revenues values end in 2018; since 2019 it reports RevenueFromContractWithCustomerExcludingAssessedTax. Request both concepts and check the period end dates. ### Why does companyfacts list the same value several times? Each 10-Q and 10-K repeats earlier periods as comparatives, and companyfacts lists the value once per filing. Sort by period end, keep one value per period from the latest filing, or filter on the frame field. The Fairquery Actor returns each period once. ### How much does the Fairquery SEC EDGAR Actor cost? $0.0015 per company resolved, $0.001 per filing row, $0.0002 per fact row and $0.002 per optional text excerpt, plus Apify's $0.00005 Actor-start event. Error rows are free. Our example run (1 company, 2 filings, 8 facts) costs $0.0051. ### Can AI agents use it? Yes. Add the Apify MCP server with tools=fairquery/edgar-company-facts-filings to Claude Desktop, Cursor or VS Code, call the Actor via the Apify API, or run the stdio MCP server from the GitHub repo. ### Is this investment advice? No. It returns public SEC EDGAR data as reported, for research and tooling, with no warranty of completeness. Check important figures against the linked filing. Not investment advice. Data comes from the U.S. Securities and Exchange Commission's EDGAR system and is returned as reported, with no warranty of completeness. Check important figures against the filing itself (every row links to it). Source and docs: github.com/fairquery/edgar-company-facts-filings (MIT) · Apify Store · all Fairquery tools --- # How do I screen names against OFAC, EU, UK and UN sanctions lists with an API? URL: https://getfairquery.com/posts/how-to-screen-names-ofac-eu-uk-un-sanctions-api/ Published: 2026-10-17 Short answer: send each name to a screening API that downloads the official OFAC, EU, UK and UN lists, fuzzy-matches the name against every primary name and alias, and returns a score plus the listing details so a person can review it. The Fairquery Sanctions & Watchlist Screener does this in one call for $3 per 1,000 names ($0.003 per name, plus $0.007 per run) on Apify, using all five lists from the official sources, never more than 6 hours old, with each list's publication date in every result row. For per-request calls with a fixed monthly plan, the same screening is on RapidAPI (free tier: 20 requests/month). Important: screening results are leads for human review, not legal determinations. Automated fuzzy matching can return false positives and can miss true matches (transliterations, very short names, a list that fails to download). Confirm every potential match against the official publication and take legal advice before acting. Fairquery is not a certified or regulated screening provider and is not affiliated with OFAC, the European Commission, the UK FCDO/OFSI or the UN. ## Which lists are covered The Actor downloads these files from the issuing authority and keeps the parsed lists in a cache in your own Apify account for at most 6 hours (configurable from 0, meaning always download, to 12 hours). Older copies are never used: the list is downloaded again, and if that download fails the list is reported as unavailable instead of being served from a stale copy. Every result row reports each list's publication date and when it was fetched, so you can show exactly which version you screened against (sources can lag or be unavailable). lists id | List | Authority / source | OFAC_SDN | Specially Designated Nationals (SDN) List | US Treasury OFAC, sanctionslistservice.ofac.treas.gov SDN.CSV + ALT.CSV + ADD.CSV | OFAC_CONS | Consolidated (non-SDN) Sanctions List | US Treasury OFAC, CONS_PRIM / CONS_ALT / CONS_ADD.CSV | EU | EU Consolidated Financial Sanctions List | European Commission (DG FISMA), webgate.ec.europa.eu/fsd/fsf CSV | UK | UK Sanctions List (incl. OFSI asset-freeze targets) | UK FCDO / HM Treasury OFSI, sanctionslist.fcdo.gov.uk CSV | UN | UN Security Council Consolidated List | UN Security Council, scsanctions.un.org XML | In our test run on October 17, 2026 the loaded entry counts were OFAC SDN 19,457, OFAC Consolidated 481, EU 6,234, UK 6,408 and UN 1,010. These change whenever the authorities publish updates. ## How the matching works - Normalization: diacritics, case, punctuation, &, legal suffixes (LLC, GmbH, JSC) and honorifics are removed before comparison. - Token-based fuzzy score (0–100): names are compared token by token with Jaro-Winkler, so word order (PUTIN, Vladimir), extra middle names and spelling variants (Usama/Osama, Gaddafi/Qadhafi) still match. The default threshold is minScore: 85; use 90+ for less noise or 80 for more recall. - Aliases: every a.k.a./f.k.a. and non-Latin-script name is screened; matched_on says whether the hit was the primary name or an alias. - Country and date of birth: an optional country adds country_match (and excludeNonMatchingCountry drops mismatches); an optional dateOfBirth drops listed individuals whose birth year differs by more than dobYearTolerance (default 1). ## How to screen names with curl This runs the Actor synchronously and returns the result rows in the same HTTP response. Replace $APIFY_TOKEN with your Apify API token: ``` curl -X POST \ "https://api.apify.com/v2/acts/fairquery~sanctions-watchlist-screener/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "names": ["Rosneft", "Jane Example Smith"], "queries": [{"name": "Banco Nacional de Cuba", "id": "cust-42", "country": "CU"}], "maxMatchesPerName": 3 }' ``` ### Real output from that request We ran exactly this input on October 17, 2026. One row per name; abridged to the first two matches per name and a subset of match fields (aliases, dates of birth and list names omitted): ``` [ { "type": "screening", "query": { "name": "Rosneft", "id": null, "country": null, "date_of_birth": null, "entity_type": "any" }, "status": "potential_match", "best_score": 100, "match_count": 12, "matches": [ { "score": 100, "matched_name": "ROSNEFT", "matched_on": "alias", "list": "OFAC_CONS", "entry_id": "17022", "entity_type": "entity", "primary_name": "OPEN JOINT-STOCK COMPANY ROSNEFT OIL COMPANY", "programs": [ "UKRAINE-EO13662", "RUSSIA-EO14024" ], "countries": [ "Russia" ], "country_match": null, "source_url": "https://sanctionssearch.ofac.treas.gov/Details.aspx?id=17022" }, { "score": 100, "matched_name": "ROSNEFT", "matched_on": "alias", "list": "UK", "entry_id": "RUS3064", "entity_type": "entity", "primary_name": "PJSC Rosneft Oil Company", "programs": [ "The Russia (Sanctions) (EU Exit) Regulations 2019" ], "countries": [ "Russia" ], "country_match": null, "source_url": "https://www.gov.uk/government/publications/the-uk-sanctions-list" } ], "lists_screened": [ "OFAC_SDN", "OFAC_CONS", "EU", "UK", "UN" ], "lists_unavailable": [] }, { "type": "screening", "query": { "name": "Jane Example Smith", "id": null, "country": null, "date_of_birth": null, "entity_type": "any" }, "status": "no_match", "best_score": null, "match_count": 0, "matches": [], "lists_screened": [ "OFAC_SDN", "OFAC_CONS", "EU", "UK", "UN" ], "lists_unavailable": [] }, { "type": "screening", "query": { "name": "Banco Nacional de Cuba", "id": "cust-42", "country": "CU", "date_of_birth": null, "entity_type": "any" }, "status": "potential_match", "best_score": 100, "match_count": 1, "matches": [ { "score": 100, "matched_name": "BANCO NACIONAL DE CUBA", "matched_on": "primary", "list": "OFAC_SDN", "entry_id": "306", "entity_type": "entity", "primary_name": "BANCO NACIONAL DE CUBA", "programs": [ "CUBA" ], "countries": [ "Switzerland", "Spain", "Japan", "Panama" ], "country_match": false, "source_url": "https://sanctionssearch.ofac.treas.gov/Details.aspx?id=306" } ], "lists_screened": [ "OFAC_SDN", "OFAC_CONS", "EU", "UK", "UN" ], "lists_unavailable": [] } ] ``` Things to notice: Rosneft scored 100 on an alias of the listed entity on several lists (match_count 12 in total, top 3 returned). Banco Nacional de Cuba matched the OFAC SDN primary name, but country_match is false because the listed addresses are outside Cuba, which is exactly the kind of detail a reviewer needs. Jane Example Smith came back no_match. The run's RUN_SUMMARY record shows what was loaded: ``` { "names_screened": 3, "names_with_matches": 2, "potential_matches": 13, "lists": { "OFAC_CONS": { "loaded": true, "entries": 481, "ms": 2222 }, "UN": { "loaded": true, "entries": 1010, "ms": 3730 }, "UK": { "loaded": true, "entries": 6408, "ms": 5768 }, "OFAC_SDN": { "loaded": true, "entries": 19457, "ms": 6604 }, "EU": { "loaded": true, "entries": 6234, "ms": 8072 } }, "charged_events": { "name-screened": 3 } } ``` The whole run took about 11 seconds, almost all of it downloading and indexing the five lists. The Actor README documents roughly 10–40 seconds for list loading when the lists are downloaded (depending on the government servers) and about 1–5 seconds from the cache, after which screening runs at hundreds of names per second, so put all your names into one run rather than one run per name. ## Python and JavaScript ``` import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("fairquery/sanctions-watchlist-screener").call( run_input={"names": ["Acme Trading LLC", "Rosneft"], "minScore": 90} ) for row in client.dataset(run["defaultDatasetId"]).iterate_items(): if row["type"] == "screening" and row["status"] == "potential_match": print(row["query"]["name"], row["best_score"], [(m["list"], m["primary_name"]) for m in row["matches"]]) ``` ``` import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('fairquery/sanctions-watchlist-screener').call({ queries: [{ name: 'Ali Khamenei', id: 'cust-42', country: 'IR', dateOfBirth: '1939' }], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const r of items.filter((i) => i.status === 'potential_match')) { console.log(r.query.id, r.best_score, r.matches.map((m) => `${m.list}: ${m.primary_name}`)); } ``` ## One name per request: the RapidAPI endpoint If you'd rather call a plain REST endpoint with a RapidAPI key, the Sanctions Screening – OFAC EU UK UN listing exposes GET /screen?name=... (also accepts POST with a JSON body) with optional country, dateOfBirth, entityType, lists, minScore, maxMatches and excludeNonMatchingCountry. RapidAPI's playground on the listing generates the request code for your language. Plans as published on the listing: Plan | Price / month | Requests included | Overage | Rate limit | BASIC | $0 | 20 (hard limit) | none | 1 req/s | PRO | $19 | 5,000 | $0.006 | 5 req/s | ULTRA | $49 | 25,000 | $0.004 | 10 req/s | MEGA | $99 | 100,000 | $0.002 | 20 req/s | Speed: the first request after a period of inactivity can take up to about 60 s while the five official lists are downloaded and indexed (about 40 s in our tests); later requests are typically fast, about 1 s or less. Use a client timeout of at least 90 s and retry once on a timeout. ## Using it from an AI agent (MCP) - Apify MCP: connect your MCP client to mcp.apify.com and call Actor fairquery/sanctions-watchlist-screener with {"names": ["Acme Trading LLC"]}. - Agent Trust MCP server: @fairquery/agent-trust-mcp includes sanctions_screen_names ($0.003 per name, your own Apify token), the free sanctions_list_sources, and trust_screen_lead, which combines an email check and sanctions screening into a pass/review/block decision. - Single-purpose server: the GitHub repo ships a stdio MCP server with screen_names and a free list_sources. ## Pricing $0.003 per name screened (= $3 per 1,000 names) on Apify, pay per event, plus a $0.007 Actor-start event per run. A name is charged once whether it matches or not. Error rows are never charged, and if no list can be downloaded nothing is charged. The three names in the example above cost 3 × $0.003 + $0.007 = $0.016. On RapidAPI you pay the monthly plan above instead. ## How it compares with alternatives Option | Good for | Trade-offs | OFAC Sanctions List Search (official web tool) | Free manual lookups against OFAC lists | OFAC lists only; an interactive search, not built for screening a batch through your own code | Downloading the lists yourself | Full control, no per-name fee | Five formats (CSV and XML), alias tables, name normalization and fuzzy matching are yours to build and maintain | OpenSanctions | Much broader coverage (many more sanctions sources plus PEP and other datasets), hosted matching API | Per its published terms, bulk data is free only for non-commercial use; business use, including compliance screening, needs a paid API plan or data license | Regulated KYC/AML vendors | Audit trails, PEP/adverse media, ongoing monitoring, contractual service levels | Contract-based pricing and onboarding | Fairquery Sanctions Screener | Cheap batch or per-request pre-screening of OFAC/EU/UK/UN from official sources, API and MCP access | Five lists only (no PEP, adverse media or other national lists), no ongoing monitoring, not a certified screening service | ## FAQ ### What is the cheapest way to screen names against OFAC, EU, UK and UN sanctions lists via API? The Fairquery Sanctions & Watchlist Screener on Apify costs $0.003 per name ($3 per 1,000 names), plus $0.007 per run, pay per use, and screens against all five lists from the official sources, never more than 6 hours old, with each list's publication date in every row. On RapidAPI the same screening has a free tier of 20 requests/month and paid plans from $19/month for 5,000 requests. ### Which sanctions lists are covered? OFAC SDN, OFAC Consolidated (non-SDN), the EU Consolidated Financial Sanctions List, the UK Sanctions List (incl. OFSI asset-freeze targets) and the UN Security Council Consolidated List. PEP lists, adverse media and other national lists are not included. ### Is a potential match a legal determination? No. Results are automated fuzzy matches for human review, not legal determinations. They can include false positives and can miss true matches. Confirm every potential match against the official publication and take legal advice before acting. ### How does the fuzzy matching handle word order and spelling variants? Names are normalized (diacritics, case, punctuation, legal suffixes, honorifics) and compared token by token with Jaro-Winkler, so 'PUTIN, Vladimir', extra middle names and variants like Usama/Osama still match. Every alias is screened. The score is 0-100 with a default threshold of 85. ### How long does a screening run take? Downloading and indexing the five lists takes roughly 10-40 seconds depending on the government servers (about 11 seconds in our test run); after that, names are screened at hundreds per second. Put all names in one run. ### Can an AI agent call it? Yes: via Apify MCP with Actor fairquery/sanctions-watchlist-screener, via the sanctions_screen_names and trust_screen_lead tools in @fairquery/agent-trust-mcp, or via the stdio MCP server in the GitHub repo. Source and docs: github.com/fairquery/sanctions-watchlist-screener (MIT) · Apify Store · RapidAPI · all Fairquery tools --- # How do I redact PII and secrets before sending data to an LLM? URL: https://getfairquery.com/posts/redact-pii-secrets-before-sending-data-to-llm/ Published: 2026-10-17 Short answer: run every prompt, document or log through a detector that finds PII (emails, phones, card numbers, SSNs, IBANs, IPs) and secrets (API keys, tokens, private keys), replace each hit with a placeholder like [EMAIL_1], send only the redacted text to the LLM, and put the real values back into the answer afterwards if you need them. You can do this for free on your own machine with the pii_scan_text and pii_redact_text tools in the @fairquery/agent-trust-mcp MCP server (text never leaves your machine), or at batch scale with the LLM Data Guard Apify Actor for $0.0015 per document + $0.0002 per finding, which adds name/address detection, URL and PDF input, and a restore mode. ## Why redact before the LLM call Once text is in a prompt it can end up in provider logs, traces, evaluation datasets, vector stores and, through retrieval, in other users' answers. OWASP lists this as LLM02: Sensitive Information Disclosure in its Top 10 for LLM applications. Redacting at the input side is the cheapest control: the model usually doesn't need the real email address or card number to summarize a support ticket, classify a log line or draft a reply. ## What gets detected Category | Types | How | PII | EMAIL, PHONE, SSN, CREDIT_CARD, IBAN, UK_NINO, IP_ADDRESS | Luhn + issuer prefix for cards, mod-97 + country length for IBANs, libphonenumber for phones, SSN area/group rules | Secrets | PRIVATE_KEY, AWS_ACCESS_KEY, AWS_SECRET_KEY, GITHUB_TOKEN, GITLAB_TOKEN, SLACK_TOKEN, SLACK_WEBHOOK, STRIPE_KEY, GOOGLE_API_KEY, OPENAI_KEY, ANTHROPIC_KEY, HUGGINGFACE_TOKEN, NPM_TOKEN, SENDGRID_KEY, TWILIO_KEY, AZURE_STORAGE_KEY, JWT, URL_CREDENTIALS, GENERIC_SECRET | Gitleaks-style rules; GENERIC_SECRET (api_key=..., password=...) uses a Shannon-entropy threshold | Opt-in | HIGH_ENTROPY_STRING (local + Actor); PERSON, LOCATION, STREET_ADDRESS (Actor only) | Entropy scan; open-source NER (compromise, English) and street-address patterns | Overlaps are resolved (secrets > cards/IBAN > SSN > email > IP > phone > address > names), so a card number is never also reported as a phone. ## Option 1 (free): redact locally with the Agent Trust MCP server Add the server to Claude Desktop (claude_desktop_config.json), Cursor (~/.cursor/mcp.json) or any MCP client. No token is needed for the free tools: ``` { "mcpServers": { "fairquery-trust": { "command": "npx", "args": ["-y", "@fairquery/agent-trust-mcp"] } } } ``` Then ask the agent to "redact this ticket before summarizing it", or call the tools from your own code with the MCP SDK. This is the script we ran (Node.js 20+, npm i @modelcontextprotocol/sdk): ``` import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; const transport = new StdioClientTransport({ command: 'npx', args: ['-y', '@fairquery/agent-trust-mcp'] }); const client = new Client({ name: 'redactor', version: '1.0.0' }); await client.connect(transport); const ticket = 'Ticket #4411 from Jane Doe , phone +1 415-555-0132. ' + 'Card 4111 1111 1111 1111 was declined. Debug log: AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE'; const res = await client.callTool({ name: 'pii_redact_text', arguments: { texts: [ticket], returnPlaceholderMap: true }, }); console.log(res.content[0].text); // JSON: meta.placeholder_map + items[].output_text await client.close(); ``` (The card number is the standard Visa test number and the AWS key is the example key from AWS's documentation.) ### Real output: pii_redact_text (placeholder style) ``` { "meta": { "engine": "local", "redaction_style": "placeholder", "placeholder_map": { "[EMAIL_1]": "jane.doe@example.com", "[PHONE_1]": "+1 415-555-0132", "[CREDIT_CARD_1]": "4111 1111 1111 1111", "[AWS_ACCESS_KEY_1]": "AKIAIOSFODNN7EXAMPLE" } }, "items": [ { "type": "redacted_document", "finding_count": 4, "risk_level": "critical", "output_text": "Ticket #4411 from Jane Doe <[EMAIL_1]>, phone [PHONE_1]. Card [CREDIT_CARD_1] was declined. Debug log: AWS_ACCESS_KEY_ID=[AWS_ACCESS_KEY_1]" } ] } ``` Send output_text to the model; keep placeholder_map on your side (it contains the originals) to swap the real values back into the answer. With redactionStyle: "mask" the same input became: ``` Ticket #4411 from Jane Doe , phone +* ***-***-0132. Card **** **** **** 1111 was declined. Debug log: AWS_ACCESS_KEY_ID=AKIA**************** ``` ### Real output: pii_scan_text (detect only) Use the scan tool when you only want to know whether text is safe to send, for example to block a prompt or route it for review. Findings carry offsets and a masked preview, never the raw value: ``` { "count": 4, "total": 4, "offset": 0, "hasMore": false, "nextOffset": null, "meta": { "documents": [ { "doc_index": 0, "chars": 162, "finding_count": 4, "counts_by_type": { "EMAIL": 1, "PHONE": 1, "CREDIT_CARD": 1, "AWS_ACCESS_KEY": 1 }, "risk_level": "critical" } ], "engine": "local" }, "items": [ { "type": "EMAIL", "doc_index": 0, "category": "pii", "rule": "email", "start": 28, "end": 48, "confidence": "high", "masked": "j***@e***.com" }, { "type": "PHONE", "doc_index": 0, "category": "pii", "rule": "libphonenumber", "start": 57, "end": 72, "confidence": "high", "masked": "+* ***-***-0132" }, { "type": "CREDIT_CARD", "doc_index": 0, "category": "pii", "rule": "card-luhn", "start": 79, "end": 98, "confidence": "high", "masked": "**** **** **** 1111" }, { "type": "AWS_ACCESS_KEY", "doc_index": 0, "category": "secret", "rule": "aws-access-key-id", "start": 142, "end": 162, "confidence": "high", "masked": "AKIA****************" } ] } ``` In the local tools, risk_level is critical if any secret is found, high for card/IBAN/SSN/NI numbers, medium for other PII and none otherwise. In our test the scan returned in 14 ms and the redaction in 5 ms (server already running). Limits per call: up to 50 texts, 200,000 characters each, 500,000 in total. ## Option 2 (API, batch): LLM Data Guard Actor Use the Actor when you need to process many documents, fetch public URLs or PDFs (text layer, no OCR, up to 10 MB each), detect person names and street addresses, use consistent placeholders across a whole dataset (for RAG or fine-tuning), or restore values server-side. ``` curl -X POST \ "https://api.apify.com/v2/acts/fairquery~llm-data-guard-pii-secrets/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "texts": ["Ticket #4411 from Jane Doe , phone +1 415-555-0132. Card 4111 1111 1111 1111 was declined. Debug log: AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE"], "mode": "redact", "detectNames": true }' ``` ### Real output from that request Abridged (first two of five findings shown). With detectNames on, the Actor also caught the name Jane Doe as a low-confidence PERSON: ``` { "type": "document", "id": "text-1", "chars": 162, "mode": "redact", "finding_count": 5, "counts_by_type": { "PERSON": 1, "EMAIL": 1, "PHONE": 1, "CREDIT_CARD": 1, "AWS_ACCESS_KEY": 1 }, "risk_level": "high", "findings": [ { "type": "PERSON", "category": "pii", "rule": "ner-compromise", "confidence": "low", "start": 18, "end": 26, "length": 8, "preview": "**** ***", "replacement": "[PERSON_1]" }, { "type": "EMAIL", "category": "pii", "rule": "email", "confidence": "high", "start": 28, "end": 48, "length": 20, "preview": "j***@e***.com", "replacement": "[EMAIL_1]" }, "… 3 more (PHONE, CREDIT_CARD, AWS_ACCESS_KEY)" ], "output_text": "Ticket #4411 from [PERSON_1] <[EMAIL_1]>, phone [PHONE_1]. Card [CREDIT_CARD_1] was declined. Debug log: AWS_ACCESS_KEY_ID=[AWS_ACCESS_KEY_1]" } ``` That run took 2.5 seconds end to end on Apify (mostly container start) and would cost $0.0015 + 5 × $0.0002 = $0.0025. The README documents roughly 1–5 MB of text per second for the regex detectors; NER is slower. In the Actor, risk_level is high for any secret or card/SSN/IBAN/NI number, medium for other confident PII and low for low-confidence hits only. ### Restore after the model answers ``` { "mode": "restore", "texts": ["Dear [PERSON_1], we emailed [EMAIL_1] about the declined card."], "placeholderMap": { "[PERSON_1]": "Jane Doe", "[EMAIL_1]": "jane.doe@example.com" } } ``` Get the map by setting returnPlaceholderMap: true in redact mode. Treat it like the original data, because it is. ### Data handling The Actor is designed not to log document content. It writes only result rows and a counts-only RUN_SUMMARY, deletes the run's INPUT record at the end by default (purgeInputAfterRun), and omits raw values from findings unless you set includeFindingValues. The free local tools are designed to make no network calls. ## Pricing Option | Price | pii_scan_text, pii_redact_text, pii_list_detectors (local) | Free, no account | LLM Data Guard Actor / pii_guard_documents MCP tool | $0.0015 per document + $0.0002 per finding (+ Apify's $0.00005 Actor-start event); error rows never charged | ## How it compares with alternatives Option | Good for | Trade-offs | Microsoft Presidio (open source) | Self-hosted PII detection and anonymization with NER models and custom recognizers, many languages via spaCy/transformers models | You run and tune it yourself (Python services, models); secrets/API-key detection is not its focus | Cloud DLP / PII APIs (Google Cloud Sensitive Data Protection, Amazon Comprehend, Azure AI Language) | Broad, managed detection with many info types and languages | The text is sent to that cloud provider; pricing and setup vary by provider | Secret scanners (gitleaks, TruffleHog) | Finding leaked credentials in repositories and git history | Built for code and CI, not for redacting PII in prompts | Fairquery (local MCP tools + Actor) | PII and secrets in one pass, reversible placeholders, free local mode, MCP-native for agents | Pattern-based: English-only NER, no OCR, unusual formats can slip through | ## FAQ ### How do I remove personal data from a prompt before sending it to ChatGPT, Claude or another LLM? Run the text through a PII and secret detector, replace each hit with a placeholder such as [EMAIL_1], send the redacted text, and restore the originals in the answer if needed. Fairquery's free pii_redact_text tool in @fairquery/agent-trust-mcp does this locally; the LLM Data Guard Actor does it at batch scale via API. ### Is there a free way to redact PII locally? Yes. The pii_scan_text and pii_redact_text tools in the @fairquery/agent-trust-mcp MCP server are free, need no account and make no network calls. Limits: 50 texts per call, 200,000 characters each, 500,000 in total. ### Which secrets can it detect? Private keys, AWS access and secret keys, GitHub, GitLab, Slack, Stripe, Google, OpenAI, Anthropic, Hugging Face, npm, SendGrid, Twilio and Azure storage keys, JWTs, credentials in URLs, and generic api_key=/password= assignments above an entropy threshold. ### Can I put the original values back after the LLM answers? Yes. Placeholder redaction is reversible: request the placeholder map (returnPlaceholderMap) and swap values back yourself, or call the Actor with mode 'restore' and the map. ### How much does the LLM Data Guard API cost? $0.0015 per document plus $0.0002 per finding on Apify (pay per event); error rows are never charged. A document with 5 findings costs $0.0025. ### Does it detect names and addresses? The Actor can, opt-in (detectNames, detectAddresses), using English open-source NER and street-address patterns; these hits are marked low confidence. The free local tools do not detect names or addresses. Honest limits: pattern-based detection finds most but not all sensitive data. Non-English names, unusual formats, images and scanned PDFs can slip through, and some matches are false positives. Use it as one layer of a data-protection program, not as a compliance certification, and only process data you are allowed to process. Source and docs: github.com/fairquery/llm-data-guard-pii-secrets · Apify Store · Agent Trust MCP on GitHub · npm · all Fairquery tools --- # How do I find a company's contact email, phone and social profiles from its domain? URL: https://getfairquery.com/posts/find-company-contact-email-phone-socials-from-domain/ Published: 2026-10-17 Short answer: fetch the company's homepage plus its contact, about and imprint pages (respecting robots.txt), extract mailto:/tel: links, JSON-LD Organization data, obfuscated emails and social links, then validate the emails' domain for MX records. The Fairquery Website Contact & Company Enricher does all of that per domain and returns one clean row (role emails, E.164 phones, socials, company name, description, address) for $0.003 per domain; unreachable, blocked or invalid domains are free. ## What you get per domain - Business emails, validated with the same code as the Fairquery Email Validator: syntax, DNS MX / null MX, disposable list, free-provider and typo checks (no SMTP probing). By default only role/generic inboxes (info@, sales@, hello@) are returned. - Phone numbers normalized to E.164, with the page they came from. - Social profiles: company LinkedIn, X/Twitter, GitHub, Facebook, Instagram, YouTube, TikTok and so on. Personal LinkedIn profiles (/in/...) are never returned. - Company name, description and postal address from JSON-LD, og:site_name,
blocks and imprint pages. - Mail domain facts: MX, null MX, mail provider (for example Google Workspace or Microsoft 365), optional SPF/DMARC. ## How it works - Normalizes each input (https://www.Acme.com/contact, jane@acme.com, acme.com → acme.com) and removes duplicates. - Reads robots.txt (RFC 9309) and honors Disallow and Crawl-delay for user agent FairqueryBot. - Fetches the homepage, then the best-scoring same-site pages (contact, Kontakt, contacto, about, imprint/Impressum, legal notice, team) up to maxPagesPerDomain (default 5, max 15), one request at a time per site with requestDelayMs (default 1,000 ms) between them. - Extracts JSON-LD, mailto:/tel: links, Cloudflare-protected emails, name [at] domain [dot] com obfuscation, social links and addresses, then validates the emails. It is plain HTTP, with no headless browser and no proxy, so sites that render contact details only with JavaScript or block datacenter IPs return fewer fields. ## How to run it with curl ``` curl -X POST \ "https://api.apify.com/v2/acts/fairquery~website-contact-company-enricher/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"domains": ["apify.com"], "maxPagesPerDomain": 4}' ``` ### Real output from that request We ran exactly this on October 17, 2026 (row shown in full, minus the timestamp and the duplicate addresses array): ``` { "type": "domain", "input": "apify.com", "domain": "apify.com", "final_url": "https://apify.com/", "company_name": "Apify Technologies s.r.o.", "description": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents.", "emails": [ { "email": "hello@apify.com", "source_url": "https://apify.com/contact", "is_role_account": true, "on_company_domain": true, "is_free_provider": false, "is_disposable": false, "verdict": "valid", "reasons": [ "ROLE_ACCOUNT" ], "has_mx": true, "mail_provider": "Google Workspace / Gmail" } ], "primary_email": "hello@apify.com", "phones": [], "social_links": { "github": [ "https://github.com/apify" ], "linkedin": [ "https://linkedin.com/company/apify" ], "twitter": [ "https://x.com/apify" ], "tiktok": [ "https://tiktok.com/@apifytech", "https://tiktok.com/@apifyoffice" ] }, "address": "Na Příkopě 959/27, 11000, Prague, CZ", "mail_domain": { "has_mx": true, "null_mx": false, "mail_provider": "Google Workspace / Gmail", "has_spf": null, "has_dmarc": null, "dmarc_policy": null }, "pages_crawled": [ "https://apify.com/", "https://apify.com/contact", "https://apify.com/contact-sales" ], "pages_blocked_by_robots": 0, "robots_txt": "ok", "email_scope": "role" } ``` The run crawled 3 pages, took 5.1 seconds and would cost $0.003. The README documents about 3–8 seconds per domain with the default politeness settings and 5 domains in parallel, so 100 domains take roughly 1–3 minutes. No phone number was published on the crawled pages here, so phones is empty rather than guessed. ## Python: enrich a CSV of company websites ``` import csv, os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) domains = [row["website"] for row in csv.DictReader(open("companies.csv"))] run = client.actor("fairquery/website-contact-company-enricher").call( run_input={"domains": domains, "emailScope": "role", "defaultPhoneCountry": "US"} ) for r in client.dataset(run["defaultDatasetId"]).iterate_items(): if r["type"] == "domain": print(r["domain"], r["primary_email"], [p["e164"] for p in r["phones"]], r["social_links"].get("linkedin")) else: print("skipped:", r.get("code"), r.get("message")) ``` ## Input options Field | Default | Meaning | domains | (required) | Domains, URLs or emails | maxPagesPerDomain | 5 | 1–15, homepage included | emailScope | role | business adds named addresses on the company's own domain; all returns everything found | validateEmails | true | MX / disposable / typo checks | checkSpfDmarc | false | SPF/DMARC for the company mail domain | defaultPhoneCountry | – | e.g. US; also parses national-format numbers in page text | domainConcurrency / requestDelayMs / requestTimeoutSecs | 5 / 1000 / 15 | Politeness and timeouts | maxDomains | 1000 | Safety cap per run | Error rows (never charged): INPUT_EMPTY, INVALID_DOMAIN, MAX_DOMAINS_EXCEEDED, SITE_UNREACHABLE, BLOCKED_BY_ROBOTS, ENRICH_FAILED. ## From an AI agent Call Actor fairquery/website-contact-company-enricher through Apify MCP, or run the stdio MCP server from the GitHub repo. It has enrich_domains (paid, your Apify token) and a free local normalize_domains tool for cleaning a messy list first. Real output of normalize_domains for ["https://www.Apify.com/contact", "jane@stripe.com", "basecamp.com", "apify.com", "not a domain"]: ``` { "domains": ["apify.com", "stripe.com", "basecamp.com"], "invalid": ["not a domain"], "duplicates_removed": 1 } ``` ## Pricing $0.003 per domain enriched (= $3 per 1,000 domains) on Apify, pay per event, plus Apify's $0.00005 Actor-start event. Only domains that return a result row are charged; unreachable, robots-blocked and invalid domains are free. ## How it compares with alternatives Option | Good for | Trade-offs | Doing it by hand | A handful of companies | Slow; easy to miss imprint pages and obfuscated emails | Writing your own scraper | Full control | robots.txt handling, page discovery, email de-obfuscation, phone parsing and email validation to build and maintain | B2B contact databases (for example Hunter, Apollo, ZoomInfo) | Finding named people, job titles and personal work emails from large pre-built databases | Subscription pricing; data comes from their database rather than a live read of the company's site | Fairquery Contact Enricher | Live, per-domain company contact data with source URLs and validated emails, pay per domain | Only what the company publishes on its own site; role inboxes by default, no person search; JavaScript-only sites yield less | ## FAQ ### How can I get a company's contact email from its website domain? Crawl the homepage and its contact, about and imprint pages, extract mailto links, JSON-LD and obfuscated emails, and validate the domain's MX records. The Fairquery Website Contact & Company Enricher does this per domain for $0.003 and returns validated role emails with the page they were found on. ### Does it find personal email addresses of employees? No. By default it returns only role/generic inboxes such as info@, sales@ or hello@ (emailScope 'role'). 'business' adds named addresses published on the company's own domain. It never returns personal LinkedIn profiles and does not search for people. ### Does it respect robots.txt? Yes. It reads robots.txt (RFC 9309), honors Disallow and Crawl-delay for user agent FairqueryBot, makes one request at a time per site with a 1-second default delay, and reports pages_blocked_by_robots. ### How fast is it? About 3-8 seconds per domain with the default settings and 5 domains in parallel, so 100 domains take roughly 1-3 minutes. Our single-domain test run took 5.1 seconds. ### What does it cost? $0.003 per domain enriched on Apify. Unreachable, robots-blocked and invalid domains are free. ### Why are some fields empty? It only returns what the company publishes on the crawled pages. Sites that render contacts with JavaScript only, or block datacenter IPs, return fewer fields, because it uses plain HTTP without a browser or proxy. Use responsibly: business contact data can still be personal data under GDPR/UK GDPR. Have a lawful basis, honor opt-outs, and follow anti-spam law (CAN-SPAM, ePrivacy/PECR) when you contact anyone. DNS checks cannot prove a mailbox exists. Source and docs: github.com/fairquery/website-contact-company-enricher (MIT) · Apify Store · all Fairquery tools --- # How do I check npm and PyPI dependencies for vulnerabilities in CI with OSV? URL: https://getfairquery.com/posts/check-npm-pypi-dependencies-vulnerabilities-ci-osv/ Published: 2026-10-17 Short answer: pin exact versions (use your lockfile or pip freeze), send them to the free OSV.dev API, and fail the build when a vulnerability at or above your severity bar is found. For a free yes/no gate you can call OSV's querybatch endpoint directly or use the free osv_query_packages tool in @fairquery/agent-trust-mcp. For CI-ready rows with CVE IDs, CVSS scores, severity filtering, alias de-duplication and a recommended upgrade version, run the Fairquery OSV Dependency Vulnerability Check Actor: $0.0005 per package checked + $0.0015 per vulnerability found. ## Why OSV OSV.dev is a free, open vulnerability database and API, an open-source project launched by Google. It aggregates advisories from the GitHub Advisory Database, the PyPA advisory database, the Go vulnerability database, RustSec and others in one schema, and covers npm, PyPI, Go, Maven, crates.io, RubyGems, NuGet, Packagist, Pub, Hex and more. It needs no API key. The catch is that a raw OSV response is a list of advisory IDs per package; severity, CVSS, CVE aliases and fixed versions are in separate detail records you must fetch and merge yourself. ## Option 1 (free): query OSV directly ``` curl -s -X POST https://api.osv.dev/v1/querybatch -d '{ "queries": [ {"package": {"ecosystem": "npm", "name": "lodash"}, "version": "4.17.20"}, {"package": {"ecosystem": "npm", "name": "express"}, "version": "4.21.2"} ]}' ``` Real response (October 17, 2026, about 0.2 s): five advisories for lodash 4.17.20, none for express 4.21.2. ``` {"results":[{"vulns":[{"id":"GHSA-29mw-wpgm-hmr9","modified":"2025-09-29T21:12:31.102523Z"},{"id":"GHSA-35jh-r3h4-6jhm","modified":"2026-10-17T02:55:38.085813Z"},{"id":"GHSA-f23m-r3pf-42rh","modified":"2026-10-17T02:56:13.084362Z"},{"id":"GHSA-r5fr-rjxr-66jc","modified":"2026-10-17T02:56:15.633068Z"},{"id":"GHSA-xxjr-mmjv-4gpg","modified":"2026-10-17T02:55:54.229635Z"}]},{}]} ``` A minimal free CI gate for an npm lockfile (v2/v3) with jq: ``` jq '{queries: [.packages | to_entries[] | select(.key | startswith("node_modules/")) | {package: {ecosystem: "npm", name: (.key | sub("^.*node_modules/"; ""))}, version: .value.version}]}' \ package-lock.json > q.json curl -s -X POST https://api.osv.dev/v1/querybatch -d @q.json > osv.json jq -e '[.results[].vulns // [] | length] | add == 0' osv.json # exit code 1 if anything is vulnerable ``` This gate fails on any advisory, including low-severity ones, because querybatch doesn't return severity. For the same lookup in Python with pagination, details and alias de-duplication, see the OSV API in Python guide. In an agent, the free osv_query_packages tool does the same lookup for up to 200 exact versions per call. Real output for lodash 4.17.20, django 3.2.0 and express 4.21.2 (abridged): ``` { "meta": { "vulnerable_count": 2, "source": "https://api.osv.dev/v1/querybatch" }, "items": [ { "ecosystem": "npm", "name": "lodash", "version": "4.17.20", "vulnerable": true, "vuln_count": 5, "vuln_ids": [ "GHSA-29mw-wpgm-hmr9", "GHSA-35jh-r3h4-6jhm", "GHSA-f23m-r3pf-42rh", "GHSA-r5fr-rjxr-66jc", "GHSA-xxjr-mmjv-4gpg" ] }, { "ecosystem": "PyPI", "name": "django", "version": "3.2.0", "vulnerable": true, "vuln_count": 66, "vuln_ids": [ "GHSA-2gwj-7jmv-h26r", "GHSA-2hrw-hx67-34x6", "GHSA-3h9f-r86x-qvjx", "… 63 more" ] }, { "ecosystem": "npm", "name": "express", "version": "4.21.2", "vulnerable": false, "vuln_count": 0, "vuln_ids": [] } ] } ``` Its sibling osv_get_vulnerability (also free) returns the full advisory, for example GHSA-35jh-r3h4-6jhm = CVE-2021-23337, "Command Injection in lodash", severity HIGH. ## Option 2 (API): severity-filtered rows from the OSV Actor The Actor parses package.json, package-lock.json (v1–v3), requirements.txt or go.mod, queries OSV, fetches every advisory, computes the CVSS v3 base score, maps GHSA/PYSEC/GO IDs to CVEs, collapses duplicate advisories (dedupeAliases), filters by minSeverity, and computes the smallest version that fixes every emitted vulnerability. We ran it on October 17, 2026 with this input (two explicit packages plus a small package.json): ``` { "packages": [ {"ecosystem": "npm", "name": "lodash", "version": "4.17.20"}, {"ecosystem": "PyPI", "name": "django", "version": "3.2.0"} ], "manifestText": "{\"dependencies\":{\"lodash\":\"4.17.20\",\"express\":\"4.21.2\"}}", "minSeverity": "HIGH" } ``` ### Real output Abridged: three package rows (lodash appeared in both inputs and was checked once) and the first of 23 vulnerability rows: ``` { "package_rows": [ { "type": "package", "ecosystem": "npm", "name": "lodash", "version": "4.17.20", "purl": "pkg:npm/lodash@4.17.20", "origin": "input", "vulnerable": true, "vuln_count": 2, "vuln_count_total": 5, "vuln_ids": [ "GHSA-r5fr-rjxr-66jc", "GHSA-35jh-r3h4-6jhm" ], "max_severity": "HIGH", "max_cvss_score": 8.1, "fixed_versions": [ "4.18.0", "4.17.21" ], "recommended_version": "4.18.0" }, { "type": "package", "ecosystem": "PyPI", "name": "django", "version": "3.2.0", "purl": "pkg:pypi/django@3.2.0", "origin": "input", "vulnerable": true, "vuln_count": 21, "vuln_count_total": 66, "vuln_ids": [ "GHSA-2gwj-7jmv-h26r", "GHSA-p64x-8rxx-wf6q", "GHSA-r3xc-prgr-mg9p", "… 18 more" ], "max_severity": "CRITICAL", "max_cvss_score": 9.8, "fixed_versions": [ "2.2.28", "3.2.13", "4.0.4", "… 47 more" ], "recommended_version": "5.2.17" }, { "type": "package", "ecosystem": "npm", "name": "express", "version": "4.21.2", "purl": "pkg:npm/express@4.21.2", "origin": "manifest", "vulnerable": false, "vuln_count": 0, "vuln_count_total": 0, "vuln_ids": [], "max_severity": "NONE", "max_cvss_score": null, "fixed_versions": [], "recommended_version": null } ], "first_vulnerability_row": { "type": "vulnerability", "id": "GHSA-r5fr-rjxr-66jc", "package_name": "lodash", "package_version": "4.17.20", "cve_ids": [ "CVE-2026-4800" ], "summary": "lodash vulnerable to Code Injection via `_.template` imports key names", "severity": "HIGH", "cvss_score": 8.1, "cvss_version": "3.1", "fixed_versions": [ "4.18.0" ], "osv_url": "https://osv.dev/vulnerability/GHSA-r5fr-rjxr-66jc" } } ``` Because a package.json holds ranges, the Actor also emitted a free warning row; feed it the lockfile instead for exact versions: ``` { "type": "error", "code": "RANGE_APPROXIMATED", "message": "package.json ranges were resolved to their lower bound (e.g. ^4.17.20 -> 4.17.20). Installed versions may differ; prefer package-lock.json.", "context": { "manifest_type": "package.json", "severity": "warning" } } ``` And the counts from the run's RUN_SUMMARY. OSV knows 66 advisories for django 3.2.0; after the HIGH filter and alias de-duplication 21 remain: ``` { "packages_checked": 3, "packages_vulnerable": 2, "vulnerabilities_emitted": 23, "vulnerabilities_filtered_out": 20, "vulnerabilities_deduplicated": 28, "severity_counts": { "NONE": 0, "UNKNOWN": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 17, "CRITICAL": 6 }, "osv_requests": 72 } ``` The run took 2.8 seconds on Apify and would cost 3 × $0.0005 + 23 × $0.0015 = $0.036. The README documents about 5–10 s for a 2-package run; time grows with the number of advisories to fetch (8 detail requests run in parallel). ## GitHub Actions: fail the PR on HIGH/CRITICAL Add your Apify token as the repository secret APIFY_TOKEN, then: ``` name: dependency-vulnerabilities on: [pull_request] jobs: osv: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: OSV check (HIGH and CRITICAL only) env: APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }} run: | jq -n --rawfile m package-lock.json \ '{manifestText: $m, minSeverity: "HIGH"}' > input.json curl -sf -X POST \ "https://api.apify.com/v2/acts/fairquery~osv-dependency-vulnerability-check/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \ -d @input.json > osv.json jq -r '.[] | select(.type=="package" and .vulnerable) | "\(.name)@\(.version) -> upgrade to \(.recommended_version // "n/a") (\(.max_severity))"' osv.json n=$(jq '[.[] | select(.type=="vulnerability")] | length' osv.json) echo "$n HIGH/CRITICAL vulnerabilities" test "$n" -eq 0 ``` Run against the dataset from our test run above, the reporting and gating steps print: ``` lodash@4.17.20 -> upgrade to 4.18.0 (HIGH) django@3.2.0 -> upgrade to 5.2.17 (CRITICAL) 23 HIGH/CRITICAL vulnerabilities ``` and exit with status 1, failing the job. Notes: when you pass only a manifest, only the packages in that manifest are checked and billed; an input with neither packages nor manifestText fails immediately with an INPUT_EMPTY error row and no package or vulnerability events are charged. For Python, use pip freeze > requirements.txt first (unpinned lines are skipped with a VERSION_NOT_PINNED warning). The default safety cap is maxPackages: 1000 unique packages per run; raise it for very large lockfiles. You can also set a maximum charge per run in the Apify run options; the Actor stops cleanly at the limit. Prefer n8n? Fairquery's free n8n templates include an OSV CI gate workflow that calls this Actor. ## Pricing Option | Price | OSV.dev API directly, osv_query_packages, osv_get_vulnerability | Free | OSV Actor / osv_audit_dependencies MCP tool | $0.0005 per package row + $0.0015 per vulnerability row emitted (after the severity filter and de-duplication), + Apify's $0.00005 Actor-start event. Error and warning rows are never charged. | Cost scales with what you check: a 500-package lockfile with no HIGH+ findings costs 500 × $0.0005 = $0.25 per run. For a gate that runs on every push, consider the free lookup for the pass/fail decision and the Actor for the detailed report. ## How it compares with alternatives Option | Good for | Trade-offs | npm audit | Built into npm, free, uses the GitHub Advisory Database | npm projects only | pip-audit | Free PyPA tool for Python environments and requirements files | Python only | OSV-Scanner | Free, open-source CLI from Google that scans lockfiles and directories against OSV, many ecosystems | A binary to install and run in CI; output formatting and gating are up to you | Dependabot / GitHub security alerts | Automatic alerts and update PRs on GitHub repositories | Works on repositories hosted on GitHub; not built for checking an arbitrary package list from an agent or another CI system | Fairquery OSV Actor | One API/MCP call from CI, agents or n8n; CVSS, CVE mapping, severity filter, de-duplication, recommended version as JSON rows | Paid per package and per vulnerability; exact versions only; no reachability analysis, license checks or container image scanning | ## FAQ ### How do I fail a CI build when a dependency has a known vulnerability? Pin exact versions (lockfile or pip freeze), query OSV.dev, and exit non-zero when results contain vulnerabilities at or above your severity bar. The post shows a free jq + OSV querybatch gate and a GitHub Actions job using the Fairquery OSV Actor with minSeverity HIGH. ### Is the OSV.dev API free? Yes. OSV.dev is free and needs no API key. Its querybatch endpoint returns advisory IDs per package; severity and fix details come from separate per-advisory requests. ### What does the Fairquery OSV Actor add over the raw OSV API? Manifest parsing (package.json, package-lock.json v1-v3, requirements.txt, go.mod), computed CVSS v3 scores, CVE mapping, a minSeverity filter, alias de-duplication and a recommended upgrade version, as JSON rows for CI or agents. ### How much does it cost? $0.0005 per package checked plus $0.0015 per vulnerability row emitted after filtering and de-duplication. Error and warning rows are free. Our test (3 packages, 23 HIGH/CRITICAL vulnerabilities) would cost $0.036. ### Which ecosystems are supported? Everything OSV supports, including npm, PyPI, Go, Maven, crates.io, RubyGems, NuGet, Packagist, Pub and Hex. Exact versions are required; package.json ranges are approximated to their lower bound with a warning. ### Does no result mean my dependencies are safe? No. It means OSV has no matching advisory. No database is complete, and the check does not cover reachability, licenses or container images. Honest limits: data comes from OSV.dev and its upstream databases. No advisory database is complete, and an empty result does not prove a package is safe. Source and docs: github.com/fairquery/osv-dependency-vulnerability-check (MIT) · Apify Store · Fairquery OSV for VS Code · all Fairquery tools --- # How do I give my AI agent trust and safety tools via MCP? URL: https://getfairquery.com/posts/give-ai-agent-trust-safety-tools-mcp-server/ Published: 2026-10-17 Short answer: add one MCP server to your agent's config: @fairquery/agent-trust-mcp gives Claude Desktop, Cursor, VS Code or any MCP client 12 trust-and-safety tools: PII and secret redaction, email checks, sanctions screening (OFAC, EU, UK, UN) and OSV dependency-vulnerability lookups. 7 tools are free and need no account; they run on your machine (the two OSV lookups call the free OSV.dev API directly). The 5 deeper tools are optional and bill your own Apify account per result. It is listed in the official MCP Registry as io.github.fairquery/fairquery-agent-trust-mcp and on Glama. ## Why an agent needs trust and safety tools Agents increasingly read untrusted input (tickets, emails, scraped pages, uploaded files) and act on it (sign up users, onboard suppliers, add dependencies, send data to other models). Four checks come up again and again: - Is there personal data or a secret in this text? Redact it before it goes into a prompt, a log or a vector store. - Is this email address real and not throwaway? Check syntax, typos, disposable and role addresses, MX records. - Is this person or company on a sanctions list? Pre-screen against OFAC, EU, UK and UN lists before onboarding or paying. - Is this package version vulnerable? Check against OSV.dev before installing or approving a dependency. ## The 12 tools Tool | Cost | What it does | pii_list_detectors | Free (local) | Catalog of detectors, redaction styles and pricing for every paid tool | pii_scan_text | Free (local) | Finds PII and secrets in text; findings are paged and show masked values only | pii_redact_text | Free (local) | Redacts with placeholder, mask or hash style; can return a placeholder map | pii_guard_documents | $0.0015/doc + $0.0002/finding | Adds NER names/addresses, URLs and PDFs, and restore mode | email_check_syntax | Free (local) | Syntax, normalization, typo suggestions, disposable/role/free-provider flags | email_validate_deliverability | $0.00079/email | DNS/MX, null MX, SPF/DMARC, mail provider, verdict | sanctions_list_sources | Free (local) | Lists covered, issuing authorities, disclaimer | sanctions_screen_names | $0.003/name + $0.007/call | Fuzzy screening against OFAC SDN/Consolidated, EU, UK and UN | osv_query_packages | Free (api.osv.dev) | Vulnerable or not, plus advisory IDs, for up to 200 exact versions | osv_get_vulnerability | Free (api.osv.dev) | Full OSV advisory records | osv_audit_dependencies | $0.0005/package + $0.0015/vuln | Manifest/lockfile audit with severity, fixes and recommended version | trust_screen_lead | $0.00079/email + $0.003/name + $0.007/call | Email check and sanctions screening in parallel, merged into a pass/review/block decision per lead | Paid tools run the public Fairquery Apify Actors (llm-data-guard-pii-secrets, email-validator-mx-disposable, sanctions-watchlist-screener, osv-dependency-vulnerability-check) with your APIFY_TOKEN, so usage is billed to your Apify account at the prices above. Every tool description has WHAT / WHEN / WHEN NOT / COST / RETURNS / EXAMPLE sections so the model can pick the right one, outputs are paged (limit, offset, nextOffset, hasMore), and errors use named codes. ## How to install it Requires Node.js 20 or later. Leave out APIFY_TOKEN if you only want the free tools. ### Claude Desktop (claude_desktop_config.json) ``` { "mcpServers": { "fairquery-trust": { "command": "npx", "args": ["-y", "@fairquery/agent-trust-mcp"], "env": { "APIFY_TOKEN": "" } } } } ``` ### Cursor (~/.cursor/mcp.json or .cursor/mcp.json) ``` { "mcpServers": { "fairquery-trust": { "command": "npx", "args": ["-y", "@fairquery/agent-trust-mcp"], "env": { "APIFY_TOKEN": "" } } } } ``` ### VS Code (.vscode/mcp.json) ``` { "inputs": [{ "type": "promptString", "id": "apify-token", "description": "Apify token (optional, paid tools only)", "password": true }], "servers": { "fairquery-trust": { "type": "stdio", "command": "npx", "args": ["-y", "@fairquery/agent-trust-mcp"], "env": { "APIFY_TOKEN": "${input:apify-token}" } } } } ``` Other ways to run it: npx -y github:fairquery/fairquery-agent-trust-mcp (straight from GitHub), node dist/index.js --http from a clone for Streamable HTTP on http://127.0.0.1:3000/mcp, or the Docker image built from the repo's Dockerfile. ## Real outputs from the free tools We connected to version 1.0.3 from npm with the MCP SDK client (no APIFY_TOKEN set) on October 17, 2026. tools/list returned the 12 tools above. Some calls: ### pii_redact_text Input: Ticket #4411 from Jane Doe , phone +1 415-555-0132. Card 4111 1111 1111 1111 was declined. Debug log: AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE (test card number, AWS documentation example key). Returned in 5 ms: ``` { "meta": { "engine": "local", "redaction_style": "placeholder", "placeholder_map": { "[EMAIL_1]": "jane.doe@example.com", "[PHONE_1]": "+1 415-555-0132", "[CREDIT_CARD_1]": "4111 1111 1111 1111", "[AWS_ACCESS_KEY_1]": "AKIAIOSFODNN7EXAMPLE" } }, "items": [ { "type": "redacted_document", "finding_count": 4, "risk_level": "critical", "output_text": "Ticket #4411 from Jane Doe <[EMAIL_1]>, phone [PHONE_1]. Card [CREDIT_CARD_1] was declined. Debug log: AWS_ACCESS_KEY_ID=[AWS_ACCESS_KEY_1]" } ] } ``` ### email_check_syntax Input ["jane@acme.com", "bob@gmial.com", "info@mailinator.com"], abridged to the key fields. No DNS lookups are done by the free tool, hence DNS_NOT_CHECKED; use email_validate_deliverability for MX/SPF/DMARC. ``` [ { "email": "jane@acme.com", "verdict": "unknown", "reasons": [ "DNS_NOT_CHECKED" ], "did_you_mean": null, "is_disposable": false, "is_role_account": false }, { "email": "bob@gmial.com", "verdict": "risky", "reasons": [ "POSSIBLE_TYPO", "DNS_NOT_CHECKED" ], "did_you_mean": "bob@gmail.com", "is_disposable": false, "is_role_account": false }, { "email": "info@mailinator.com", "verdict": "risky", "reasons": [ "DISPOSABLE_DOMAIN", "ROLE_ACCOUNT", "DNS_NOT_CHECKED" ], "did_you_mean": null, "is_disposable": true, "is_role_account": true } ] ``` ### osv_query_packages ``` { "meta": { "vulnerable_count": 2, "source": "https://api.osv.dev/v1/querybatch" }, "items": [ { "ecosystem": "npm", "name": "lodash", "version": "4.17.20", "vulnerable": true, "vuln_count": 5, "vuln_ids": [ "GHSA-29mw-wpgm-hmr9", "GHSA-35jh-r3h4-6jhm", "GHSA-f23m-r3pf-42rh", "GHSA-r5fr-rjxr-66jc", "GHSA-xxjr-mmjv-4gpg" ] }, { "ecosystem": "PyPI", "name": "django", "version": "3.2.0", "vulnerable": true, "vuln_count": 66, "vuln_ids": [ "GHSA-2gwj-7jmv-h26r", "GHSA-2hrw-hx67-34x6", "GHSA-3h9f-r86x-qvjx", "… 63 more" ] }, { "ecosystem": "npm", "name": "express", "version": "4.21.2", "vulnerable": false, "vuln_count": 0, "vuln_ids": [] } ] } ``` ### A paid tool without a token Calling sanctions_screen_names without APIFY_TOKEN does not fail silently or spend anything; it returns a named error: ``` { "error": "APIFY_TOKEN_MISSING", "message": "This is a paid tool. Set APIFY_TOKEN to YOUR OWN Apify API token (https://console.apify.com/settings/integrations); usage is billed to your Apify account. Free tools work without a token.", "retryable": false } ``` With a token, the paid tools return the same rows as the underlying Actors; see the real runs in our guides on sanctions screening, PII redaction and OSV checks in CI. ## Example prompts - "Redact this support ticket, then summarize it." (pii_redact_text) - "Is lodash@4.17.20 vulnerable? Show the advisories." (osv_query_packages, then osv_get_vulnerability) - "Check these signup emails for typos and throwaway domains." (email_check_syntax) - "Screen these 20 new suppliers and tell me which need review." (trust_screen_lead or sanctions_screen_names, paid) ## Security and privacy The free tools never contact Apify or Fairquery; PII tools make no network calls at all, and the OSV tools call only api.osv.dev. The code contains no secrets, and your token is read only from your own environment. The source is MIT-licensed on GitHub, with a CI workflow and a SECURITY.md. ## How it compares with alternatives Option | Good for | Trade-offs | Apify MCP server (mcp.apify.com) | Calling any Apify Actor, including all Fairquery Actors, from an agent | Generic remote Actor interface; Actor runs are billed per run and there are no offline PII or email tools | Single-purpose Fairquery MCP servers (in each Actor's GitHub repo) | Exposing just one capability | Separate servers to configure for each job | Building your own tools | Full control | You maintain detectors, list parsers, OSV handling and tool descriptions yourself | Fairquery Agent Trust MCP | Four common safety checks in one server, free local tier, paid depth on demand | Pattern-based PII detection; sanctions screening is not a certified service; no SMTP mailbox probing | ## FAQ ### What is the easiest way to add PII redaction and safety checks to Claude Desktop or Cursor? Add @fairquery/agent-trust-mcp to your MCP config with command 'npx' and args ['-y', '@fairquery/agent-trust-mcp']. Its free tools work without an account; set APIFY_TOKEN only for the paid tools. ### Which tools are free? Seven: pii_list_detectors, pii_scan_text, pii_redact_text, email_check_syntax and sanctions_list_sources run locally; osv_query_packages and osv_get_vulnerability call the free OSV.dev API directly. ### What do the paid tools cost? pii_guard_documents $0.0015/document + $0.0002/finding; email_validate_deliverability $0.00079/email; sanctions_screen_names $0.003/name + $0.007/call; osv_audit_dependencies $0.0005/package + $0.0015/vulnerability; trust_screen_lead $0.00079 per lead with an email + $0.003 per lead with a name (+ $0.007 per call when any lead has a name). They are billed to your own Apify account. ### Is it in the official MCP Registry? Yes, as io.github.fairquery/fairquery-agent-trust-mcp (npm package @fairquery/agent-trust-mcp, stdio transport). It is also listed on Glama. ### Does my data leave my machine? Not with the free PII and email tools; they make no network calls. The free OSV tools call only api.osv.dev. Paid tools send their input to the corresponding Fairquery Apify Actor run in your Apify account. ### What happens if the agent calls a paid tool without a token? It gets a named APIFY_TOKEN_MISSING error and nothing is charged. Disclaimer: sanctions results come from automated fuzzy matching and are not a certified screening service or a legal determination; review every potential match yourself. Email checks do not probe SMTP mailboxes. PII detection is pattern-based and can miss data. Links: npm: @fairquery/agent-trust-mcp · GitHub · MCP Registry entry · Glama · Fairquery on Apify · all Fairquery tools --- # How to validate email addresses via API: MX, disposable and typo checks URL: https://getfairquery.com/posts/validate-email-addresses-api-mx-disposable-typo/ Published: 2026-10-17 Short answer: to validate an email address via API without sending mail, run four checks: (1) syntax, (2) DNS: does the domain exist and publish MX records (and not a "null MX"), (3) risk flags: disposable domain, role account (info@, sales@), free webmail, and (4) typo detection on the domain (gmial.com → gmail.com). The Fairquery Email Validator does all of this in one call for $0.00079 per email on Apify, and is also available on RapidAPI. It returns a valid / risky / invalid / unknown verdict with reason codes. ## What "email validation" can and cannot tell you Email validation is about answering one practical question before you store or email an address: can this address plausibly receive mail, and is it worth keeping? There are two families of techniques: - DNS-based checks (what this post covers): look at the address and its domain. Is it well-formed? Does the domain exist? Does it publish mail servers? Is it a known throwaway domain? These checks are fast, cheap, and never touch the recipient's mailbox. - SMTP probing: connecting to the recipient's mail server and asking whether a mailbox exists without delivering a message. It can confirm more, but many providers deliberately give ambiguous answers ("catch-all" domains accept everything), and probing at volume can damage your IP reputation and conflict with provider terms. The Fairquery validator deliberately does not probe SMTP. So valid means "well-formed, deliverable domain, no risk flags", not "mailbox confirmed". That is an honest limit of every DNS-only validator, and you should design your workflow around it (for example, combine it with a double opt-in confirmation email for signups). ## The checks, one by one ### 1. Syntax The full RFC 5322 grammar allows strange things (quoted local parts, comments) that almost no real mailbox uses. A pragmatic validator accepts the RFC 5321/5322 subset people actually use and converts internationalized domains (IDN) to punycode before looking them up. Regex-only validation stops here, which is why it lets jane@acme.con through. ### 2. Domain and MX records Mail is delivered to the hosts listed in the domain's MX records. If the domain doesn't exist (DOMAIN_NOT_FOUND) or has no mail server at all (NO_MAIL_SERVER), mail will bounce. Two subtleties: - Null MX (RFC 7505): a domain can publish an MX record of . to say "I never accept mail". That is a definite invalid (NULL_MX). - Implicit MX: with no MX record, senders fall back to the domain's A/AAAA record. Mail may work, but it is unusual for real mail domains, so it is flagged IMPLICIT_MX_ONLY (risky) rather than passed silently. ### 3. Disposable, role and free-provider flags - Disposable domains (temporary inboxes) are a common source of fake signups and trial abuse. The validator uses a bundled CC0 list from the open-source disposable-email-domains project (9,800+ domains, subdomains matched). No list is complete; new throwaway domains appear constantly. - Role accounts like info@, sales@, support@ reach a team or a shared inbox, not a person. Fine for some uses, undesirable for others; set allowRoleAccounts: false to mark them risky. - Free webmail (Gmail, Outlook.com, Yahoo and so on) is normal for consumers but often a signal on B2B lead forms. Set allowFreeProviders: false for B2B lists. ### 4. Typo suggestions A surprising number of bad addresses are simple domain typos: gmial.com, hotmial.com, outlok.com. The validator compares the domain against popular providers by edit distance and returns a did_you_mean suggestion. The best place to use this is on the signup form itself ("Did you mean jane@gmail.com?"), before the bad address ever reaches your database. ### Bonus: mail provider, SPF and DMARC The response also reports the detected mail provider (for example Google Workspace or Microsoft 365) and whether the domain publishes SPF and DMARC, including the DMARC policy. That's useful context for lead qualification and for spotting domains that are set up for real business mail. ## Free options - Do it yourself. A syntax check plus a DNS MX lookup is a few lines in most languages (for example dig MX example.com, Node's dns.resolveMx, or Python's dnspython). Add the free disposable-email-domains list and you have a basic validator. You'll maintain the typo logic, role list, caching and edge cases (null MX, IDN) yourself. - Fairquery's free offline tool. The open-source repo (MIT) ships a local MCP server with a free check_email_syntax tool that runs offline without a token. ## API option: Fairquery Email Validator Inputs (all optional except emails): Field | Default | Meaning | emails | – | Addresses; entries can be comma/semicolon/newline lists or Name | checkDns | true | MX/A lookups (off = syntax only) | checkSpfDmarc | true | Report SPF/DMARC and DMARC policy | allowRoleAccounts | true | If false, role addresses become risky | allowDisposable | false | If true, disposable domains are not risky | allowFreeProviders | true | If false, free webmail becomes risky | maxEmails | 10000 | Safety cap per run | ### curl Run the Actor synchronously and get results back in one request (replace $APIFY_TOKEN with your Apify API token): ``` curl -X POST \ "https://api.apify.com/v2/acts/fairquery~email-validator-mx-disposable/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"emails": ["jane@acme.com", "info@apify.com", "bob@gmial.com"], "allowFreeProviders": false}' ``` ### JavaScript (Node.js) ``` import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('fairquery/email-validator-mx-disposable').call({ emails: ['jane@acme.com', 'bob@gmial.com'], allowRoleAccounts: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const r of items.filter((i) => i.type === 'email')) { console.log(r.email, r.verdict, r.reasons, r.did_you_mean ?? ''); } ``` ### Python ``` import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("fairquery/email-validator-mx-disposable").call( run_input={"emails": ["jane@acme.com", "bob@gmial.com"], "allowFreeProviders": False} ) for r in client.dataset(run["defaultDatasetId"]).iterate_items(): if r["type"] == "email": print(r["email"], r["verdict"], r["reasons"], r.get("did_you_mean")) ``` Prefer RapidAPI's key-based billing and code snippets? Use the RapidAPI listing; its endpoint docs and playground show request formats for many languages. ### What comes back Each address gets one row. Abridged shape (field names are real; values depend on live DNS at check time): ``` { "type": "email", "email": "bob@gmial.com", "domain": "gmial.com", "verdict": "risky", "reasons": ["POSSIBLE_TYPO", "..."], "syntax_valid": true, "is_disposable": false, "is_role_account": false, "is_free_provider": false, "did_you_mean": "bob@gmail.com", "domain_exists": true, "has_mx": false, "null_mx": false, "implicit_mx": true, "mx_records": [], "mail_provider": null, "has_spf": false, "has_dmarc": false, "dmarc_policy": null, "source_url": "https://cloudflare-dns.com/dns-query?name=gmial.com&type=MX", "checked_at": "..." } ``` Reason codes: INVALID_SYNTAX, DOMAIN_NOT_FOUND, NULL_MX, NO_MAIL_SERVER (→ invalid); DISPOSABLE_DOMAIN, ROLE_ACCOUNT, FREE_PROVIDER, POSSIBLE_TYPO, IMPLICIT_MX_ONLY (→ risky, depending on the allow flags); DNS_NOT_CHECKED (→ unknown). If a DNS lookup fails you get an error row (DNS_LOOKUP_FAILED, with retryable), and error rows are never charged. Duplicates (case-insensitive) are skipped. ### Using it from an AI agent Agents can call the Actor through Apify MCP with {"actor": "fairquery/email-validator-mx-disposable", "input": {"emails": ["jane@acme.com"]}}, or run the stdio MCP server from the GitHub repo (validate_emails, plus the free check_email_syntax). ## Pricing $0.00079 per email checked on Apify (pay per event; error rows are free). DNS lookups are cached per domain, so lists with many addresses at the same company are efficient. RapidAPI pricing is shown on the listing. ## Practical recommendations - Signup forms: reject invalid, show did_you_mean inline, and block or challenge disposable domains. Still send a confirmation email. - B2B lead lists: set allowFreeProviders: false and allowRoleAccounts: false; review risky rows rather than deleting them blindly. - Old CRM data: re-check periodically; domains expire and companies change mail setups. - Compliance: validation is not consent. Follow CAN-SPAM, GDPR/ePrivacy and similar rules when emailing. ## FAQ ### How do I check if an email address is valid without sending an email? Check the syntax, then look up the domain's DNS: it must exist and publish MX records (and not a null MX). Add disposable-domain, role-account and typo checks. The Fairquery Email Validator does all of this via API without sending mail or probing SMTP. ### Can an API confirm that a mailbox really exists? Not reliably without SMTP probing, and even then catch-all domains accept everything. The Fairquery validator does not probe SMTP; 'valid' means well-formed, deliverable domain and no risk flags, not mailbox confirmed. ### How do I detect disposable email addresses? Match the domain (including subdomains) against a maintained disposable-domain list. Fairquery uses the CC0 disposable-email-domains list (9,800+ domains) and returns is_disposable plus a DISPOSABLE_DOMAIN reason code. ### What is a null MX record? An MX record pointing to '.' (RFC 7505) that says the domain never accepts email. Addresses at such domains are invalid. ### How much does the Fairquery Email Validator cost? $0.00079 per email checked on Apify, pay per event. Error rows are never charged. It is also listed on RapidAPI. ### Can AI agents use it? Yes. Call Actor fairquery/email-validator-mx-disposable through Apify MCP, or run the stdio MCP server from the GitHub repo, which exposes validate_emails and a free offline check_email_syntax tool. Honest limits: DNS-based validation cannot guarantee a mailbox exists or accepts mail. Use results to reduce bounces and filter obvious junk, not as proof of deliverability. Source and docs: github.com/fairquery/email-validator-mx-disposable · Apify Store · RapidAPI · all Fairquery tools