How do I track changes to competitors' pricing, terms and careers pages without false alarms?

Fairquery · October 18, 2026 · 9 min read

Why screenshot and raw-text diffs are noisy on competitor pages

Pricing and legal pages look static, but their HTML is not. Every request can carry a new CSRF token, a "last updated 3 minutes ago" label, a rotating testimonial, a cookie banner, an A/B-tested headline or a cache-busting query string. Visual and plain-text monitors report all of that as a change. After a week of false alarms people stop reading the alerts, and the one real price increase goes unnoticed.

What you actually want to know is narrower: did a plan's price, currency or billing period change, was a plan added or removed, which section of the Terms changed and how, which roles opened or closed, which product pages appeared? That needs a comparison that understands the page type.

What to compare on each page type

A free, minimal change check in Python

This script respects robots.txt, identifies itself, drops scripts, nav and footer, removes dates and times, and stores a hash per URL. Run it from cron; it prints baseline, unchanged or CHANGED. It tells you that something changed, not what, and it will still trip on A/B tests and rotating content.

# Minimal "did this page's text change?" check: polite, robots-aware, standard library only.
import hashlib, json, os, re, sys, urllib.error, urllib.request, urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urlsplit

UA = "MyChangeCheck/0.1 (+mailto:you@example.com)"
url = sys.argv[1]                      # e.g. https://basecamp.com/about/policies/terms

base = "{0.scheme}://{0.netloc}".format(urlsplit(url))
rp = urllib.robotparser.RobotFileParser()
try:   # fetch robots.txt with our own user agent; a 4xx means no rules
    rp.parse(urllib.request.urlopen(urllib.request.Request(base + "/robots.txt", headers={"User-Agent": UA}), timeout=20).read().decode("utf-8", "replace").splitlines())
except urllib.error.HTTPError as e:
    if e.code >= 500: sys.exit("robots.txt unavailable; not fetching")
    rp.parse([])
if not rp.can_fetch(UA, url):
    sys.exit("robots.txt disallows " + url)

class Text(HTMLParser):
    skip, out = 0, []
    def handle_starttag(self, t, a): self.skip += t in ("script", "style", "nav", "footer")
    def handle_endtag(self, t): self.skip -= t in ("script", "style", "nav", "footer") and self.skip > 0
    def handle_data(self, d):
        if not self.skip and d.strip(): self.out.append(d.strip())

html = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": UA}), timeout=20).read().decode("utf-8", "replace")
p = Text(); p.feed(html)
text = re.sub(r"\b(19|20)\d\d-\d\d-\d\d\b|\b\d{1,2}:\d\d\b", "", "\n".join(p.out))   # drop dates/times
digest = hashlib.sha256(text.encode()).hexdigest()

state = json.load(open("state.json")) if os.path.exists("state.json") else {}
print("baseline" if url not in state else ("unchanged" if state[url] == digest else "CHANGED"))
state[url] = digest; json.dump(state, open("state.json", "w"))

Semantic tracking with the Fairquery Actor

The Actor does the type-aware comparison for you. Give it pages (or just competitor domains plus the page types to find), schedule it, and each run returns only the changes, each with a plain-English summary, a severity and the details. The first run stores a baseline in a named key-value store in your own Apify account.

{
 "watchlistName": "fq-test-main",
 "pages": [
  {
   "url": "https://basecamp.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://linear.app/pricing",
   "type": "pricing"
  },
  {
   "url": "https://tailscale.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://zapier.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://about.gitlab.com/pricing/",
   "type": "pricing"
  },
  {
   "url": "https://github.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://www.notion.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://basecamp.com/about/policies/terms",
   "type": "terms"
  },
  {
   "url": "https://github.com/site/terms",
   "type": "terms"
  },
  {
   "url": "https://linear.app/terms",
   "type": "terms"
  },
  {
   "url": "https://linear.app/privacy",
   "type": "privacy"
  },
  {
   "url": "https://tailscale.com/terms",
   "type": "terms"
  },
  {
   "url": "https://slack.com/terms-of-service",
   "type": "terms"
  },
  {
   "url": "https://jobs.ashbyhq.com/linear",
   "type": "careers"
  },
  {
   "url": "https://tailscale.com/careers",
   "type": "careers"
  },
  {
   "url": "https://www.figma.com/careers/",
   "type": "careers"
  },
  {
   "url": "https://linear.app/changelog",
   "type": "changelog"
  },
  {
   "url": "https://posthog.com/changelog",
   "type": "changelog"
  },
  {
   "url": "https://tailscale.com/",
   "type": "product"
  },
  {
   "url": "https://job-boards.greenhouse.io/anthropic",
   "type": "careers"
  },
  {
   "url": "https://jobs.lever.co/palantir",
   "type": "careers"
  },
  {
   "url": "https://slack.com/pricing",
   "type": "pricing"
  },
  {
   "url": "https://www.dropbox.com/plans",
   "type": "pricing"
  },
  {
   "url": "https://www.amazon.com/gp/cart/view.html",
   "type": "generic"
  }
 ],
 "careersKeywords": [
  "AI",
  "machine learning",
  "enterprise"
 ],
 "watchKeywords": [
  "AI",
  "SOC 2"
 ],
 "minSeverity": "low",
 "includeDiff": true
}

Rows from our test runs on 2026-10-18. The old values were simulated by editing the stored baseline, so the live page then differed from it; the current values are the real ones from Basecamp, Linear and Linear's Ashby board:

[
 {
  "schema_version": "1.0.0",
  "watchlist": "fq-test-main",
  "checked_at": "2026-10-18T02:15:41.208Z",
  "url": "https://linear.app/pricing",
  "page_type": "pricing",
  "label": null,
  "domain": "linear.app",
  "type": "change",
  "final_url": "https://linear.app/pricing",
  "severity": "high",
  "change_kind": "price_changed",
  "summary": "\"Business\" price changed from $14/month per user to $16/month per user (up 14.3%).",
  "details": {
   "plan": "Business",
   "from": {
    "amount": 14,
    "currency": "USD",
    "period": "month",
    "billing": null,
    "unit": "user",
    "raw": "$16"
   },
   "to": {
    "amount": 16,
    "currency": "USD",
    "period": "month",
    "billing": null,
    "unit": "user",
    "raw": "$16"
   },
   "change_pct": 14.3
  },
  "previous_checked_at": "2026-10-18T02:10:52.136Z"
 },
 {
  "schema_version": "1.0.0",
  "watchlist": "fq-test-main",
  "checked_at": "2026-10-18T02:15:41.208Z",
  "url": "https://basecamp.com/pricing",
  "page_type": "pricing",
  "label": null,
  "domain": "basecamp.com",
  "type": "change",
  "final_url": "https://basecamp.com/pricing",
  "severity": "high",
  "change_kind": "price_changed",
  "summary": "\"Pro\" price changed from $79/month to $99/month (up 25.3%).",
  "details": {
   "plan": "Pro",
   "from": {
    "amount": 79,
    "currency": "USD",
    "period": "month",
    "billing": null,
    "unit": null,
    "raw": "$79"
   },
   "to": {
    "amount": 99,
    "currency": "USD",
    "period": "month",
    "billing": null,
    "unit": null,
    "raw": "$99"
   },
   "change_pct": 25.3
  },
  "previous_checked_at": "2026-10-18T02:10:52.136Z"
 },
 {
  "schema_version": "1.0.0",
  "watchlist": "fq-test-main",
  "checked_at": "2026-10-18T02:15:41.208Z",
  "url": "https://linear.app/terms",
  "page_type": "terms",
  "label": null,
  "domain": "linear.app",
  "type": "change",
  "final_url": "https://linear.app/terms",
  "severity": "high",
  "change_kind": "legal_text_changed",
  "summary": "Terms of service: 2 sections changed: \"Terms of Service\" (modified: +1/−1 sentences), \"Linear’s Ownership\" (added). Added: “(“Linear”).”",
  "details": {
   "sections_changed": 2,
   "changed_chars": 872,
   "sections": [
    {
     "heading": "Terms of Service",
     "change": "modified",
     "added": [
      "(“Linear”)."
     ],
     "removed": [
      "Customer may terminate this Agreement at any time with thirty days written notice and will receive a pro-rata refund of prepaid fees."
     ],
     "more_added": 0,
     "more_removed": 0
    },
    {
     "heading": "Linear’s Ownership",
     "change": "added",
     "added": [
      "Linear owns the Service, Software, Linear Content, Documentation, and anything else provided by Linear to Customer (collectively the “Linear Materials”)."
     ],
     "removed": [],
     "more_added": 0,
     "more_removed": 0
    }
   ]
  },
  "previous_checked_at": "2026-10-18T02:10:52.136Z"
 },
 {
  "schema_version": "1.0.0",
  "watchlist": "fq-test-main",
  "checked_at": "2026-10-18T02:15:41.208Z",
  "url": "https://jobs.ashbyhq.com/linear",
  "page_type": "careers",
  "label": null,
  "domain": "jobs.ashbyhq.com",
  "type": "change",
  "final_url": "https://jobs.ashbyhq.com/linear",
  "severity": "low",
  "change_kind": "jobs_changed",
  "summary": "3 new roles (3 Product); 1 role closed (1 Engineering). Open roles: 27 → 29 (+7%). Examples: Senior / Staff Product Engineer, Product Engineer, Senior / Staff Product Engineer.",
  "details": {
   "open_before": 27,
   "open_after": 29,
   "opened": [
    {
     "id": "069c4628-88d7-4e4d-b393-c996fc7f3076",
     "title": "Senior / Staff Product Engineer",
     "department": "Product",
     "location": "Europe"
    }
   ],
   "closed": [
    {
     "id": "sim-closed-1",
     "title": "Senior Simulated Engineer",
     "department": "Engineering",
     "location": "North America"
    }
   ],
   "keyword_matches": []
  },
  "previous_checked_at": "2026-10-18T02:10:52.136Z"
 }
]

Two noise filters matter most in practice. First, a change is confirmed with a second read before it is reported: in our tests Slack's pricing page served two variants within seconds, and the confirmation read filtered it out. Second, the Actor remembers recent versions, so a page that alternates between two versions across runs alerts once and then only as a low-severity "switched back" item. Immediate reruns over 23 real pages (pricing, terms, careers, changelogs, a 1,281-URL sitemap) reported zero changes.

Politeness rules we follow

• Public, no-login pages only. No cookies or sessions, no CAPTCHA solving, no proxies to get around blocks.
• robots.txt (RFC 9309) is checked for every page and redirect; disallowed pages are skipped and reported. Crawl-delay is honoured.
• Honest user agent (FairqueryChangeTracker/1.0 with a contact address), one request at a time per host, at least 1.5 s apart, conditional requests (ETag / If-Modified-Since).
• Plain HTTP first. A headless browser is opt-in, only for JavaScript-rendered pricing pages, and still respects robots.txt.
• A 403 or 429 stops all requests to that host for the rest of the run.

Track a competitor's pricing and hiring together

Price and headcount move together. A competitor that raises prices while opening a dozen enterprise sales and solutions-engineering roles is moving upmarket; one that freezes hiring and quietly discounts is defending share. Watching both at once turns a single data point into a trend you can act on.

The tracker already reads a careers page that runs on Greenhouse, Lever or Ashby through that board's public API and reports new and closed roles by department, so a pricing change and a hiring change land in the same run, with the same severity and the same webhook alert. For deeper hiring intelligence — every open role across many companies, how long each has been open, repost and ghost-job signals, and keyword filters such as "AI" or "enterprise" — pair it with the Fairquery Company Jobs Feed (guide). Run the tracker on the pages you want watched for change, and the Jobs Feed when you want the full, filterable list of who is hiring and for what.

Cost compared with change-monitoring services

Prices from the vendors' pricing pages on 2026-10-18. Freshness is set by your schedule in every case; none of these, including ours, sees a change made and reverted between two checks.

FAQ

Why do website change monitors send so many false alerts?

Most compare screenshots or raw text, so rotating tokens, timestamps, cookie banners, testimonials and A/B-tested headlines all count as changes. Comparing by page type (plans and prices, legal sections, job ids, sitemap URLs) and masking dates and tokens removes most of that noise.

How can I detect a competitor's price change automatically?

Fetch the pricing page on a schedule, extract each plan's name, price, currency and billing period, and compare with the previous run. The Fairquery Competitor Change Tracker does this and reports changes such as a Pro plan price moving from $79/month to $99/month (up 25.3%), plus plans added or removed.

Can I see exactly what changed in a Terms of Service or Privacy Policy?

Yes. The Actor splits legal pages into sections by heading and reports which sections were added, removed, renamed or modified, with the removed and added sentences. Edits about liability, arbitration, data sharing, AI training, fees or termination are marked high severity.

Is it real-time?

No. Each run is a check, so freshness equals your schedule (for example hourly or daily with an Apify Schedule). A change made and reverted between two runs is not seen.

Does it respect robots.txt?

Yes. robots.txt is checked for every page and redirect, disallowed pages are skipped and reported, Crawl-delay is honoured, and a 403 or 429 stops requests to that host for the run. It reads public pages only: no login, cookies, CAPTCHA solving or proxies.

Can it track hiring at competitors?

Yes. If a careers page uses Greenhouse, Lever or Ashby, the Actor reads that board's public API and reports new and closed roles by department, with optional keyword counts such as AI or enterprise. For the full, filterable list of who is hiring, pair it with the Fairquery Company Jobs Feed.

How much does it cost?

Pay-per-event pricing: $0.00049 per page checked and $0.0049 per change reported, $0.0059 per page rendered with the optional headless browser. Twenty pages checked hourly is about $7 a month plus changes.

Docs: Apify Store · all Fairquery tools