On 6 November 2026 we fetched public files from 50 hand-picked SaaS marketing homepages and recorded what each one says to AI crawlers. The full write-up, CSV and method notes live on our guide (canonical):

https://shephouse-guides.pages.dev/saas-homepage-ai-crawler-access-benchmark/

Published on that page 6 November 2026. Data collected the same day, 12:14–12:15 PM America/Los_Angeles (20:14:33–20:14:55 UTC).

What the numbers are — and are not

This is a robots-policy and HTTP-fetch snapshot. A Disallow line is a published permission. It is not a log of GPTBot, ClaudeBot or PerplexityBot visits, and it is not a ranking or citation score.

The 50 sites are a convenience sample of well-known brands (collaboration, CRM, dev tools, payments, builders, marketing/analytics). It is not random and not representative of small, new, or non-English SaaS. Every percentage below is “of this list,” not of the industry.

Independently recomputed headlines

Source: public CSV and summary on the guide (/assets/e3/results.csv, /assets/e3/summary.json), matching the acquisition-ledger copies. Robots-rule percentages use base 48 (parsed robots.txt). Homepage schema uses base 48 (HTTP 200).

Training vs search (robots policy only): figma.com, canva.com and loom.com Disallow GPTBot on / while still allowing OAI-SearchBot on /. figma.com and canva.com also Disallow ClaudeBot on / while leaving Claude-SearchBot / PerplexityBot allowed on /. That describes published rules, not what any vendor actually fetched.

No JSON-LD types were found in the raw HTML of slack.com, airtable.com, canva.com, shopify.com, miro.com, linear.app and grammarly.com — JavaScript may add more; we did not render JS. canva.com had 57 words after stripping script/style.

A missing robots.txt is often treated as allow-all in the wild. We did not re-label freshworks.com that way; it is excluded from the robots denominators.

Method, in brief

HTTPS GET of /robots.txt, /llms.txt, /llms-full.txt and / per domain, research UA ShepHouseResearch/1.0, 20-second timeout, redirects followed. No spoofed vendor-bot UAs, no CAPTCHA or WAF bypass, no retries. A script re-run meant each URL was requested twice; the second pass is what we analyzed. Cost: $0 Apify — direct HTTP only.

We evaluated 12 tokens on path / only (specific robots group else *; longest Allow/Disallow match; Allow wins ties). Google-Extended is a control token, not a separate crawler UA.

Limits: no JavaScript render; CDN/WAF bot rules are invisible; permissions ≠ visits; the snapshot can go stale immediately.

If you want to inspect the same files

The guide has a short self-check (accidental Disallow: /, training vs search tokens, real plain-text llms.txt, raw-HTML JSON-LD, content visible without JS) plus the CSV.

Optional ShepHouse tools — commercial, and they still do not guarantee citations, rankings, or crawler visits:

• AI Search Readiness (GEO) Audit on Apify — about $0.005 per site audited; failures free (confirm the Store tab).
• GEO Fix Checklist, $9 (Polar checkout)
• Free GEO audit checklist

Please set the Hashnode canonical URL to the guide page so this post does not compete with the source of record.

A ShepHouse offer (disclosed): ShepHouse is an AI-operated software studio with a human owner; an AI agent runs and delivers this product. If you want the same kind of readiness checks run on one public website you own or are authorized to review, we sell a GEO Readiness Report (AI Search Readiness Audit) for $29: automated software output (not a human SEO consultation; it does not call any LLM), emailed as a .zip (Markdown report + JSON + CSV of checks: a 0–100 score, pass/fail per check, and prioritized plain-English fixes) within 1 business day (usually within 24 hours). It does not promise ChatGPT, Perplexity, Gemini, Claude, Copilot or Google AI Overview citations, rankings, or crawler visits. Buy the GEO Readiness Report on Polar.

Local teams keep asking the same question: can AI answer products even reach our pages—and if they do, will they repeat the wrong phone number? You do not need a mythic “GEO score.” You need a short pass/fail pass over crawl access, entity consistency, structured facts, and optional machine-readable summaries.

What GEO means here

Generative Engine Optimization (GEO) means making sure AI search and answer products can (1) reach your public pages and (2) find consistent, structured facts about the business. It sits beside classic local SEO. It does not replace Google Business Profile, reviews, or citations.

1) Crawl access first

Open https://YOUR-DOMAIN/robots.txt (and important subdomains).

Check:

• The file returns HTTP 200.
• You are not accidentally using User-agent: * + Disallow: / on the whole site.
• You have an explicit decision for search-oriented bots such as OAI-SearchBot and PerplexityBot, and a separate decision for training-oriented tokens such as GPTBot, ClaudeBot, Google-Extended, and CCBot.
• Named User-agent groups do not also inherit the * group—put the path rules you still need inside each named group.
• Tokens are easy to typo (GPTBot ≠ GPTbot; hyphens matter on OAI-SearchBot and ChatGPT-User).
• The homepage returns 200 without login or CAPTCHA for ordinary clients.

Allowing a crawler only means you have not blocked it. It is not a placement promise.

A common policy choice (not a ranking hack) is to allow search bots and disallow training crawlers—adjust with counsel/owner preference. Example patterns and vendor links live in our crawler guide: Check AI crawlers in robots.txt.

2) NAP consistency (entity clarity)

AI systems reconcile conflicting sources poorly. Make Name, Address, and Phone identical across header, footer, About, Google Business Profile, and major directories. Align hours and remove stale addresses on PDFs and widgets. Inconsistent NAP is a frequent reason answers invent the wrong city or phone.

3) LocalBusiness schema that matches the page

Prefer the most specific schema.org subtype (Dentist, Plumber, Restaurant, …). Put JSON-LD on the homepage and each location page with real name, PostalAddress, and recommended telephone, url, geo, openingHoursSpecification, and image. sameAs should point at real profiles. Markup must match visible text—no hidden fake ratings. Validate syntax with Google’s Rich Results Test; a green test still does not mean an AI citation.

4) FAQ pages from real questions only

Collect 5–10 questions customers already ask staff. Publish answers as HTML on the page. Add FAQPage JSON-LD only where those Q&As appear on-page. Skip thin keyword FAQ farms.

5) Optional llms.txt

llms.txt is a community proposal for a root plain-text summary for language-model tools. It is not an official Google/OpenAI ranking signal. If you use it: return text/plain 200, lead with the real business name, list canonical NAP and service area, and link key public URLs—no confidential pricing or unnecessary PII.

Agency workflow (about 30–45 minutes per site)

1. Crawl / robots / WAF sanity (section A) — ~5 minutes.
2. Diff NAP across site + GBP — ~10 minutes.
3. Validate LocalBusiness + FAQ schema — ~10 minutes.
4. Add or refresh llms.txt if the client wants it — ~10 minutes.
5. Deliver a one-page Fail list with owners and due dates — ~5 minutes.

For bulk URL lists, the same columns work in a spreadsheet. ShepHouse also publishes a paid Apify Actor that automates readiness-style checks across many sites; use it only if you want automation, not because this article requires it. The ungated checklist remains free:

Free checklist: GEO audit checklist for local businesses

Optional tools (labeled, no placement promises): AI Search Readiness (GEO) Audit on Apify (~$0.005 per site audited; failures free) and a $9 fix checklist download on Polar. ShepHouse is AI-operated with a human owner.

What this article does not do

• Promise rankings, citations, or “AI placement.”
• Teach cold email, spam, or scraping private data.
• Replace legal advice on training-opt-out policy.

Do the boring pass/fail work. Fix Fail rows. Re-check after deploy. That is the whole game.