Most businesses shopping for AI visibility are about to buy the wrong fix. They sign for a monthly tool or a 60-page audit before checking whether ChatGPT, Claude and Perplexity can fetch their key pages, let alone quote them.

So test that first. Pick the pages, write down what you expect, then check what the engines can receive: your robots rules, your raw HTML without JavaScript, your verified bot visits and the words the engines quote back. Keep a control group you leave alone. I used to run ad copy tests this way because, without a control, you will credit the last change you made for whatever happened next.

Question: is the page blocked, unreadable or simply not quoted?

This protocol separates three outcomes that call for different spending:

  • Blocked: a crawler cannot get the page.
  • Readable but not quoted: the page is accessible, but the answer does not appear in the live test.
  • Readable and quoted: this test has not found a technical access problem.

Do not roll those into one visibility score. A robots rule, a JavaScript-dependent price and a page that buries its answer are different problems. Test which one you have before you buy a fix.

Setup: choose 10 money pages and 5 pages you will leave alone

Pick 10 money pages that would cost you something if AI answers skipped them: pricing, core services, top product categories and the two or three blog posts that actually drive pipeline. Then pick 5 stable controls you will not edit during this test. An About page, an old post or a careers page will do. The controls will not be perfect matches for your money pages; they give you a check on whether quotes are shifting even where you changed nothing.

Before opening a tool or an AI chat, make a sheet with a row for each money page and control. For each page, write your prediction on three lines:

  1. Can the relevant crawler fetch it?
  2. Can it read the key facts without JavaScript?
  3. Will a live session quote those facts today?

Write yes or no. Then leave room for the observed result, the exact failing URL, the response you received and the date. If you check results first and decide what you expected afterward, you will find exactly what the tool vendor wanted you to find.

My working expectation for a small business site is that 2 to 3 of the 10 money pages will have an access problem, 4 to 5 will be readable but say little worth quoting, and 1 or 2 will be fine. That is a prediction, not a finding about your site. Config creates blocks; thin or buried answers create a different kind of silence. Prove me wrong on your pages.

Test 1: read robots.txt token by token

Open yourdomain.com/robots.txt. Do not stop when you find one line about AI. Check these user agents individually: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Check Google-Extended separately as a training-related control. A broad User-agent: * rule can matter too; crawler references still flag blanket disallows in the wild.

The distinction between agents matters. OpenAI separates GPTBot from its search and user-fetch crawlers. If your policy is to restrict training, do not assume the same decision should apply to OAI-SearchBot and ChatGPT-User. Those are the access paths this citation test needs to examine. What the deck calls “blocking AI scraping,” I call a setting you had better read before flipping.

Record the rule that applies to each agent and page path. For this test, access for the search and user-fetch agents is a pass; a disallow that covers a money page is a fail. A clean robots file is not the end of Test 1, though. Cloudflare can block AI crawlers independently of robots.txt. If a later request gets a 403, check the layer in front of the site before blaming the page.

Practical takeaway: write down the exact agent, path and rule. “Robots looks fine” is not a test result.

Test 2: inspect the HTML the first fetch returns

Robots.txt can allow a request and still deliver a page with no useful answer in it. A pricing page built around JavaScript can look complete in your browser while its initial HTML contains little more than an app shell. Googlebot has rendered with evergreen Chromium since 2019; do not take a Google ranking as proof that every other fetcher receives your rendered prices and specs.

Run three checks on each money page:

  1. View Source. Search the initial HTML for the price, spec or core claim a visitor needs. Use the page’s actual answer, not a generic keyword.
  2. Disable JavaScript and reload. Note whether the headings and key facts remain visible. If they vanish, record a readability fail.
  3. Fetch the response body. Use curl with a normal browser user agent and again with a crawler user agent. Compare the status and the full body, not just the status code. A 200 does not tell you whether the answer arrived.

Keep the results separate. A browser-user-agent fetch that succeeds while a crawler-user-agent fetch fails points you back toward access controls. Two successful fetches with an empty answer point toward rendering. The key fact needs to be in the initial response for this test to pass. If it exists only after scripts run, ask a developer about server-side rendering or static generation before paying a subscription to watch the problem persist.

Test 3: verify visits instead of trusting bot costumes

Pull 30 days of server or CDN logs, if you have them, and filter for the agents from Test 1. A user-agent string is a claim, not an identity: anyone can call a request GPTBot. Compare the IPs of claimed visits with the providers’ published ranges. OpenAI publishes lists at openai.com/gptbot.json, searchbot.json and chatgpt-user.json; Anthropic lists crawlers at claude.com/crawling/bots.json; Perplexity publishes a perplexitybot.json file and a Perplexity-User file. Use reverse DNS where it helps you check a claimed identity, rather than treating the user-agent field as verification by itself.

One 30-day log study found that 6 to 22% of claimed AI-bot hits failed verification. You need not assume the same share on your site. The point is simpler: if a claimed hit is not genuine, counting it as evidence of crawler access can send you after the wrong fix.

Perplexity deserves a separate note. Perplexity describes Perplexity-User as an agent that generally ignores robots.txt, while reports of undeclared crawler behavior make its footprint harder to infer from a robots rule alone. Check WAF and server behavior as well as the file. For an example of the kind of request to look for, this AI SEO audit teardown shows Claude-User robots.txt fetches in server logs.

Record verified visits, rejected requests and any 403s by page and agent. No verified hit in your 30-day window is a reason to investigate, not proof that access is blocked. A live fetch in Test 4 can answer the immediate access question; logs can tell you whether the problem has shown up in ordinary visits.

Test 4: request a live quote, not a brand summary

Choose a money page that passed the access and raw-HTML checks. In fresh ChatGPT, Claude and Perplexity sessions with browsing available, paste its exact URL and use the same prompt each time:

Read this URL and quote its price, core spec and main claim word for word. If one is absent or you cannot read the URL, say so.

Replace “price” or “spec” with the relevant fact if the page does not contain one. Save the exact quote, omission or refusal, along with the session date. You are testing whether the live session returns words from that URL, not whether a model has heard of your brand. A plausible paraphrase without the requested words does not pass the quote test.

Expect the access result and quote result to diverge. Access makes a page eligible to be read; it does not make its answer compelling or easy to extract. Self-contained answers, specific numbers and dates, initial-HTML JSON-LD and third-party corroboration are among the signals discussed for quotable pages. I used to tell clients a clear page was enough. I was wrong. A page can fetch cleanly and still bury the number three scrolls down while another page states it in sentence one.

Note which engine quoted the page and which did not. A fetch pass followed by a quote fail sends you toward the answer on the page, not back to robots.txt.

Test 5: edit five readable pages, then retest on day 14

Do not run a content experiment on pages you have already found to be blocked or empty in the initial HTML. Fix access or rendering first, confirm the pages pass and take a new baseline. Otherwise, a later quote will not tell you which change mattered.

For pages that pass, split the 10 money pages into two groups of five. On Group A, make one defined edit: put a self-contained answer block in the first 100 words of the initial HTML. State the relevant price, number or spec plainly and include matching JSON-LD in that response. Leave Group B and all 5 controls alone. Keep a record of the exact edit and its date.

Re-run Tests 1 through 4 on day 14, using the same URLs, facts and live prompt. Compare the results in this order:

  1. Did access or raw HTML change anywhere you did not intend it to?
  2. Did untouched controls start or stop getting quoted?
  3. Did untouched Group B move?
  4. Did edited Group A change differently from both untouched groups?

That last comparison is the useful one. A before-and-after screenshot cannot distinguish your edit from background movement. This is the controlled addition to the fetch-check sequence, not a promise that every page will be quoted within 14 days. If nothing moves, mark the result inconclusive rather than selling yourself a win. Say you spend $20k a month on content and someone proposes a $1,500-a-month visibility tool using an uncontrolled screenshot. This sheet is the reply.

Read the result before choosing the bill

Blocked is an access fix, not a citation-software purchase. If a relevant robots rule disallows the path or your WAF returns a 403 to the fetcher, hand a developer the URL, agent, rule and failed request. Change the setting, then confirm the request succeeds. Do not pay a monthly fee to monitor a door that stays shut.

Readable but not quoted calls for a page edit before another audit. If the initial HTML contains the facts and a live session reads the URL but will not quote the answer, look at where and how the page states it. Try the Group A answer block; leave the comparison pages alone and measure again. If the edited pages move while the others do not, you have a reason to roll that edit wider. If they do not, do not pretend another robots check will solve it.

Readable and quoted means this test has not found a technical access problem. You may still care how often the page gets cited, but a technical audit should have to identify something more specific than “AI cannot see you.” Keep the working page as a reference when you assess the others.

Three test outcomes: blocked, readable but not quoted, and readable and quoted

What a tool must do that this afternoon cannot

A tool earns its fee if it keeps checking verified fetches after this afternoon, tracks live quotes against the pages you left alone and shows what changed on each page and why. It should not charge you to announce that robots disallows a fetcher or that JavaScript hides a price. You just checked both.

If a vendor leads with a 60-page audit and a monthly PDF, pass. I would rather pay for a system that carries out the retest and shows every action with the reason behind it. If you want to see the mechanics on a live page first, this 45-minute fetch check follows the same sequence. That is the appeal of groas on the organic side: continuous execution inside your guardrails, with a human accountable for direction, instead of another report for you to interpret.

Your next step comes from the sheet you wrote before testing. Blocked: send the developer the failed request, fix access and confirm it with a fresh fetch. Readable but not quoted: edit Group A, protect the comparison pages and see what day 14 changes. Readable and quoted: stop shopping for an access fix you cannot show you need. File the sheet either way. The point was never the score. It was knowing whether you needed a tool, a developer for one afternoon, or nothing, before anyone sent you an invoice.