How it happens
- A firewall or CDN challenge page answers every unknown client with a puzzle
or a 403. The crawler gets the puzzle, not the product.
- A rule blocks by user agent, and a legitimate crawler is caught along with
impostors using the same name.
- robots.txt allows a crawler that the firewall in front of it never lets
through — the file and the network disagree.
How to check
- Look in your server logs for requests from each AI provider you care about,
verified against the provider's documented rule (how: /learn/how-to-verify-ai-crawlers-in-your-logs).
- For each, look at the status code your server returned. A verified crawler
receiving 403 or a challenge page is blocked, whatever robots.txt says.
- Fetch a product page yourself without JavaScript and without cookies. If
you get a challenge, so did the crawler.
Allow the verified, not the named
The fix is not to allow everything that says GPTBot — on our site, 2,290 requests using OpenAI's crawler name came from outside OpenAI's published ranges (study). Allow requests that pass the provider's check, and keep blocking the rest. Our own crawler, and how to allow it, is described at /bot; the evidence rules are at /evidence.