Our crawler
ProductKnowledgeGraph reads product pages that customers ask us to read, and re-reads them to check that the facts we publish are still there. We do not train models on what we fetch and we do not resell page contents.
- User-Agent
ProductKnowledgeGraphBot/1.0 (+https://productknowledgegraph.com/bot)- Respects robots.txt
- Yes. A disallowed path is not fetched, and a source that disallows us is reported to its owner as unreadable rather than read anyway.
- Request rate
- One page at a time, with backoff on 429 and 503. We read a handful of pages per customer per day, not a catalogue sweep.
- What we fetch
- robots.txt, sitemaps, the public Shopify catalogue endpoint, and product pages named by the customer who owns them.
- What we store
- The structured data on the page, the SHA-256 of the body we read, and the time we read it. Not the page itself.
If you are blocking us
Most refusals we see are bot protection rather than a decision about AI: the same sites allow GPTBot and ClaudeBot in robots.txt and refuse us at the CDN, because our identity could not be checked. Adding the user-agent above to the same allowlist resolves it.
If the refusal is deliberate, nothing further is needed — we will report the source as unreadable to whoever asked us to watch it, and publish no facts we cannot re-read.
Questions
Write to [email protected]. A request to stop reading a site is honoured without argument.