Website crawl failed or is incomplete

A failed or partial crawl means your AI Twin is starting life with an incomplete knowledge graph. Walk this list to identify why the crawler didn't get everything, then fix and re-crawl.

1. robots.txt blocks the crawler

Fetch https://your-site.com/robots.txt and look for Disallow rules that apply to user-agent * or RealityTwinBot. Fix:

User-agent: RealityTwinBot
Allow: /

2. Bot protection (Cloudflare, WAF, DataDome)

Managed challenge or bot fight may block automated fetches. Fix:

• Allow-list our crawler IP range — email hello@realitytwin.io for the current range.
• Add RealityTwinBot to your bot-management verified-bot list.
• In Cloudflare, create a WAF rule: (http.user_agent contains "RealityTwinBot") allow.

3. JavaScript-only content

The crawler renders JavaScript with a 10-second budget per page. If content only appears after a long client-side fetch, it may be missed. Fix:

• Publish a static HTML export of critical pages.
• Add server-side rendering (SSR) or pre-rendering for indexable pages.
• Or upload the content as a document.

4. Orphan pages

The crawler follows internal links from the root. Pages not linked from anywhere will be missed. Fix:

• Link to them from a sitemap page or footer.
• Or submit a sitemap.xml under Knowledge → Sources → Add sitemap.

5. Rate-limited (429 responses)

The crawler respects Retry-After. If your site aggressively rate-limits, the crawl slows to a crawl. Fix:

• Raise the limit for the RealityTwinBot user-agent.
• Or allow-list the crawler IP range.

6. Very large sites without a sitemap

The crawler will stop at your plan's page cap. Fix:

• Submit a sitemap so we prioritise the most important pages.
• Upgrade to a plan with a higher cap.

Frequently asked questions

How do I retry a failed crawl?

Knowledge → Sources → your website → Re-crawl. Fix the root cause first.

Where do I see which pages failed?

Knowledge → Sources → your website → Failed pages. Each row shows the HTTP status and reason.

Can I contact support to diagnose?

Yes — hello@realitytwin.io with your site URL. We can inspect logs on our side.