Connect your website: how the AI Twin crawler works

The website connection is the first and most important knowledge source for any Reality Twin AI Twin. This guide explains exactly how the crawler works, which pages it can and cannot see, and how to control what ends up in your twin's knowledge graph — so your visitors get accurate answers grounded in the pages you approve.

How the crawler works

Reality Twin's crawler behaves like a well-mannered search-engine bot. It identifies itself as RealityTwinBot, respects robots.txt, honours rate limits, and follows internal links from the root URL you provide. Every crawled page is parsed into structured chunks and embedded into your workspace's knowledge graph.

The crawl lifecycle

1. You submit a root URL from onboarding or Knowledge → Sources.
2. The crawler fetches robots.txt and builds an allow-list for your host.
3. It requests the root page, extracts links, and enqueues them.
4. Each page is rendered (with JavaScript, 10-second budget), parsed and chunked.
5. Chunks are embedded and stored in your workspace — private to you.

What the crawler fetches

• HTML pages linked from your root, up to your plan's page cap.
• Meta title, meta description, canonical URL and language.
• Headings (h1–h4), body copy, lists, tables, and visible link anchor text.
• Structured data (JSON-LD) — used as extra signal, not as authoritative content.

What the crawler intentionally skips

• URLs disallowed by robots.txt for user-agent RealityTwinBot or *.
• Login-gated content — the crawler does not authenticate.
• Admin, wp-admin, checkout, and similar sensitive paths.
• Binary assets (images, video, audio). Upload documents separately.
• Duplicate URLs with tracking parameters (utm_*, gclid, fbclid).

Control what gets indexed

Option 1: robots.txt

Add a targeted rule for our crawler user-agent. This is the recommended approach for permanent exclusions.

User-agent: RealityTwinBot
Disallow: /internal/
Disallow: /drafts/
Allow: /

Option 2: dashboard toggles

Open Knowledge → Sources → your website. Every indexed page is listed with a visibility toggle. Turn a page off and it is removed from retrieval within seconds — no re-crawl required.

Option 3: meta noindex

The crawler honours <meta name="robots" content="noindex"> — useful for individual pages you want out of both search and your twin.

Submit a sitemap for large sites

For sites over 200 pages, submit a sitemap.xml URL under Knowledge → Sources → Add sitemap. The crawler uses it as an authoritative list of URLs, guaranteeing no page is missed because it is only reachable through search or JavaScript navigation.

Refresh cadence and manual re-crawls

By default the crawler re-runs weekly. You can trigger a manual re-crawl any time — see Force a re-crawl of your website. Enterprise plans support daily and event-driven re-crawls fired by CMS webhooks.

Best practices

• Use a single canonical domain (either www or apex) to avoid duplicate indexing.
• Keep your pricing on one page — not spread across several — so citations are unambiguous.
• Fix broken internal links; unreachable pages never enter the graph.
• Publish product specs as HTML pages where possible; PDF is fine but text HTML is easier to update.

Common mistakes

• Blocking the crawler in Cloudflare or a WAF then wondering why the twin knows nothing.
• Pointing at a staging URL — the twin will happily learn from unfinished copy.
• Leaving Disallow: / from a legacy robots.txt — nothing gets indexed.

Frequently asked questions

Does the crawler execute JavaScript?

Yes, with a 10-second budget per page. Content that only appears after a long client-side fetch may be missed — publish it as static HTML or upload a document.

Can I connect multiple websites to one AI Twin?

Yes on Starter and above. Add each root URL as a separate source; they merge into one knowledge graph per twin.

How often is my website re-crawled?

Weekly by default. Manual re-crawls are unlimited on all paid plans, and Enterprise plans support webhook-triggered incremental refreshes.

Is the crawl private to my workspace?

Yes. Crawled content is stored per workspace and never used to train shared models. Read Where is my data stored? for full details.