Force a manual re-crawl of your website

The scheduled crawl runs weekly, which is fine for most sites. When you publish a new landing page or update a pricing figure, trigger a manual re-crawl so your AI Twin picks up the change within minutes.

From the dashboard

1. Open Knowledge → Sources.
2. Find your website entry and click Re-crawl.
3. The status changes to Crawling. Typical time: 30–90 seconds for small sites.
4. When status returns to Indexed, the new content is retrievable.

From the REST API

Automate re-crawls from your deploy pipeline or CMS. Get your API key under Settings → API.

curl -X POST https://realitytwin.io/api/v1/sources/{sourceId}/recrawl \
  -H "Authorization: Bearer $RT_API_KEY"

Response includes a jobId you can poll for status via GET /api/v1/jobs/{jobId}.

From a CMS webhook

Point your CMS's publish webhook at:

POST https://realitytwin.io/api/v1/sources/{sourceId}/recrawl
Header: X-RT-Webhook-Secret: your-shared-secret

Configure the shared secret under Settings → API → Webhook secrets. WordPress, Contentful, Sanity, Prismic and Strapi all support outgoing webhooks natively.

Incremental vs. full re-crawls

• Incremental (default) — refetches only pages whose Last-Modified header or sitemap lastmod has changed. Fast and cheap.
• Full — refetches every page. Use when you've restructured URLs or changed sitewide copy.

The dashboard button runs an incremental crawl by default; hold Shift when clicking to run a full crawl.

How long do changes take to reach conversations?

As soon as the crawl finishes, new chunks are available for retrieval — usually within 30–90 seconds after the button click for a small site. There is no additional embedding lag.

Best practices

• Automate re-crawls on publish to eliminate stale answers permanently.
• For pricing pages, re-crawl the moment the change goes live.
• For blog content, weekly is enough.

Frequently asked questions

Is there a limit on manual re-crawls?

Unlimited on all paid plans. The free plan is limited to one manual re-crawl per day.

Will a re-crawl duplicate my content?

No. Existing chunks are updated in place; deleted pages are removed.

Can I re-crawl just a single page?

Yes via the API: POST /api/v1/sources/{sourceId}/pages with a JSON body { urls: ["https://..."] }.