Crawl your website to train your AI agent
Add your website to an intoCHAT agent: find pages, pick them in a folder tree, exclude paths, crawl settings (country, slow mode, ?query pages), sites that block crawlers, crawl limits and keeping pages up to date.
The quickest way to give your agent knowledge is to let it read your website. You enter an address, intoCHAT lists the pages it finds, and you choose which ones to add.
Creating a new agent? Start from your website in the create-agent dialog reads your homepage and up to 4 key pages and trains the agent for you. See Quick start.
Find and add pages
- Open the agent's Knowledge tab and choose Website under Add knowledge.
- Enter your site in Website URL, for example
https://example.com. - Optional: in Exclude URLs containing, list parts of addresses to skip, separated by commas, for example
/blog, /careers, ?lang=. - Click Find pages.
- Review the list, which groups the pages by folder. Use Filter pages…, Select all or None, tick a folder to select all its pages, or tick pages one by one.
- Click Add … selected pages and wait for the progress bar to finish.
- Click Train agent at the top of the tab. Training continues on the server if you leave the page.
Added pages are saved right away as Pending. The agent only uses them after training.
How pages are found
Find pages combines two methods:
- Sitemap: intoCHAT looks for a sitemap at common locations such as
/sitemap.xmland/wp-sitemap.xml, and in theSitemap:line ofrobots.txt. Entries for login, sign-up and admin pages are skipped. - Link map: a crawler collects the links on the same domain.
Each method returns up to 500 addresses. Duplicates such as /about and /about/, images, feeds and other files that aren't pages are removed. Links to PDF files stay in the list and are read like pages. Tracking parameters such as utm_source, gclid and fbclid are removed from addresses, because they never change what a page says. How other ?query addresses are handled depends on crawl settings.
On a multilingual site with language folders such as /de/ and /fr/, a Languages row appears above the list. Pages in the language of the address you entered start selected, plus the home page; click a language to select or deselect all of its pages. Pages already in your knowledge show already added and start unselected.
Choose pages in the folder tree
The pages found are shown as folders, based on their addresses, such as /blog and /blog/2026. A folder's own page, such as /blog, is listed inside the folder. When the pages come from more than one host, for example a subdomain, each site gets its own folder.
- Each folder shows how many pages it holds, or for example "3 of 12" when some of them are selected.
- A folder's checkbox selects all of its pages. On a folder that is fully selected, it clears them. When only some pages are selected, the checkbox shows a partly selected state.
- Folders with more than 15 entries start closed. Open all folders and Close all open or close every folder at once.
- While you type in Filter pages…, every folder opens so that each matching page shows.
- Pages show already added, added, or failed with the reason when you hover over it. Pages that are added or being read can't be selected.
Below the list, a line counts your selection, for example "12 of 340 pages selected. One crawl from your plan reads up to 600 pages · 8 crawls left this month." The crawls left appear only when your plan has a crawl limit. With more than 600 pages selected, you see "More than 600 pages are selected, so adding them may use a second crawl."
A scan stops at about 500 pages. When the list reaches that size, intoCHAT says that pages may be missing. Add those with Single page mode.
What is read from each page
intoCHAT reads the main content of each page. Chat widgets and cookie banners are removed, and menus or banners that repeat on every page are kept only once.
Some sites build their content with JavaScript after the page loads. When several pages come back as the same empty shell, intoCHAT reads them again with a few seconds to load, and does the same for the rest of the run, so such sites take longer. If the crawler is busy, the run pauses with a countdown and then continues by itself.
Pages that time out are tried once more at the end. A page that still fails is marked failed; hover over it to see why, then select it and add it again.
Crawl settings
Below the address and its options, open Crawl settings to change how this agent's website is read. The closed panel sums them up, for example "Automatic country · normal pace · ?query pages ignored". Each change is saved at once and applies to everything that reads the agent's website: Find pages, adding pages, Refresh from website and auto-resync. Viewers can see the settings but not change them.
Scraping country
Choose Automatic or a country: United States, United Kingdom, Germany, France, Italy, Spain, Switzerland, Austria or the Netherlands. Pages are then read as a visitor from that country would see them, which helps on sites that change prices, language or content by location. With Automatic, intoCHAT reads pages itself, and our crawling provider chooses for the pages it reads.
With a country chosen, every page is read by our crawling provider from that country: adding pages, refreshing them and auto-resync. Find pages reads the sitemap and the homepage's links directly from intoCHAT, so the country doesn't apply to them.
Slow mode
Turn on Slow mode for sites that block or time out when many pages are read at once. intoCHAT then reads 5 pages per request instead of 10, 2 of them at a time, and pauses 4 seconds between requests. Crawls take longer, and one crawl from your plan covers about 300 pages instead of 600. Auto-resync reads fewer pages per run and checks the rest on the next run.
Pages with a ?query
Some sites tell pages apart only by what follows the ? in the address, for example /product?id=12 and /product?id=13.
- With Treat ?query pages as separate pages off (the default), the link map leaves out addresses with a
?query. Addresses your sitemap lists with a query, such as?page_id=12, stay in the list as written. - With it on, the link map lists
?queryaddresses too. Tracking parameters are removed and the remaining parameters are put in a fixed order, so?size=m&color=redand?color=red&size=mare one page. At most 100?querypages are listed, and at most 20 for the same path, so a shop's filter combinations can't flood the list. When some were left out, the list says how many; add those with Single page mode.
The setting decides which pages are found. Pages you already added keep their address and are read again as they are.
Single page mode
Turn on Single page mode to add one address without scanning the site. The button changes to Add page. Use it for a page that discovery missed. To update a page you already added, use Refresh from website on its source instead.
Crawl limits
| Plan | Website crawls per month | Characters per agent |
|---|---|---|
| Free | 1 | 100K |
| Starter | 9 | 500K |
| Pro | 20 | 1M |
| Enterprise | Custom | Custom |
- Scanning with Find pages doesn't use a crawl.
- Adding pages uses one crawl, whether a selection or a single page. Pages are read in batches of 10, and one crawl covers 60 batches, so about 600 pages. On sites that need time to load their JavaScript, and in slow mode, batches hold 5 pages, so one crawl covers about 300. A larger run uses another crawl. Refresh from website on a source counts like adding a single page.
- Refreshing a group of pages from the source list uses a crawl in the same way.
- Auto-resync uses one crawl for each run that reads website pages, for up to 30 pages.
- Start from your website, when you create an agent, uses one crawl.
- Crawls are counted per account, across all agents, and reset every month.
- Crawled text counts toward the agent's character limit. If a run reaches it, the remaining pages are marked failed.
When none are left, intoCHAT shows "Monthly crawl limit reached". You can still add files, text and Q&A.
Sites that block crawlers
Some sites turn away automated visitors with bot protection or a firewall. You then see "No pages found", or pages fail with one of these reasons:
- "The site refused the crawler (401)." or "(403)."
- "The site turned the crawler away for sending too many requests (429)."
- "The site showed a bot check (such as Cloudflare's) instead of the page."
- "No readable text on this page." or "No content returned for this page." An empty page can also simply have no text.
When pages fail this way, a note "The site may be blocking our crawler" appears below Crawl settings after the run. A refresh that fails this way says so and offers How to fix, and a source whose last auto-resync check failed this way shows "The site may be blocking our crawler: how to fix" under it.
Let the crawler in
- Pages are read by intoCHAT itself, with the user agent
intoCHAT-Crawler/1.0. Pages that need a browser to show their text, pages that refuse intoCHAT or show it a bot check, and every page when a scraping country is chosen are read by our crawling provider. Its user agent and IP addresses aren't controlled or published by intoCHAT, so there is no IP list to allow. If your firewall can allow a user agent, allowintoCHAT-Crawler/1.0; otherwise ask whoever runs the site to let these pages through the bot check, for example with a firewall rule that skips the challenge for them, then add the pages again. If you only pause the bot check while you crawl, later refreshes and auto-resync checks fail the same way. - The sitemap is fetched by intoCHAT itself, with the user agent
intoCHAT-Sitemap-Fetcher/1.0. Allowing this one too lets Find pages read your sitemap. - Too many requests (429): turn on slow mode and try again.
Add the content another way
- Paste the page's text in the Text tab. See Text and Q&A.
- Save the page as a PDF from your browser and upload it in the Files tab. See Files.
- For a page discovery missed, try Single page mode.
Only public http and https addresses can be crawled. Pages behind a login, addresses that contain a user name and password, and private or internal network addresses are refused.
Keeping pages up to date
Re-index saved text only trains the text read before again. To pick up changes on your site:
- Auto-resync: turn it on in the Auto-resync card on the Knowledge tab. Daily or weekly, intoCHAT reads your pages again and retrains only those whose text changed. Text repeated across the site, such as menus and footers, doesn't count as a change. See Auto-resync.
- One page: click Refresh from website (the cloud icon) on its source. intoCHAT reads that page again and, if its text changed, saves and retrains it right away; otherwise you see "No changes found".
- A site or folder: in the source list, choose Group by site and folder and use the cloud icon on the group. See Managing sources.
- Many pages after a scan: add them again with Find pages, then train. Existing sources are updated rather than duplicated.
A page read within the last hour may come from a cached copy.