Build the agent

Knowledge base

How crawling, embedding, and source citations work — and how to keep your bot grounded in fresh, accurate content.

How indexing works

When you connect a URL, we crawl pages discoverable from sitemap.xml and internal links, respecting robots.txt. Each page is chunked into ~500-token passages and embedded with the configured embedding model. The embedded passages live in a vector database, scoped per bot.

Managing sources

Everything lives in Bot → Knowledge sources. The Website crawl card at the top shows the connected site and crawl status, with two actions:

• Re-crawl — replace the current index with a fresh crawl. The bot stays usable throughout.
• Crawl a URL — add a single page (a new doc, a policy page on another domain) without re-crawling the whole site.

Re-crawls count against the workspace's training pages quota — see Billing & quotas.

Uploading files

Drag files onto the drop zone (or Choose files) on the same page. We accept PDF, DOCX, TXT, and Markdown up to 20 MB each. Files are parsed in your browser before import — the raw file never leaves your machine, only the extracted text does. Parsed content is chunked and embedded exactly like crawled pages.

Text snippets

For content that doesn't live anywhere yet — an internal FAQ, a policy you haven't published — hit Add text, give it a title, and paste the content. Markdown is fine. Snippets join the index immediately.

Citations

Every assistant reply records which passages it pulled from. The widget shows source links inline. From the dashboard, opening a conversation transcript shows citation chips under each turn — click one to jump straight to the source URL.

What the agent does when sources are missing

The system prompt instructs the agent to say “I don't know” and offer to escalate rather than guess. If your traffic starts producing “I don't know” replies, that question shows up in Insights as a coverage gap — usually a sign your content needs a doc page.

Trained Q&A pairs

Crawled pages cover the common case, but some answers need to be exactly right — refund language, pricing edge cases, legal-adjacent replies. Open any chat transcript and use Improve answerto save a canonical Q&A pair. It joins this same vector index and the agent prefers it over crawled passages on the next match. See Improve answers.

Adding context the agent can't infer

Some context isn't indexable from a public site — internal policies, escalation rules, your tone of voice. Set these in Bot → Settings → Chat behavior:

• Business summary— 1–2 sentences anchoring the agent's mental model.
• Persona — who the agent is and how it talks.
• Escalation rules— owner-defined hand-off triggers (e.g. “always escalate refunds over $100”).
• Forbidden topics — topics the agent must refuse and offer to connect a human.