What to Actually Measure When You Add an AI Agent to Your Site

It's tempting to judge an AI agent by how many conversations it's having. That number goes up over time almost by default as more visitors discover it — it doesn't tell you whether those conversations are actually going well.

The metric that actually matters: unanswered questions

Every question your agent couldn't answer is a direct, specific signal — it tells you exactly what content is missing, not a vague sense that "the bot could be better." Reviewing that list regularly and closing the gaps is the single highest-leverage thing you can do after initial setup.

Leads with context beat lead count

Ten leads with real context attached (what they asked, what they were interested in) are worth more to a sales team than fifty leads that are just a name and an email. If you're tracking lead capture, look at what's attached to each one, not just the total.

Watch for the same question repeating

If the same unanswered question shows up over and over, that's not noise — that's a real content gap costing you conversions or tickets every single day it stays open. A transparency view into what your agent has been trained on (and what it's been asked that it couldn't answer) turns this from guesswork into a checklist.