Anyone can wire up a help bot in an afternoon. The hard part is that the docs underneath it are usually wrong. So I built a second agent to run the app, prove what was true, and rewrite the help center to match.
DailyPB is my live pickleball platform. Its help center covers everything a host or player asks: how to run a recurring session, how refunds work, what happens to the waitlist when someone drops.
I wanted an AI rep on top of it. You ask in plain English, it finds the right article, answers only from that article, and shows its sources. Because it can only speak from the real docs, it can't invent an answer. If the docs don't cover you, it says so.
I had it working in days. It answered real questions and cited every source. On the surface, done. This is where most AI demos take a bow.
Before real users touched it, I wrote hard questions and scored every answer by hand. The agent never made anything up. But some answers were wrong anyway, because the docs it faithfully quoted were out of date.
I'd changed the real deadline months earlier and never fixed the doc. The agent did its job perfectly. The source lied. And you can't catch a wrong-but-confident answer by reading it, because it looks flawless.
Right there the project changed. The bot was fine. The real job was fixing every doc it read, and making sure each one matched what the app actually does.
The only reliable source of truth is the app itself. So instead of reading the docs to see if they were right, I built an agent that opens the product and confirms each answer firsthand.
Reads the question, finds the most relevant docs, answers only from them, with citations. The part everyone builds. On its own, only as trustworthy as the docs behind it.
Opens the live product on a safe test copy, clicks through each feature, and confirms what actually happens. When a doc says the refund window is 48 hours, this agent goes and checks.
This second agent is the real work. It walked 66 areas of the product and captured 286 screenshots of what actually happens on screen. Then I rebuilt the help center to match every one of them.
One section, one answer. Each article is split so a single searchable piece fully answers one question, instead of burying five answers in one long page.
Hidden question metadata. Every section carries the real questions a user would actually type, in their words ("how do I charge for my sessions," not "payment configuration"), so search matches question to question, which is where most retrieval quietly fails.
Cross-references that don't pollute search. Related links show on the page but get stripped before the content reaches the search index, so link text never muddies the results. The searchable index grew to 291 clean chunks.
I built an eval suite: a fixed set of questions with known-correct answers, scored automatically, so I could change something and watch the number move instead of guessing. Fast automatic checks for what a computer can verify, plus a second, stronger AI as a judge for what only reading catches, like whether an answer invented a fact or dodged the question.
Better content produced measurably better answers. Proven, not asserted. The suite now stands as a permanent guard: every future change to the docs or the agent gets re-scored before it ships, so a fix in one place can't quietly break an answer somewhere else.
A demo proves something works once. I built the loop that keeps it working: answer, check the answer against the app, fix what's wrong, re-score it, and keep learning from real users.
That last step is live. Every answer has a thumbs up or down, and a thumbs down opens a box for the right answer. Each correction lands in a review queue tied to the exact question, so the people using it become the ongoing test set that keeps it honest.
Plain Python, no RAG framework, on purpose, so I understood every stage instead of hiding it behind a library. Docs are read from the live source, embedded with OpenAI, retrieved by cosine similarity, and answered by Claude, grounded strictly in the retrieved text.
Each chunk carries its article's title in the embedding, so a section like "FAQ" still knows its topic.
The hidden question blocks turn retrieval into question-to-question matching. I'd reinvented a named technique by hand before I knew its name; my edge is that I'm the domain expert, so the questions are real user language, not model guesses.
A stronger model grades the production model's answers in a separate call, against a rubric, returning structured output. Validated against a deliberately hallucinated input before I trusted it. A judge you've never watched fail is worthless.
At this size a flat file is the right call. pgvector and Pinecone solve a scale problem I don't have. Knowing when not to reach for the heavy tool is the skill.
The hard part is making it reliable enough to put in front of real customers, and that's the part I build. If your team is buried in the same questions over and over, let's talk about what this looks like for you.
Book a call →