All case studies
Case studyOriginal ConceptDemo Ready

Tideline: Knowledge-Base Support Agent

A support agent that answers from a 24-article knowledge base using BM25 retrieval, quotes the source article verbatim, and hands off to a human when its confidence is below the floor. It is not a language model and does not claim to be.

This is a working demonstration, not a client engagement. The brand and data are fictional.

Tideline support agent with suggested questions and a knowledge base panel

Background

Tideline is an invented outdoor-equipment retailer. The agent exists to show retrieval and escalation behaviour, which is where most support automation actually succeeds or fails.

The challenge

The failure mode that matters in support automation is a confident wrong answer. A system that answers everything will answer badly; a system that answers nothing is useless. The real work is the boundary: knowing when retrieval has not found the answer.

Objectives

  • Answer in-scope support questions from a knowledge base without inventing policy.
  • Know when retrieval has not found the answer, and escalate instead of guessing.
  • Make the retrieval inspectable, so a wrong answer can be diagnosed rather than re-prompted.

My role

Sole designer and developer

  • Implemented Okapi BM25 ranking from the formula, with title boosting and a tuned confidence floor
  • Built bidirectional synonym expansion so "refund" and "money back" reach the same articles
  • Wrote the 24-article knowledge base and the conversation layer around it
  • Made every sourced reply quote its article verbatim, with a test asserting the agent never paraphrases a source into something the article does not say
  • Built the escalation path for queries that fall below the confidence floor, rather than letting the agent guess
  • Wrote 43 unit tests and 17 browser checks

Technical approach

Lexical retrieval rather than a language model, deliberately: BM25 scores are inspectable, the confidence floor is a number you can tune, and the agent can only ever say what an article says, because replies are quoted from the source. Where the score is below the floor, it escalates instead of improvising.

What is implemented

  • BM25 retrieval over a 24-article knowledge base with title boosting
  • Bidirectional synonym expansion, so 'refund' and 'money back' reach the same articles
  • Replies quoted verbatim from the source article, with the article shown alongside
  • A confidence floor that routes low-scoring queries to a human hand-off
  • Suggested starter questions and a visible knowledge-base panel

Process

  1. Implemented BM25 with K1 1.4 and B 0.72, plus a title boost so an article titled for the question outranks one that merely mentions it.
  2. Built synonym expansion as a bidirectional map, so every word in a synonym group reaches the others rather than only mapping one way.
  3. Set a confidence floor and tested the boundary in both directions: known questions answer, out-of-scope questions escalate.
  4. Wrote a test asserting every sourced reply is verbatim article text, so the agent cannot drift into inventing support policy.

Engineering and creative decisions

  • Lexical retrieval, not a language model

    BM25 scores are numbers you can inspect and a floor you can tune. The interface says so plainly, because a visitor should know what they are looking at.

  • The agent can only say what an article says

    Sourced replies are verbatim article text. A test asserts this, which removes the failure mode that matters most in support automation: a confident answer that invents policy.

  • Synonyms expand in both directions

    The first implementation mapped one way only, so a query using the less common word found nothing. Each group now maps every word to every other.

  • Escalation is a feature, not a fallback

    Below the confidence floor the agent hands off rather than returning its best weak match. A system that answers everything answers badly.

What was delivered

An agent that answers in-scope questions with sourced, quoted text, shows which article it used, and escalates rather than guessing when nothing scores highly enough.

Results

  • 43 unit tests and 17 browser checks pass, including the verbatim-source assertion and the escalation boundary.
  • The interface states plainly that this is lexical retrieval, not a language model, so no one mistakes what they are looking at.
  • A test expectation of mine was wrong about the stemmer's output; the stemmer was right and the test was corrected.

These are tested capabilities of the demonstration, not commercial outcomes.

Technology stack

  • TypeScript
  • React 19
  • esbuild
  • Node.js test runner
  • Playwright

Get in touch

Have a similar project in mind?

Whether you're launching a brand, developing a digital product, or automating business workflows, let's discuss how to bring your idea to life.