Personal project · not affiliated with Supabase

Does a docs tool make the agent better, and in what shape?

A small MCP server with one tool, search_docs, over Supabase's public guides, evaluated against a no-tool baseline and in two exposure formats, on 30 questions with exact answers.

Built by Ulf Fäldt Reifenberg with coding agents, as part of an application to Supabase's AI Tooling team. Uses only public documentation (supabase/supabase on GitHub, commit ). Not affiliated with or endorsed by Supabase, and says nothing about how Supabase's own MCP server or Assistant work internally.

Setup

Corpus: all 592 guide pages (apps/docs/content/guides), split into 8,401 heading-level sections.
Tool: TypeScript MCP server (stdio, JSON-RPC: initialize → tools/list → tools/call), BM25 search. Exposure format is a server option: snippets = top-3 matching sections (~1k chars each), page = the whole best-matching page (up to 6k chars).
Arms: no_tool (model knowledge only) · tool_snippets · tool_page. Same prompt rule in every arm; up to two tool calls; prompted JSON tool calls so any chat model works.
Questions: 30 hand-written questions with a single verifiable answer (limits, flags, headers, SQL function names, config keys) from 13 pages. Every answer key was checked to appear in its source page.
Grading: deterministic regex on the final answer, no LLM judge. Paired bootstrap 95% CIs over questions (10,000 resamples).
Models: Llama 3.2 3B, Llama 3.1 8B and Llama 3.3 70B on Cloudflare Workers AI, temperature 0.

Accuracy

Differences (paired, 95% CI)

Cost and failure modes

Input tokens and latency are per question, summed over all model calls. A retrieval miss means the answer key never appeared in any tool result; a reading miss means it did and the model still answered wrong.

Findings

All answers

What this does not show

30 questions is small; the CIs are wide for close arms. Questions are factoid look-ups, not multi-step build tasks, which is where a docs tool matters most and where grading needs more than a regex.
Retrieval is plain BM25. Supabase's real search_docs may use a different index; this measures the shape of the tool output, not their implementation.
Small open models with prompted tool calls; frontier models with native tool calling will have a higher no-tool baseline and fewer reading misses.
Questions were written by someone who had read the pages, so phrasing can favour lexical retrieval.