{ "generated": "2026-11-11T10:02:58.965Z", "models": [ { "model": "llama-3.1-8b-instruct-fast", "n": 30, "arms": { "no_tool": { "correct": 3, "acc": 0.1, "avgInTok": 86, "avgOutTok": 24, "avgMs": 451, "toolUseRate": 0, "retrievalMiss": null, "readingMiss": null }, "tool_snippets": { "correct": 27, "acc": 0.9, "avgInTok": 1109, "avgOutTok": 51, "avgMs": 836, "toolUseRate": 1, "retrievalMiss": 3, "readingMiss": 0 }, "tool_page": { "correct": 23, "acc": 0.7666666666666667, "avgInTok": 1842, "avgOutTok": 52, "avgMs": 927, "toolUseRate": 1, "retrievalMiss": 7, "readingMiss": 0 } }, "diffs": { "tool_snippets - no_tool": { "diff": 0.8, "lo": 0.6333333333333333, "hi": 0.9333333333333333 }, "tool_page - no_tool": { "diff": 0.6666666666666666, "lo": 0.5, "hi": 0.8333333333333334 }, "tool_snippets - tool_page": { "diff": 0.13333333333333333, "lo": -0.03333333333333333, "hi": 0.3 } } }, { "model": "llama-3.2-3b-instruct", "n": 30, "arms": { "no_tool": { "correct": 2, "acc": 0.06666666666666667, "avgInTok": 86, "avgOutTok": 32, "avgMs": 313, "toolUseRate": 0, "retrievalMiss": null, "readingMiss": null }, "tool_snippets": { "correct": 24, "acc": 0.8, "avgInTok": 1041, "avgOutTok": 47, "avgMs": 640, "toolUseRate": 1, "retrievalMiss": 5, "readingMiss": 1 }, "tool_page": { "correct": 19, "acc": 0.6333333333333333, "avgInTok": 1495, "avgOutTok": 49, "avgMs": 619, "toolUseRate": 1, "retrievalMiss": 11, "readingMiss": 0 } }, "diffs": { "tool_snippets - no_tool": { "diff": 0.7333333333333333, "lo": 0.5333333333333333, "hi": 0.9 }, "tool_page - no_tool": { "diff": 0.5666666666666667, "lo": 0.4, "hi": 0.7333333333333333 }, "tool_snippets - tool_page": { "diff": 0.16666666666666666, "lo": 0, "hi": 0.3333333333333333 } } }, { "model": "llama-3.3-70b-instruct-fp8-fast", "n": 30, "arms": { "no_tool": { "correct": 4, "acc": 0.13333333333333333, "avgInTok": 86, "avgOutTok": 27, "avgMs": 979, "toolUseRate": 0, "retrievalMiss": null, "readingMiss": null }, "tool_snippets": { "correct": 26, "acc": 0.8666666666666667, "avgInTok": 1129, "avgOutTok": 57, "avgMs": 2465, "toolUseRate": 1, "retrievalMiss": 4, "readingMiss": 0 }, "tool_page": { "correct": 22, "acc": 0.7333333333333333, "avgInTok": 2102, "avgOutTok": 56, "avgMs": 2766, "toolUseRate": 1, "retrievalMiss": 8, "readingMiss": 0 } }, "diffs": { "tool_snippets - no_tool": { "diff": 0.7333333333333333, "lo": 0.5666666666666667, "hi": 0.8666666666666667 }, "tool_page - no_tool": { "diff": 0.6, "lo": 0.43333333333333335, "hi": 0.7666666666666667 }, "tool_snippets - tool_page": { "diff": 0.13333333333333333, "lo": -0.06666666666666667, "hi": 0.3333333333333333 } } } ], "examples": [ { "id": "fn-region-param", "arm": "no_tool", "correct": false, "answer": "The query parameter that forces a Supabase Edge Function to run in a specific region is `region`.", "queries": [] }, { "id": "mcp-features", "arm": "no_tool", "correct": false, "answer": "The URL parameter that enables only specific tool groups on the Supabase MCP server is `tool_groups`.", "queries": [] }, { "id": "cron-pgver", "arm": "no_tool", "correct": false, "answer": "Supabase Cron jobs can be scheduled with a seconds interval from Postgres version 13.", "queries": [] }, { "id": "fn-region-param", "arm": "tool_snippets", "correct": true, "answer": "The query parameter that forces a Supabase Edge Function to run in a specific region is `forceFunctionRegion`.", "queries": [ "edge function region" ] }, { "id": "mcp-features", "arm": "tool_snippets", "correct": true, "answer": "The URL parameter that enables only specific tool groups on the Supabase MCP server is `features=`.", "queries": [ "MCP server tool groups URL parameter" ] }, { "id": "cron-pgver", "arm": "tool_snippets", "correct": true, "answer": "You can input seconds for your Job schedule interval as long as you're on Postgres version 15.1.1.61 or later.", "queries": [ "Postgres version Supabase Cron jobs seconds interval" ] }, { "id": "fn-region-param", "arm": "tool_page", "correct": true, "answer": "The query parameter that forces a Supabase Edge Function to run in a specific region is `forceFunctionRegion`.", "queries": [ "edge function region query parameter" ] }, { "id": "mcp-features", "arm": "tool_page", "correct": false, "answer": "The URL parameter that enables only specific tool groups on the Supabase MCP server is `tool_groups`.", "queries": [ "MCP server tool groups URL parameter" ] }, { "id": "cron-pgver", "arm": "tool_page", "correct": true, "answer": "You can input seconds for your Job schedule interval as long as you're on Postgres version 15.1.1.61 or later.", "queries": [ "Postgres version Supabase Cron jobs seconds interval" ] } ], "findings": "
    \n
  1. The tool is the whole result. Without it the models got 2\u20134 of 30 right, even Llama 3.3 70B, and 80 of their 81 wrong answers were confident, specific and wrong (e.g. \"128 MB\", \"100 connections\"): plan limits, config keys and flags are exactly what a model cannot know. With section snippets they got 24\u201327 of 30 (+73 to +80 points, every CI well above zero).
    2. \n
  2. Snippets beat whole pages in all three models (27 vs 23, 24 vs 19, 26 vs 22), at roughly half the input tokens (~1.1k vs 1.5\u20132.1k per question). Pooled over models the gap is +14 points, 95% CI \u22122 to +32: a consistent direction, but 30 questions cannot separate it from noise.
    3. \n
  3. Why pages lose: two mechanisms, both visible in the traces. (a) The best section ranked 2nd or 3rd, so \"top page\" returned the wrong page (idle timeout \u2192 the error-codes page; Realtime free-plan connections \u2192 the reports page). (b) The page was right but long, and the answer sat past the 6k-character cut (the MCP guide is ~25k characters; its URL-parameter table is near the end). Top-3 sections from different pages hedge against both.
    4. \n
  4. Almost every miss is retrieval, not reading. Of 13 wrong answers in the snippets arm, 12 never had the answer key in any tool result. The one reading miss: Llama 3.2 3B answered too_many_joins from a list that also held the right too_many_channels. The next gain is in the index and the tool's query handling (e.g. \"memory limit\" pulls the error-codes page before the limits page), not in the prompt.
    5. \n
  5. A tool changes how the model fails. With snippets, 5 of 13 wrong answers said the docs did not state the value; without a tool, 1 of 81 did. Still, 8 confident wrong answers remain, so a production gate should check abstention as well as accuracy.
    6. \n
\n

Neurons used for all 270 runs: \u22484.4k on the Workers AI free tier.

" }