Does a docs chatbot need Sonnet?
I wrote a documentation-retrieval assistant and found that, on straightforward questions, DeepSeek V4-Flash scored within run-to-run noise of Claude Sonnet 4.6 at about 1/56th the cost per query.
I’m a big fan of Pure Data (Pd), a music programming language, so I built a RAG chatbot that synthesizes reference material from three sources: Miller Puckette’s book The Theory and Techniques of Electronic Music (Puckette is Pd’s creator), the official Pd manual, and the IEM object reference. Retrieval is hybrid (vector + BM25, merged with reciprocal rank fusion), with prompt caching and streaming responses. Once it worked, the real question was whether the model behind it was overspecified. Did answering Pd questions actually need Claude Sonnet, or would a much cheaper model do just as well?
How I tested it
I scored every answer with an LLM-as-judge on three axes:
- Faithfulness: does it invent anything not in the docs?
- Relevance: does it answer what was asked?
- Citation correctness: are the sources it points to real?
I ran this across two question sets:
- An “easy” set: 30 questions mixing object lookups like “what do
osc~’s inlets do”, conceptual questions about patching and DSP, and a couple of multi-turn conversations. - A “hard” set: 12 questions requiring multi-object synthesis, tradeoff reasoning, or knowledge of things the docs never cover, designed specifically to try to break each model.
One methodology detail worth calling out: I judged everything with Gemini (gemini-3.5-flash) rather than a Claude model, so the judge wasn’t grading its own family’s homework. That choice mattered. My first judge was Haiku, which is in the same family as two of the four models being compared. Under Haiku, DeepSeek’s citation-correctness scores (4.07–4.43) looked dramatically worse than Sonnet’s (4.83). Under Gemini, the gap nearly disappeared (4.90–4.98 vs 5.00).
Results: easy set
n=30, 3 trials, mean ± std. Cost is computed from measured token usage, including cache reads, at each provider’s published rates in July 2026.
| Model | Faithfulness | Relevance | Citation correctness | Cost / 1,000 queries |
|---|---|---|---|---|
| Claude Sonnet 4.6 | 4.92 ± 0.02 | 5.00 ± 0.00 | 5.00 ± 0.00 | $24.62 |
| Claude Haiku 4.5 | 4.89 ± 0.11 | 4.99 ± 0.02 | 4.98 ± 0.02 | $6.46 |
| DeepSeek V4-Pro | 4.89 ± 0.07 | 5.00 ± 0.00 | 4.90 ± 0.06 | $1.86 |
| DeepSeek V4-Flash | 4.78 ± 0.10 | 4.99 ± 0.02 | 4.94 ± 0.02 | $0.44 |
All four models land within 0.14 points of each other on every axis. With only three trials that is not enough to separate them, and cost is the only axis that clearly distinguishes them here.
Results: hard set
n=12, 2 trials, mean.
| Model | Faithfulness | Relevance | Citation correctness |
|---|---|---|---|
| Claude Sonnet 4.6 | 4.92 | 4.96 | 5.00 |
| DeepSeek V4-Pro | 4.83 | 4.96 | 4.92 |
| Claude Haiku 4.5 | 4.63 | 4.80 | 4.96 |
| DeepSeek V4-Flash | 4.62 | 5.00 | 4.79 |
Once questions force real synthesis and honesty about undocumented behavior, a gap opens up. But even the cheapest model stayed above 4.6/5, and Sonnet wasn’t perfect either: on a reverb-design question it stated a specific feedback-gain value and attributed it to the docs, which never give one.
Takeaway
The size of the quality gap between a frontier model and a fraction of its cost depends entirely on how hard you’re willing to look for it. And cost separates these models by orders of magnitude well before quality does.
The code, eval harness and question sets are on GitHub.
Update, October 2026: DeepSeek has since retired V4-Flash. Requests to deepseek-v4-flash are now served by V4.1-Flash, so the V4-Flash numbers above describe a model you can no longer call. The Claude prices used here are unchanged.