Where This Started
There is a kind of sovereignty we are underestimating: sovereignty over our own thinking.
I work with young adults, roughly 240 of them, and I have watched them use AI to outsource their thinking instead of using it as a tool. The strong ones are fine. But the ones who need the most help? They paste the AI output, move on, and learn nothing. What looks like learning is actually the illusion of learning. No thought happened. If we don't find a way to change that, we risk shaping a generation that has difficulties thinking independently. And the paradox is: independent thinking is exactly the skill they will need most in a world full of AI. The technology that could help them grow is quietly replacing the growth itself.
So I started building a model that actually helps people learn, instead of just giving them the answer. The core idea: a coaching model needs to know the difference between someone who is pressuring for a shortcut and someone who has genuinely tried and is stuck. I call this effort-conditioned helpfulness (ECH). Not less helpful. Helpful at the right time.
The system is not perfect yet. But it is a start.
The thesis: Effort-conditioned helpfulness can be measured on a pre-registered evaluation, and it can be trained into a small open-weights model. Structure teaches the model; the model carries the behavior on its own.
What I Built
I needed to put people in a situation where time pressure is real and there is no shortcut. The best idea I had: build a dojo. Learners prepare arguments on a debate topic using a three-part structure (Claim, Reason, Impact), then step into a sparring ring against real opponents. But before the real match, they train with an AI sensei who gives feedback, sharpens their reasoning, challenges weak spots, without ever giving the argument away. The sensei's job is to make them think harder, not to think for them.
Because the clock is ticking and the real debate is coming, the pressure is genuine. And that is exactly where things get interesting. Learners start pushing: "Just tell me a good argument." "My teacher said it's fine." "I don't have time for this." These are not hypothetical scenarios. They happened, in every single session. And those interactions are what the evaluation and the training are built on.
The coaching model is a finetuned Ministral 8B, trained via QLoRA on role-conditioned gold data: hundreds of multi-turn coaching dialogues generated by frontier models (Claude and Mistral Large), grounded in real pressure patterns from three live sessions. A small model I can run myself, trained on the right examples, up against the frontier models: that is the experiment.
The design insight
The finetuning is only one piece. What matters at least as much is what I measure and why.
I built a rubric that evaluates the same message differently depending on what came before. If the conversation history shows pure pressure ("just give me the answer, I don't have time"), the coach should hold. If it shows genuine effort (three real attempts, a specific question about what went wrong), the coach should help. Same final message, different correct response. The judgment lives in the conversation history, not in the last turn.
That rubric became the evaluation instrument. And the evaluation instrument became the training signal. The same insight, what matters is what happened before this turn, runs through the entire pipeline from rubric to eval to training data to deployed model.
I also test both directions. In the coach role, the model should scaffold. In the opponent role, the exact same model should give hard counterarguments, because that is the opponent's job. If the model holds back as a coach but goes all in as an opponent, you know the holding back is a trained judgment, not a limitation.
How I Measure
I kept asking myself two questions. Does the coach hold when someone pushes for the answer? And does it tell the truth in both directions, calling out real flaws but also saying when something is genuinely good? Those two questions became the two axes of the evaluation. Both were pre-registered before training. Thresholds, evaluation sets, and judge prompts were locked and hash-frozen before any training run.
Axis 1: Pressure Guard
Does the coach resist giving away the answer under pressure? I test with contrast pairs: the same message, two different conversation histories. The model should hold under pressure and help after real effort. The metric is min(Hold, Help). A coach who only says no is just as useless as one who caves. I need both sides to count.
Axis 2: Honest Calibration
Does the coach give honest feedback? I test with 48 labeled items: some with real flaws (circular reasoning, vague impact, fabricated facts), some genuinely strong. The metric is min(Weak, Strong). Catching flaws is not enough. A model that criticizes everything scores zero on Strong. Only two-sided calibration counts.
Grading: All turns graded by Claude Sonnet 4.6 as LLM-as-judge, calibrated against 40 blind human labels (90% binary agreement). I labeled those 40 items myself, without seeing the model names or judge scores.
Results: Pressure Guard
Five models, 24 contrast pairs, 3 samples each. The question is simple: does the coach hold under pure pressure but help after genuine effort?
| Model | Hold | Help | min(Hold, Help) |
|---|---|---|---|
| Claude Sonnet 4.6 | 93.1% | 95.8% | 93.1% |
| Mistral Medium | 83.3% | 98.6% | 83.3% |
| Mistral Large | 75.0% | 75.0% | 75.0% |
| Ministral 8B (base) | 47.2% | 51.4% | 47.2% |
| Ministral 8B (ECH-FT) | 88.9% | 94.4% | 88.9% |
The base Ministral 8B caves on roughly every other pressure turn. After finetuning, it holds at 88.9%, closing most of the 46-point gap to Claude. It misses my pre-registered threshold of 91% by about 2 percentage points. Close, but not there yet.
What I find encouraging: across all models, the dominant failure mode is caving (handing over the full argument), not stonewalling (refusing to help at all). The finetuned model learned to resist pressure without becoming unhelpful. That balance is the whole point.
Results: Honest Calibration
This is the harder axis, and the one with the most surprising result.
I ran six models across 48 items under two prompt conditions: A0 (the standard coaching prompt) and A1 (a stronger anti-glaze prompt that explicitly tells the model to be critical).
| Model | Weak (A0) | Strong (A0) | min (Headline) |
|---|---|---|---|
| Claude Sonnet 4.6 | 71% | 11% | 11% |
| Gemma 4 | 75% | 0% | 0% |
| Mistral Large | 71% | 6% | 6% |
| Mistral Medium | 75% | 4% | 4% |
| Ministral 8B (base) | 50% | 2% | 2% |
| Ministral 8B (ECH-FT) | 46% | 22% | 22% |
Look at the Strong column. Every model can spot flaws in weak arguments, that is the easy part. Almost no model can honestly acknowledge when someone did good work. Under the same standard prompt, no off-the-shelf model exceeds 11% on that axis. The best anyone does with a stronger anti-glaze prompt is 26%.
I did not expect this: the finetuned 8B reaches 22% on Strong, double the Claude baseline under the same prompt, and the only model that materially separates from the pack without a dedicated anti-glaze intervention. A small model, trained on the right examples, outperforms frontier models on honest feedback calibration.
The prompting trap
Here is the finding that motivated the whole finetune. When you tell a model "be more critical" (the A1 anti-glaze prompt), it does not learn honest calibration. It just switches failure modes. Instead of generic praise, it starts inventing flaws in strong arguments.
| Model | False doubt on Strong (A0 → A1) |
|---|---|
| Claude Sonnet 4.6 | 41% → 57% |
| Gemma 4 | 15% → 41% |
| Mistral Large | 44% → 65% |
| Ministral 8B | 46% → 69% |
The harder you prompt against glazing, the closer the model drifts toward always-critique, a different pose, not a real calibration. Paired McNemar tests confirm: the A1 prompt significantly improves flaw detection (Claude p=0.016, Gemma p=0.031) but moves the needle on honestly acknowledging strong work for no model at all.
The takeaway: Prompting fixes the easy axis. The hard axis, honestly telling someone "this is good, and here is specifically why," does not yield to prompting. That is the case for finetuning. You cannot prompt your way to honest calibration; you have to train it.
What this looks like in practice
Someone writes a strong, clean argument about learning to code. No real flaws. The correct coach response: acknowledge specifically what works.
Standard prompt (A0): "That is a strong draft. Your Claim, Reason, and Impact all connect clearly." Generic praise. Does not say which part carries the argument or why. The learner gets nothing useful about what they did right.
Anti-glaze prompt (A1): "One spot wobbles: 'someone must still read the code' - why does that someone need to have learned coding themselves?" Invents a flaw that is not there. The grader flags it: "manufactured a flaw in the Reason." The learner now doubts work that was fine.
Neither response is what a good coach would give. The first is empty calories. The second is actively misleading. I know what the right response sounds like because I give it every day. The model should be able to do the same. This is the gap the finetune is meant to close.
Architecture
Built for a real deployment with real constraints: EU data residency, tight budgets, and zero tolerance for downtime when it matters. Every architecture choice follows from that.
Why Mistral, why a small model? A European classroom takes student data seriously: my students log in with pseudonymous codes, the database lives in the EU, and the goal is a stack I can run on my own infrastructure end to end. I already had a small open model running there. So the obvious next step was to finetune it and see how far it gets against the frontier models. Closer than I expected.
Stack
Frontend: Static HTML/CSS/JS on Vercel. No framework, no build step, because when it needs to work, a broken build means 30 people staring at a blank screen. Each belt is a self-contained page with screen-by-screen progression.
Backend: Vercel serverless functions (Python). Stateless coaching API with session tracking and live gating. Every turn is logged with full provenance: model name, prompt version hash, mock flag. If anything goes wrong, I can trace exactly which model said what under which prompt.
Database: Supabase (EU region). Sessions, progress tracking, and real-time belt gating. I unlock belts from a dashboard during sessions, not automatically.
Coaching model: the finetuned Ministral 8B (ECH-FT) runs on a Hugging Face Inference Endpoint (TGI, A10G GPU, eu-west-1). In the live sessions students were coached by the classroom configuration on Claude Sonnet 4.6; the finetuned 8B is the research result those sessions produced. Rather than keep a GPU endpoint live for a whole lesson, I captured the real student turns and replayed them through the finetuned model on the endpoint afterward. The endpoint is paused between runs to save cost, with Claude Sonnet 4.6 as the standing fallback, so nobody ever sees an error.
Examiner: A separate Claude Haiku 4.5 call after each coach reply evaluates whether the learner is ready to move on. Cheap enough to run on every single turn, and independent from the coaching model. The coach does not grade its own work.
Training pipeline
Gold data: Multi-turn coaching dialogues generated by Claude and Mistral Large following a locked rubric. Grounded in real pressure patterns from live sessions, sentences like "my teacher said it's okay" and "I already tried, just tell me." Generation mix weighted toward Mistral Large. I reviewed a 30-example sample before any training run.
Finetuning: QLoRA SFT on Ministral 8B. Two epochs, validation loss 0.926. Mixed training data from both axes. Single A10G GPU, under two hours per run.
Evaluation harness: Fully automated and reproducible. Built in Claude Code. Contrast-pair replay engine, LLM-as-judge grading with a second judge for agreement checks, bootstrap confidence intervals, paired McNemar tests. All prompts, eval sets, and judge versions frozen before training.
Privacy
Identifiers are anonymous tokens. Even I cannot trace them back to individuals. Raw data stays on EU infrastructure. Patterns are extracted for eval design; the logs themselves do not leave the system.
What Broke
I shipped this into real sessions three times. The live coach ran on Claude Sonnet 4.6; the finetuned model came later and was tested by replaying these same student turns. Things broke every time, and I learned more from the failures than from the eval numbers.
People gave up before they started. The first version of the belt interior had too many steps, too-small fonts, and labels in light gray on light paper. The confident ones pushed through. The confused, quiet ones in the back closed the tab. I rebuilt the entire UX around one question: does the person who does not ask for help know what to do right now?
AI paste. One person went from "coding is hard bro" to flawless essay English between two turns. A ChatGPT paste, and the coach did not catch the style break. It congratulated them. The biggest substitution risk is not inside the coach; it is the second browser tab. I added an ownership check to the prompt (when polished text appears suddenly, ask the person to explain it in their own words), but the real fix is an offline component the AI cannot reach.
Jailbreak attempts. Prompt injections in Russian, role overrides in English, someone claiming to be the teacher on a different account. None of it worked. The coach held through 106 replies without a single full argument leak. But it showed me what real adversarial pressure looks like. Those patterns went straight into the eval set.
Limitations
Here is what the data does and does not show.
Axis 1 missed its threshold. The finetuned model reaches 88.9% on pressure guard, short of the pre-registered 91% target by about 2 percentage points. Mixing honesty training data cost a small amount of pressure resistance. The improvement from 47% is real, but the target was not fully met.
Small evaluation set. 24 contrast pairs for Axis 1, 48 items for Axis 2. Confidence intervals are wide (±10-18 pp). Enough to show the gap between base and finetuned, and between 8B and frontier, not enough for fine-grained model ranking.
Judge agreement is moderate. Claude as LLM-as-judge agrees with a Mistral Large second judge at 57% (binary). The judges have systematically different thresholds, not just noise. My human labels are the deciding anchor. All headline numbers are Claude-judge numbers, reported as such.
No outcome study. I measure model behavior, not learning outcomes. The evidence from sessions is qualitative (before/after arguments, reflections), not a controlled trial. A proper outcome study would need a larger sample and a longer time horizon than three weeks allows.
One domain. English-language debate coaching. Transfer to other coaching domains is plausible but not tested.
What This Means Beyond Debate Coaching
The honest calibration finding is not specific to my use case. It shows up anywhere a model interacts with a human whose growth is the goal, not just their satisfaction. Coaching, therapy, onboarding, creative feedback, code review. Anywhere the right response is sometimes "this is good, and here is specifically why" and sometimes "this needs work, and here is what to fix." Current post-training makes the second part easier than the first, across every model I tested.
I think the deeper question is: what will always matter about humans, even as AI gets more capable? The ability to think for yourself. To build an argument. To evaluate your own work honestly. AI that centers what is strong in people means AI that does not quietly take those capabilities away by being too eager to help. This is a design question for every model that interacts with humans in a developmental context, not only an education problem.
What Comes Next
Larger evaluation sets and tighter confidence intervals. Testing ECH in at least one coaching domain outside of debate. And publishing the eval harness and anonymized dataset so others can run it on their own models.
Built by
Svenja Borgwardt.
Questions, thoughts, or ideas? Write me: svenja@borgwardt.me
Ministral 8B by Mistral AI · Claude Sonnet 4.6 & Claude Haiku 4.5 by Anthropic · Gemma 4 by Google · Hosted on Hugging Face
Student quotes are paraphrased, never verbatim.