Distillation, watched live
Knowledge distillation trains a small student model to imitate a larger teacher. The usual version is a training job you can't see. This is my vllm-model-distillation backend ported to the browser: a teacher and a student answer the same prompt side by side via WebGPU, with latency, token counts, a teacher-as-judge pass, and a one-click dataset export.
Two models, one GPU. Teacher Qwen2.5-1.5B-Instruct and student Llama-3.2-1B
together are ~2 GB of VRAM, so they load sequentially and the previous one is
unload()ed before the next runs — exactly the constraint the server sidesteps with a real GPU.
No WebGPU? Hit Run demo for a recorded comparison so the charts still render.
The comparison
Same prompt in. Two very different budgets out.
The teacher gets a plain helpful-assistant prompt (build_single_model_prompt); the student gets
the teacher's answer plus distillation instructions to match its quality in its own words
(build_distillation_prompt). Every run is logged as a
(prompt, teacher_out, student_out) triple.
The payoff
Is the student cheaper — and is it good enough?
That's the whole distillation question. Latency answers "cheaper"; the teacher-as-judge scores answer "good enough".
Latency per prompt — teacher vs student
Lower is cheaper. The gap is what distillation is trying to buy you.
Teacher-as-judge — student scores
The teacher grades the student 1–5 on three axes (build_comparison_prompt). Run a judge pass to fill this in.
The dataset
Every run is a training row you can keep.
Each comparison is stored to IndexedDB and can be exported as JSON in the exact shape of the backend's
GET /prompts/export/project/{id} — a project with prompts, each carrying its teacher and student
responses, latencies and judge scores. That file is a distillation dataset: prompts paired with teacher
targets, ready to fine-tune a smaller student.