What is an eval?

An eval is a test suite for a model. You can't assert output === expected on free-form text, so instead you score it — several different ways, because every scoring method is wrong in its own way. This page runs a real eval suite entirely in your browser and shows you what each method sees.

The four scorers

Every method disagrees with the others. That's the point.

Each test case below is scored four ways. Watch where they disagree — a case can pass the cheap string check and fail the judge, or produce broken JSON while still being semantically right.

Ground truth

The hand-labeled right answer. Everything else is measured against it. Expensive to produce, which is why eval sets stay small and precious.

gold: { sentiment: "neutral", summary: "…" }

Exact match

Did the label come back exactly right? Cheap, deterministic, zero nuance — perfect for classification, useless for prose.

pred === gold → 1 else 0

Fuzzy similarity (token F1)

Word overlap between the model's summary and the reference. Catches "close enough", but rewards copying words and misses meaning.

2·P·R / (P+R) over shared tokens

LLM as a judge

A second model grades the answer 1–5 against a rubric and explains itself. Catches meaning the string metrics miss — but it's slower, costs money, and can be confidently wrong.

rubric → score 1–5 → normalized 0–1

The run

Score 8 support tickets, four ways.

The task: read a customer message, return JSON with a sentiment label and a one-line summary. Demo uses recorded outputs from a small model so the charts render instantly. Live runs the real model on your GPU — first load downloads ~1 GB of weights.

Live model Showing recorded demo run.

Score by method

Mean score across all 8 cases, per scorer. Darker = higher.

Where the cheap metric and the judge disagree

One row per test case. Distance between the two marks is the disagreement.

Token F1 (cheap) LLM judge

Confusion matrix — sentiment label

Rows are the true label, columns are what the model predicted. The diagonal is correct.

Case by case

The table view — every score, every judge rationale.

This is the same data as the charts above, in full. Charts show you the shape; the table is where you debug.

#	Customer message	Gold	Predicted	Exact	F1	Judge	JSON

Wiring this to real tooling

In production you'd ship these scores to Langfuse.

Langfuse is a server-side observability platform — it needs a backend and secret API keys, so it can't run on a static site like this one (a key in frontend JS is a public key). What this page can do is emit traces in Langfuse's shape: hit Export Langfuse traces above to download the run. In a real service you'd POST the same objects from your backend:

// server-side only — never in the browser
import { Langfuse } from "langfuse";
const lf = new Langfuse({ secretKey: process.env.LANGFUSE_SECRET_KEY });

const trace = lf.trace({ name: "sentiment-eval", metadata: { suite: "support-tickets" } });
for (const c of cases) {
  const gen = trace.generation({ name: "classify", input: c.input, output: c.raw });
  gen.score({ name: "exact_match", value: c.exact });   // 0 or 1
  gen.score({ name: "token_f1",    value: c.f1 });      // 0..1
  gen.score({ name: "llm_judge",   value: c.judge,      // 0..1
              comment: c.rationale });
}