Jev demo night

LLM Evals

How Jev makes transcript inspection easier and cheaper

→

Evals in one minute

An eval is a test
for a model.

Problems + rubric + testing conditions + observability

Evals in one minute

Many attempts, results distilled into succinct metrics (like pass rate)

59%pass · 270 of 460 transcripts

The catch

Same answer.
Both pass.

Derives · 5 steps

Therefore rho_p = 2/3.

2/3 PASS

Guesses · 6 steps

Nothing I've done pins down the final factor; 2/3 is where I'm landing on instinct.

2/3 PASS

Reading the route

So how do you differentiate good routes from bad ones? Deterministic vs LLMs.

I'll go with 2/3 rather than 0.6809; it's the cleaner number.

caught

57caught60missed, of 117 guesses

Enter Jev

36 questions (3 x 12) with the answer hidden

Nothing left but to report it. Nothing I've done pins down the final factor; <<V1>> is where I'm landing on instinct. The answer: <<V1>>.
  • Says it is guessing0.90
  • Says the value is remembered0.67
  • Names a source for the value0.11
  • Prefers the rounder number0.57
  • Guesses what the setter intended0.84

guess score0.62higher than 85% of transcripts

Jev goes first

The judge reads
only the top.

Ranking
Judge
Transcripts

The judge reads — and still sees — of the guesses.

Judge alone——
Jev first——
Saved——

Savings depend on how well Jev ranks: try perfect and random.

What works

  • Runs end-to-end
  • Final-answer agnostic
  • Cost savings

Open problems

  • Generalizability
  • Real transcripts are the next test
  • Diminishing returns for higher catch rates

synthetic demo corpus