← ExperimentsRESEARCH NOTES / 01

Evaluating Jev

I’m testing how typed judgments behave across classification, prompt changes, and tool selection. These are small, inspectable probes. They don’t establish benchmark accuracy.

Amaljith Kuttamath17–18 Sep 2026jev-latestMethod & limitations ↓
EXPLORATORY · NOT A BENCHMARK
Playground ↗

Inspect state, questions & criteria

Model output

Illustrative policy applied in code. This cutoff is not calibrated; moving it does not rerun inference.

Exact response

Method & limitations

READ BEFORE GENERALIZING

Sample & procedure

15 hand-authored classification examples across six tasks, eight prompt trials, and three agent traces. Saved requests and responses were recorded on September 17–18, 2026. Inputs are synthetic. There is no held-out test set or aggregate accuracy estimate.

Edit a task input to run a new classification. Each response belongs to the exact state and question set shown. The model alias jev-latest may change over time.

What the outputs mean

Choice returns a distribution over labels. Noul estimates a yes probability. Score applies an ordered rubric. Selected-label probability is distinct from the API’s confidence field; neither establishes correctness.

Latency is a single observed request duration, not a controlled speed benchmark. Review thresholds and composite weights are illustrative policies.

Observed failure

In the reversed-guard code probe, the comparison on L2 needs repair. Jev selects L3, including after prompt refinement. Both requests and their original responses remain in Prompt revisions.

Agent traces use a bounded synthetic workspace. Code enforces policy; Jev selects tools and closed-set arguments. Live agent execution requires the local Deep Agents runtime.

Cookbook reference notes18 RECIPES / REVIEWED 18 SEP 2026

Provider recipes mapped to applications and evaluation caveats. These are documentation notes; the workflows have not all been reproduced here.

RecipePattern & applicationEvaluation caveat
Self-consistency: nouls ↗Decision policies

Keep a review band around uncertain yes/no probabilities.

Triage ambiguous cases without forcing a binary action.

Repeatability is not accuracy; the sample varies an irrelevant uid and covers one claim.
Self-consistency: choices ↗Decision policies

Abstain when the winning label probability is too low.

Moderation and routing with a visible human-review option.

Top-label probability and the separate confidence field are different. Thresholds are illustrative.
Parallel questions ↗Batching

Ask independent Choice, Noul, and Score questions over one shared state.

Evaluate many review dimensions in a single document request.

The reported speed comparison uses sequential single-question calls; gains depend on workload.
Re-ranking ↗Search & retrieval

Use fast retrieval for a shortlist, then Noul-score each query–candidate pair.

Search results, help-center retrieval, and RAG reranking.

Reranking cannot recover documents missing from the shortlist; the example uses 40 queries.
Line-by-line search ↗Search & retrieval

Rank line IDs with Choice and ask a separate Noul about answer existence.

Find supporting clauses and highlight exact source lines.

A winning line can still be irrelevant. More than 255 candidate lines requires staged search.
Structure recovery ↗Document processing

Batch line-continuation decisions, then classify blocks and render markup in code.

Recover headings, lists, quotations, and code from flattened text.

The document takes two dependent passes; join thresholds and ambiguous block types need evaluation.
Function calling ↗Agents

Map functions and enum arguments to Choices; flags, sets, and optional presence to Nouls.

Typed tool dispatch for agents and natural-language interfaces.

Consume only the selected branch. Free text and arbitrary numeric arguments are not generated by this recipe.
Skill suggestion ↗Agents

Rank a skill roster, then inspect the top few with richer descriptions and fit checks.

Choose an agent skill or abstain when no skill applies.

A suggestion can make an already-correct agent worse; the study includes generated requests and a documented false suggestion.
Knowledge graph entity alignment ↗Data matching

Use an ordered match rubric plus independent field-agreement checks.

Deduplicate products or align records across catalogs.

Candidates must already exist. A three-level Score is rounded into different/review/same; semantic levels still need domain validation.
Classifying RAG passages ↗Search & retrieval

Check relevance, answer evidence, contradiction, and injection for each passage.

Separate useful evidence, conflicting evidence, and excluded passages before generation.

The filter is not a security boundary. A generative model still writes the answer.
Double-checking citations ↗Verification

Check quote presence in code, then judge whether its context supports the claim.

Audit generated citations and surface unsupported claims.

Exact normalized matching can reject paraphrases. Source support does not establish the source’s truth.
Guardrails for LLMs ↗Verification

Evaluate independent hazards and severity, then apply a named policy in code.

Screen both model inputs and outputs; route to pass, review, block, or support.

Policy thresholds need labeled traffic; a probability threshold is not guaranteed protection.
SDE cascade ↗Verification

Verify a cheap extractor’s fields and escalate when any specific error signal fires.

Spend stronger-model calls on difficult structured extraction cases.

Requires additional extraction models. The walkthrough fixes one fabricated example; its aggregate quality chart is an internal snapshot.
Date extraction ↗Extraction

Choose date components and mode, then resolve calendar arithmetic in code.

Read deadlines, relative dates, and explicit dates with review flags.

Date conventions, missing years, out-of-range years, and unsupported expressions need explicit code policies.
Pre-parsed value extraction ↗Extraction

Find candidate spans first, let Jev select by role, then copy and normalize.

Extract the right email, phone number, or invoice amount without rewriting it.

Candidate discovery limits recall; names need a roster or another extractor, and money normalization is locale-dependent.
Hierarchical classification ↗Classification

Keep multiple taxonomy paths and batch frontier questions using beam search.

Navigate product catalogs, code trees, patents, and subject taxonomies.

The recipe shows four examples, not broad accuracy evidence. Path scores use a geometric mean, not a calibrated leaf probability.
Autoresearch feature discovery ↗Feature discovery

An LLM proposes questions; Jev produces numeric features; supervised errors guide revisions.

Learn useful text features for rating prediction and other labeled-text tasks.

Needs labeled data, a proposer model, a learner, and an untouched test set. Inference cost scales with rows and rounds.
Classification using confidence ↗Classification

Use the returned confidence to choose a narrow label or its broader parent.

Return useful coarse categories instead of over-specific guesses.

The study uses 60 filtered, self-labeled filings. Reporting a parent category is still fallible.

Evaluation notebook by Amaljith Kuttamath. Recorded outputs are preserved as returned, including misses. Inspect or download the raw responses alongside each probe.

Recorded data ↗Runtime & setup ↗