Evaluating Jev
I’m testing how typed judgments behave across classification, prompt changes, and tool selection. These are small, inspectable probes. They don’t establish benchmark accuracy.
Inspect state, questions & criteria
Model output
Illustrative policy applied in code. This cutoff is not calibrated; moving it does not rerun inference.
Exact response
Demo workspace · no real orders are changed
Execution trace
RECORDED RUNJev selects a tool and closed-set arguments in one request. Deep Agents executes the selected Python function, then passes its result back. Code enforces the demo’s policy. Expand a step to inspect its probabilities and returned data.
Edit the questions +
Method & limitations
READ BEFORE GENERALIZINGSample & procedure
15 hand-authored classification examples across six tasks, eight prompt trials, and three agent traces. Saved requests and responses were recorded on September 17–18, 2026. Inputs are synthetic. There is no held-out test set or aggregate accuracy estimate.
Edit a task input to run a new classification. Each response belongs to the exact state and question set shown. The model alias jev-latest may change over time.
What the outputs mean
Choice returns a distribution over labels. Noul estimates a yes probability. Score applies an ordered rubric. Selected-label probability is distinct from the API’s confidence field; neither establishes correctness.
Latency is a single observed request duration, not a controlled speed benchmark. Review thresholds and composite weights are illustrative policies.
Observed failure
In the reversed-guard code probe, the comparison on L2 needs repair. Jev selects L3, including after prompt refinement. Both requests and their original responses remain in Prompt revisions.
Agent traces use a bounded synthetic workspace. Code enforces policy; Jev selects tools and closed-set arguments. Live agent execution requires the local Deep Agents runtime.
Cookbook reference notes18 RECIPES / REVIEWED 18 SEP 2026
Provider recipes mapped to applications and evaluation caveats. These are documentation notes; the workflows have not all been reproduced here.
| Recipe | Pattern & application | Evaluation caveat |
|---|---|---|
| Self-consistency: nouls ↗Decision policies | Keep a review band around uncertain yes/no probabilities. Triage ambiguous cases without forcing a binary action. | Repeatability is not accuracy; the sample varies an irrelevant uid and covers one claim. |
| Self-consistency: choices ↗Decision policies | Abstain when the winning label probability is too low. Moderation and routing with a visible human-review option. | Top-label probability and the separate confidence field are different. Thresholds are illustrative. |
| Parallel questions ↗Batching | Ask independent Choice, Noul, and Score questions over one shared state. Evaluate many review dimensions in a single document request. | The reported speed comparison uses sequential single-question calls; gains depend on workload. |
| Re-ranking ↗Search & retrieval | Use fast retrieval for a shortlist, then Noul-score each query–candidate pair. Search results, help-center retrieval, and RAG reranking. | Reranking cannot recover documents missing from the shortlist; the example uses 40 queries. |
| Line-by-line search ↗Search & retrieval | Rank line IDs with Choice and ask a separate Noul about answer existence. Find supporting clauses and highlight exact source lines. | A winning line can still be irrelevant. More than 255 candidate lines requires staged search. |
| Structure recovery ↗Document processing | Batch line-continuation decisions, then classify blocks and render markup in code. Recover headings, lists, quotations, and code from flattened text. | The document takes two dependent passes; join thresholds and ambiguous block types need evaluation. |
| Function calling ↗Agents | Map functions and enum arguments to Choices; flags, sets, and optional presence to Nouls. Typed tool dispatch for agents and natural-language interfaces. | Consume only the selected branch. Free text and arbitrary numeric arguments are not generated by this recipe. |
| Skill suggestion ↗Agents | Rank a skill roster, then inspect the top few with richer descriptions and fit checks. Choose an agent skill or abstain when no skill applies. | A suggestion can make an already-correct agent worse; the study includes generated requests and a documented false suggestion. |
| Knowledge graph entity alignment ↗Data matching | Use an ordered match rubric plus independent field-agreement checks. Deduplicate products or align records across catalogs. | Candidates must already exist. A three-level Score is rounded into different/review/same; semantic levels still need domain validation. |
| Classifying RAG passages ↗Search & retrieval | Check relevance, answer evidence, contradiction, and injection for each passage. Separate useful evidence, conflicting evidence, and excluded passages before generation. | The filter is not a security boundary. A generative model still writes the answer. |
| Double-checking citations ↗Verification | Check quote presence in code, then judge whether its context supports the claim. Audit generated citations and surface unsupported claims. | Exact normalized matching can reject paraphrases. Source support does not establish the source’s truth. |
| Guardrails for LLMs ↗Verification | Evaluate independent hazards and severity, then apply a named policy in code. Screen both model inputs and outputs; route to pass, review, block, or support. | Policy thresholds need labeled traffic; a probability threshold is not guaranteed protection. |
| SDE cascade ↗Verification | Verify a cheap extractor’s fields and escalate when any specific error signal fires. Spend stronger-model calls on difficult structured extraction cases. | Requires additional extraction models. The walkthrough fixes one fabricated example; its aggregate quality chart is an internal snapshot. |
| Date extraction ↗Extraction | Choose date components and mode, then resolve calendar arithmetic in code. Read deadlines, relative dates, and explicit dates with review flags. | Date conventions, missing years, out-of-range years, and unsupported expressions need explicit code policies. |
| Pre-parsed value extraction ↗Extraction | Find candidate spans first, let Jev select by role, then copy and normalize. Extract the right email, phone number, or invoice amount without rewriting it. | Candidate discovery limits recall; names need a roster or another extractor, and money normalization is locale-dependent. |
| Hierarchical classification ↗Classification | Keep multiple taxonomy paths and batch frontier questions using beam search. Navigate product catalogs, code trees, patents, and subject taxonomies. | The recipe shows four examples, not broad accuracy evidence. Path scores use a geometric mean, not a calibrated leaf probability. |
| Autoresearch feature discovery ↗Feature discovery | An LLM proposes questions; Jev produces numeric features; supervised errors guide revisions. Learn useful text features for rating prediction and other labeled-text tasks. | Needs labeled data, a proposer model, a learner, and an untouched test set. Inference cost scales with rows and rounds. |
| Classification using confidence ↗Classification | Use the returned confidence to choose a narrow label or its broader parent. Return useful coarse categories instead of over-specific guesses. | The study uses 60 filtered, self-labeled filings. Reporting a parent category is still fallible. |