Use real decisions and agents

Continue from the repository quickstart. This is the same research workflow, with a runner that can switch from scripted responses to your configured models.

Should our team move its docs from a wiki into the code repository? Search reads a bundled fictional corpus about reviews, editing and search. Jev selects evidence for each subquestion; agent steps turn that evidence into findings and a report.

Connect Jev first

Configure TYPESAFE_API_KEY through your usual secret configuration. Organizations using a gateway also set their approved TYPESAFE_BASE_URL. See Jev configuration.

npm run eval:research -- --live --out evidence-live.json

This tests the evidence selector without running search, planning or writing agents. It prints six PASS/FAIL results and saves the full JSON. Use a new output filename each time; existing reports are never overwritten.

The evaluator uses the actual screen-evidence step, including its rubric and threshold. Expected labels are kept separately and never sent to Jev. The report retains the workflow digest, threshold, probabilities, expected selections and failures.

Add an agent for the report

If Pi is configured, the example uses its saved default. Otherwise, open ./node_modules/.bin/pi, configure access using Pi's setup instructions, and use /model to save your preferred available default. The example reads it without selecting a fallback or changing your settings.

npm run demo:research -- --live --out research-live.json

The planner proposes subquestions, the workflow researches them in parallel, and a final agent writes the report. A question the corpus cannot answer may cause escalation. Pi's agent steps receive the selected evidence; this example grants them no filesystem or shell tools.

Use your existing agents

Pi is optional. Supply a trusted module that default-exports { runJudge, runNode }:

npm run demo:research -- --live --config ./approved-adapters.mjs --out research-live.json

runNode({ system, user, schema, signal }) calls your existing agent with the instructions, JSON input and output schema, then returns its parsed output object. runJudge implements the decision adapter contract; you can use createJevRunner() for it. Evaluation needs only runJudge.

Both adapters must honor signal. Importing the module executes its code. Hosts requiring broker routing should use their approved adapters without a direct-provider fallback. Tools, permissions and provider access stay with the host.

Understand the result

These six fictional cases are a starting point, not a quality benchmark. Add independently labeled cases from your task before choosing a threshold. Citation checks verify that source IDs and text match the corpus; they do not establish that every generated claim is supported or evaluate the planner and writer. No live quality measurements are bundled.

For an offline baseline, omit --live. The output is labeled scripted: it tests execution and evaluation wiring, not model quality. Scripted responses do not adapt to prompt edits.

The npm commands print summaries. Run the underlying .mjs scripts directly for JSON; --out always saves full JSON with private file permissions. Reports appear only after a complete write. Raw SDK responses, provider errors and credentials are omitted; custom adapter logging remains the host's responsibility.

configuration points to missing access or saved-default setup; ERR_MODULE_NOT_FOUND points to a config path or dependency. Confirm the offline command works, then correct the prerequisite.

Ctrl-C and SIGTERM cancel the run. The default whole-command deadline is 600 seconds; use --deadline 900 to change it (1–3600 seconds). The default Pi/Jev limits permit a 540-second execution path, leaving setup margin. Cancellation cannot undo an admitted provider call; custom adapters must cooperate.