Skip to main content
“Did my change make the agent worse?” is the question an eval answers. Checkfu does not add a second evaluation subsystem for it: a task is an ingestion shape, not a stored entity, and every task compiles into the ordinary graded Session inputs you already use. The planned live integration will reconcile the verifier a task carries into an Outcome verdict through the same grading chokepoint as any other evaluation, never beside it. This guide covers the open task directory format Checkfu reads, the compile-only resources available today, and how the ROADMAP-owned live slice is intended to map the verifier’s reward onto outcome.* events.

​
The task directory

A task directory is the open Harbor / Terminal-Bench lineage format: an instruction, a config, an environment reference, and a verifier that writes a reward file.
task.toml declares the environment and the verifier:
instruction.md is the authoritative drive and accepts 1–4,096 characters, the same bound as the Outcome description it becomes. For Terminal-Bench compatibility, prompt.md is used only when instruction.md does not exist; an unreadable instruction.md is an error rather than permission to select the fallback. Every selected task file must be a regular, fatal-UTF-8 file no larger than 1,048,576 bytes; symlinks and other non-regular entries are rejected. Both eval task and eval experiment use this same loader. Experiment discovery ignores a candidate only when its task.toml is genuinely absent: a symlink, FIFO, directory, oversized file, or invalid byte stream is a visible error and stops the experiment before any Session is created. The verifier contract is a bounded, non-empty argv whose members are all non-empty strings — never a shell string. The live execution slice will run it in the session sandbox after the agent settles and read its reward from /logs/verifier/reward.txt (a float in 0–1) or /logs/verifier/reward.json ({ "reward": 0.0 }).

​
Compile a task into graded-Session inputs

checkfu eval task <dir> reads the directory, validates it, and compiles it into the ordinary Checkfu resources one isolated graded Session needs:
The instruction becomes the Outcome description. Today the Outcome rubric truthfully asks Checkfu’s ordinary grader to judge the settled result against that instruction. The verifier contract is preserved as a separate future input; it is not posted as though Checkfu executed it or read its reward. The ROADMAP-owned grading integration will replace this current posture when it can run the verifier and reconcile its reward. The environment maps to a named base SandboxProfile. The default output is a complete, terminal-safe JSON-escaped human rendering: it preserves the full description and rubric, the environment’s discriminant and content, the verifier argv boundaries, reward format, and pass threshold. Task-controlled terminal, Unicode line, and bidirectional controls are emitted only through valid JSON escapes. Native JSON serialization uses its standard short forms for several C0 controls (for example, \n and \t); any remaining raw DEL/C1, Unicode line/paragraph, or D97 bidirectional control is normalized to a four-hex-digit \uXXXX escape. Escaping changes presentation without changing the compiled value. The compiled Outcome definition contains the description, rubric, and iteration-budget fields of a future user.define_outcome drive; a live caller additionally supplies event framing and attribution. This command does not create a Session; the live verifier-backed slice owns one-task/one-Session execution and minted-identity isolation. Pass --json to read the same complete compiled resources as one terminal-safe machine document. A JSON parser reconstructs the exact original values:
This command is distinct from checkfu eval (Blueprint plan evaluation, which judges an authoring project’s plan and cannot authorize promotion). The two instruments are never conflated.

​
Compare harnesses under controlled conditions

checkfu eval experiment <dir> can run the same task set against multiple published AgentDefinitions. Give every lane a stable label, repeat the sample enough times to expose variance, and request the machine report:
Before creating a Session, matrix mode reads each lane’s latest published AgentVersion, resolves its public HarnessProfile, and refuses the run unless every non-harness definition control matches: model routing profile and tier, sandbox profile, tools, instructions, Skills, exposure and compaction policy, output schema, PermissionAssignment scopes, mounts, lifecycle hooks, and the remaining frozen fields. Active AgentDefinition-scoped Budget policies must also match; Workspace- and Principal-scoped Budgets are already common because every sample uses the same Workspace and Principal. Each lane must use a distinct active AgentDefinition and harness profile. A matrix is bounded to 1,000 Sessions (tasks × lanes × repetitions) so a broad directory or typo cannot silently create an unbounded bill. The report does not trust that preflight snapshot after work begins. It reads every run.created.execution_pins and model.routed event and fails control equivalence if a Session used a different AgentVersion, harness profile, model model routing profile revision, realized model/tier, harness driver, SandboxProfile/image, or plane policy. Unpinned and @latest harness references are resolved to the current profile version before the first sample, and that resolved profile and immutable release digest must remain constant across every sample in the lane; both coordinates are included in the report. Per lane it reports pass rate and population variance for iteration count, tokens, active latency, tool calls, retries, tool failures, compactions, and recoveries. Optional cost dimensions also report observed and missing sample counts; missing measurements remain unknown and are never scored as zero. A JSON report is emitted even when a sample fails; the command exits zero only when every Outcome is satisfied and runtime controls remained equivalent. The task directories can mix coding and operational tasks. This experiment currently uses the ordinary Outcome grader described above: it does not claim to run the task’s retained verifier or realize its environment until the live verifier-backed ROADMAP slice lands.

​
Observe one production harness release

Controlled experiments answer whether two harness lanes differ under frozen inputs. After a release has real traffic, checkfu eval observe answers a narrower question: what behavior did one exact harness release exhibit across an explicit set of existing Sessions?
The command reads only the named Sessions through the public event-list API. It accepts at most 500 unique Sessions and 10,000 events per Session, and it never lists an ambient customer population or changes any Session. Every sample must prove the requested exact HarnessProfile and the cohort must resolve to one immutable release digest. Every named Session must contribute a completed turn, and the machine report records its last observed event sequence so the measured append-only prefix is explicit. Recovery counters must be present on every completed turn; an older or third-party harness that omits them is reported as missing evidence, never as zero retries. Without reviewed limits, complete evidence is marked ready for human review. With limits, the report says whether measured retries and overflow recoveries are within them or need review. This is decision support, not an auto-tuner: the command does not mutate a HarnessProfile, AgentDefinition, model route, or harness configuration.

​
How the verifier reward is intended to become a verdict

Verifier-not-judge is the intended posture, matching how Checkfu proves its own claims: the verifier program — not a model judge — will decide whether the work satisfied the task. The platform-free reconciliation mapping below is built and tested, but the live grading chokepoint does not yet run the task verifier or read its reward. Once that ROADMAP slice lands, a settled Run with an active Outcome will reconcile the reward as follows:
  • A reward at or above pass_threshold (default 1.0) settles the Outcome satisfied.
  • A lower reward produces needs_revision, driving the grade-and-revise loop until the iteration budget exhausts, at which point the chokepoint converts the terminal max_iterations_reached.
A reward that cannot be read or is out of range is a retryable grading failure, never a verdict — the same posture as an undecodable model reply. No reward writes an Outcome except through the grading chokepoint (law 7): evaluation enforcement is domain truth on the event surface, not observability.

​
Honest gaps in this version

Two design questions the plan records as open are answered at the honest floor rather than silently:
  • SandboxProfile realization. A task names a base SandboxProfile by image. A task Dockerfile is read, preserved in full in both output modes, and recorded as an unrealized image_build gap — v1 does not content-hash it into a derived image, so the surface says exactly what it can run.
  • Live execution. This surface compiles and validates a task directory into the resources a graded Session needs. The existing eval experiment command can drive those Outcome fields through public Session APIs, but it does not realize the task SandboxProfile or execute the task verifier. The customer-eval ROADMAP item retains ownership of that complete verifier-backed loop: realizing the SandboxProfile, driving one isolated Session against a target, running the verifier in the sandbox, reconciling its reward through the Outcome chokepoint, writing a run manifest under .local/, exporting the format in reverse, and documenting the result. No server-side eval entity exists: an “eval run” is a client-side composition whose results are projections of session events, and retained manifests are local artifacts.