REST API
Use Authorization: Bearer <tenant-key> (or x-api-key) for /v1 resources. Discovery routes are public. Admin routes require the separate VERDICT_ADMIN_KEY and a database-backed key manager. Requests below use JSON; rubric create/replace/validate also accept YAML with Content-Type: application/yaml.
Retries and errors
Send Idempotency-Key (1–255 characters) on evaluation or batch creation. A matching retry within the tenant returns HTTP 200 with Idempotency-Replayed: true and stored results, without judging again. A different body with the same key returns 409. Keys expire after 24 hours by default. async and includeEvidence do not change request identity. Async replays return the current { id, status, url } summary.
POST /v1/evaluations
Authorization: Bearer your-key
Content-Type: application/json
Idempotency-Key: review-123
{"rubric":"rub_ID","inputs":[{"type":"text","text":"Evidence"}]}Errors use this envelope (validation errors may also include issues):
{"error":{"type":"not_found","message":"evaluation eval_ID not found"}}Types are validation, not_found, idempotency_conflict, http, and internal. Project routes add already_exists, in_use, unknown_version, unknown_run, pointer_conflict, nothing_to_roll_back, and — when the server is set up to refuse some requests — forbidden, limit_exceeded, approval_required (HTTP 403). A failed synchronous evaluation can return HTTP 422 with an evaluation document rather than an error envelope. Rubric validation returns HTTP 200 with valid: false for invalid rubrics.
Routes and examples
Response examples below are abbreviated projections of JSON documents; omitted fields remain part of the API. IDs are placeholders. Evaluation documents include id, status, subject, a rubric summary, evidence metadata, criterion results, ratings, pending tasks and aggregate. Evidence text is omitted unless requested. List and batch responses omit individual ratings.
GET /
Service discovery.
GET /Response: HTTP 200
{
"name": "verdict",
"version": "0.1.0",
"docs": "/v1/openapi.json"
}GET /v1/health
Health check.
GET /v1/healthResponse: HTTP 200
{
"ok": true
}GET /v1/openapi.json
Discovery specification; currently a stub, not the full route contract.
GET /v1/openapi.jsonResponse: HTTP 200
{
"openapi": "3.1.0"
}POST /v1/rubrics
Create a rubric; defaults are filled and references validated.
POST /v1/rubrics
Content-Type: application/json
{"name": "Presence", "criteria": [{"id": "present", "name": "Has text", "judge": {"type": "benchmark", "method": "length", "min": 1}}]}Response: HTTP 201
{
"id": "rub_ID",
"name": "Presence",
"criteria": [
{
"id": "present",
"name": "Has text",
"judge": {
"type": "benchmark",
"method": "length",
"min": 1
}
}
]
}POST /v1/rubrics/validate
Validate without storing. Invalid response: { "valid": false, "error": "..." } (error can be an issues array).
POST /v1/rubrics/validate
Content-Type: application/json
{"name": "Presence", "criteria": [{"id": "present", "name": "Has text", "judge": {"type": "benchmark", "method": "length", "min": 1}}]}Response: HTTP 200
{
"valid": true,
"rubric": {
"name": "Presence",
"criteria": [
{
"id": "present",
"name": "Has text",
"judge": {
"type": "benchmark",
"method": "length",
"min": 1
}
}
]
}
}GET /v1/rubrics?limit=100
List tenant rubrics.
GET /v1/rubrics?limit=100Response: HTTP 200
{
"data": [
{
"id": "rub_ID",
"name": "Presence",
"criteria": [
{
"id": "present",
"name": "Has text",
"judge": {
"type": "benchmark",
"method": "length",
"min": 1
}
}
]
}
]
}GET /v1/rubrics/rub_ID
Read a rubric; unknown ID returns 404.
GET /v1/rubrics/rub_IDResponse: HTTP 200
{
"id": "rub_ID",
"name": "Presence",
"criteria": [
{
"id": "present",
"name": "Has text",
"judge": {
"type": "benchmark",
"method": "length",
"min": 1
}
}
]
}PUT /v1/rubrics/rub_ID
Replace a rubric. Existing evaluation snapshots stay frozen.
PUT /v1/rubrics/rub_ID
Content-Type: application/json
{"name": "Presence", "criteria": [{"id": "present", "name": "Has text", "judge": {"type": "benchmark", "method": "length", "min": 1}}]}Response: HTTP 200
{
"id": "rub_ID",
"name": "Presence",
"criteria": [
{
"id": "present",
"name": "Has text",
"judge": {
"type": "benchmark",
"method": "length",
"min": 1
}
}
]
}DELETE /v1/rubrics/rub_ID
Delete; unknown ID returns 404. Response has no body.
DELETE /v1/rubrics/rub_IDResponse: HTTP 204.
GET /v1/rubrics/rub_ID/calibration?target=0.8&buckets=5&limit=1000
Compare stored decision ratings with human consensus. Target is 0..1; buckets 1..1000; limit is a positive integer. See the report type for metrics and null semantics.
GET /v1/rubrics/rub_ID/calibration?target=0.8&buckets=5&limit=1000Response: HTTP 200
{
"suggestedMinConfidence": null
}POST /v1/evaluations
rubric accepts an ID or inline object. Provide inputs or subject: { name, inputs }; optional fields: name, metadata, webhook, includeEvidence, async. With async: true, a fresh request returns 202 { "id": "eval_ID", "status": "queued", "url": "/v1/evaluations/eval_ID" }. Poll the URL. Sync failures return 422; idempotent replays return 200.
POST /v1/evaluations
Content-Type: application/json
{"rubric": "rub_ID", "inputs": [{"type": "text", "text": "Evidence"}]}Response: HTTP 201
{
"id": "eval_ID",
"status": "completed",
"aggregate": {
"score": 100,
"partial": false
},
"criteria": [
{
"criterionId": "present",
"score": 10,
"normalized": 1,
"ratings": []
}
]
}GET /v1/evaluations?rubric=rub_ID&status=completed&limit=50
List tenant evaluations with optional rubric/status filters. Individual ratings are omitted.
GET /v1/evaluations?rubric=rub_ID&status=completed&limit=50Response: HTTP 200
{
"data": [
{
"id": "eval_ID",
"status": "completed",
"aggregate": {
"score": 100,
"partial": false
},
"criteria": [
{
"criterionId": "present",
"score": 10,
"normalized": 1,
"ratings": []
}
]
}
]
}GET /v1/evaluations/eval_ID?evidence=true
Fetch current status and ratings; evidence=true includes extracted text. Unknown ID returns 404.
GET /v1/evaluations/eval_ID?evidence=trueResponse: HTTP 200
{"id":"eval_ID","status":"completed","aggregate":{"score":100,"partial":false}}POST /v1/evaluations/eval_ID/retry
Retry failed work and reset attempts. Unknown ID: 404; non-failed evaluation: 409.
POST /v1/evaluations/eval_ID/retryResponse: HTTP 202
{"id":"eval_ID","status":"completed","aggregate":{"score":100,"partial":false}}POST /v1/evaluations/eval_ID/scores
Submit a human score as a number, scale label, or choice ID. Optional confidence is 0..1. Returns the updated evaluation.
POST /v1/evaluations/eval_ID/scores
Content-Type: application/json
{"criterionId": "team", "raterId": "reviewer", "score": 4, "rationale": "Relevant experience"}Response: HTTP 200
{"id":"eval_ID","status":"completed","aggregate":{"score":100,"partial":false}}POST /v1/evaluations/eval_ID/aggregate
Reaggregate stored ratings without judges. persist defaults to false.
POST /v1/evaluations/eval_ID/aggregate
Content-Type: application/json
{"aggregation": {"method": "min"}, "persist": false}Response: HTTP 200
{"id":"eval_ID","status":"completed","aggregate":{"score":100,"partial":false}}GET /v1/tasks?rater=reviewer
Pending human inbox. Optional rater filter includes allowed generic slots. Scans up to 500 pending evaluations.
GET /v1/tasks?rater=reviewerResponse: HTTP 200
{
"data": [
{
"evaluationId": "eval_ID",
"criterionId": "team",
"raterId": "reviewer",
"rubric": "Review"
}
]
}POST /v1/batches
Evaluate multiple subjects using one rubric. Optional metadata/webhook; supports Idempotency-Key and returns 200 on replay.
POST /v1/batches
Content-Type: application/json
{"rubric": "rub_ID", "subjects": [{"name": "Acme", "inputs": [{"type": "text", "text": "Evidence"}]}]}Response: HTTP 201
{
"ranking": [
{
"rank": 1,
"evaluationId": "eval_ID",
"subject": "Acme",
"score": 100,
"status": "completed"
}
],
"evaluations": [
{
"id": "eval_ID",
"status": "completed",
"aggregate": {
"score": 100,
"partial": false
},
"criteria": [
{
"criterionId": "present",
"score": 10,
"normalized": 1,
"ratings": []
}
]
}
]
}POST /v1/admin/keys
Use the admin bearer token. The raw key is returned once; store it securely.
POST /v1/admin/keys
Content-Type: application/json
{"tenantId": "acme", "name": "service"}Response: HTTP 201
{
"id": "key_ID",
"key": "vk_live_SECRET",
"prefix": "vk_live_..."
}GET /v1/admin/keys?tenant=acme
Admin bearer token; tenant query is required. Raw keys are never listed.
GET /v1/admin/keys?tenant=acmeResponse: HTTP 200
{
"data": [
{
"id": "key_ID",
"name": "service",
"prefix": "vk_live_...",
"createdAt": "2026-11-05T00:00:00.000Z",
"lastUsedAt": null,
"revokedAt": null
}
]
}DELETE /v1/admin/keys/key_ID
Revoke a key using the admin bearer token. Unknown ID returns 404. Response has no body.
DELETE /v1/admin/keys/key_IDResponse: HTTP 204.
Where next
Use the TypeScript SDK for typed requests and polling, connect an agent through MCP, or deploy a persistent API before storing production evaluations.
Publishing to the hub
Signing in: GET /v1/hub/config · POST /v1/hub/login
Neither needs a key. config returns { "github": { "clientId": "…" } | null, "publishing": "github" | "tenant" }. A client runs GitHub's device flow with that client id (empty scope) and posts the token:
POST /v1/hub/login
Content-Type: application/json
{"githubToken": "gho_…"}HTTP 201 { "apiKey": "vk_live_…", "keyId": "key_…", "handle": "alice", "tenantId": "gh:42" }. The hub asks GitHub whose token it is — through POST /applications/{client_id}/token when GITHUB_CLIENT_SECRET is set, so tokens of other apps are refused, else GET /user — and keeps only the account id and login. 401 for a token GitHub does not accept, 502 when GitHub cannot be reached, 501 when sign-in is not configured. With sign-in configured, only signed-in accounts publish, under their login; a ref (owner/slug) stays with the account that first published under it. GET /v1/hub/me says who a key publishes as; DELETE /v1/hub/keys/:id revokes one of the account's keys.
A publication is a copy of exactly what an author chose to make public: one rubric version, the cases marked visibility: public, and the numbers of chosen runs. Public reads of the hub (below) read only publications, without the publishing tenant; nothing else on the server becomes public.
POST /v1/publications/preview
Builds the publication without storing it. Always HTTP 200: ok, issues (each with a path), warnings (personal-looking data, unreviewed labels) and, when ok, the publication that POST /v1/publications would store.
POST /v1/publications/preview
Content-Type: application/json
{"slug": "agent-intent-ko", "license": "CC-BY-4.0",
"version": {"id": "rv_…", "rubric": {"name": "…", "criteria": [{"id": "wants_agent", "name": "…", "scale": {"type": "binary"}, "threshold": 0.6}]}},
"cases": [{"id": "c1", "name": "explicit request", "inputs": [{"type": "text", "text": "상담원 연결해 주세요."}], "expected": {"wants_agent": "yes"}, "visibility": "public", "labelSource": "human"}],
"runs": [{"run": {"…": "a stored run"}, "report": {"…": "its agreement report"}}]}Rules: version.id must be what the rubric freezes to; exactly one criterion; licence one of CC-BY-4.0, CC-BY-SA-4.0, CC0-1.0, MIT, Apache-2.0; yes/no criteria need threshold; no judge baseUrl; cases must be visibility: public with text, json or url inputs; runs must be of the published version. Runs keep only their fingerprint, counts and agreement numbers (origin: "author_reported"); case ids, inputs and outputs are dropped. A published parent version is linked; an unpublished one is shown only as private.
POST /v1/publications
Same body. HTTP 201 { publication, warnings, created: true }; 200 with created: false when the same owner/slug version is already published; 400 { error: { type: "validation", issues } }; 409 when that version was withdrawn; 413 over 1 MB. On a server without GitHub sign-in, a tenant publishes under its tenant id as owner.
GET /v1/publications · DELETE /v1/publications/:id
List the caller's publications (withdrawn included); withdraw one (204; 404 for another tenant's). Withdrawal removes it from search; copies others already downloaded or forked are not recalled.
Reading the hub
No key needed. These routes read only publications and never return the publishing tenant.
GET /v1/public/rubrics
Search: q (title, owner, slug, domain; case-insensitive), domain, locale, type (score, noul, choice), owner, limit (1–50, default 50). One result per card (owner/slug) at its newest live version, newest first:
{"data": [{"ref": "alice/agent-intent", "owner": "alice", "slug": "agent-intent", "title": "…", "judgmentType": "noul", "license": "CC-BY-4.0",
"latest": {"publicationId": "pub_…", "versionId": "rv_…", "createdAt": "…"}, "cases": 12, "runs": 1,
"parent": {"public": true, "ref": "support/agent-intent-ko", "versionId": "rv_…", "publicationId": "pub_…"}}]}GET /v1/public/rubrics/:owner/:slug
{ publication, versions }: the card at ?version=rv_… (default: its newest live version) and every version it has had, newest first, withdrawn ones with withdrawnAt. 404 for an unknown card or version; 410 { error: { type: "withdrawn", message, withdrawnAt } } for a withdrawn version.
GET /v1/public/publications/:id
One publication by id; 404 / 410 as above.
Pages: /hub and /hub/:owner/:slug
The same reads as HTML: a search form and a card page with the question, its answers and threshold, lineage (a published parent is linked, an unpublished one only mentioned), reported runs (author-reported numbers, whether anyone can repeat them), public cases, versions and the rubric YAML. Each card shows copyable verdict pull and verdict fork commands. Pages carry no scripts and a strict Content-Security-Policy; everything authors wrote is escaped.
Hub operations
Limits
Sign-in (POST /v1/hub/login) is limited per client address, publishing (POST /v1/publications and its preview) per account, and the public reads and pages per client address — by default 10, 30 and 300 requests a minute. Over the limit the answer is HTTP 429 { "error": { "type": "rate_limited", … } } with Retry-After: 60. On Cloudflare, bind HUB_LOGIN_LIMITER, HUB_PUBLISH_LIMITER and HUB_READ_LIMITER (Workers Rate Limiting) to share counts across isolates; without them each isolate counts on its own. The client address is Cloudflare's cf-connecting-ip on Workers and the socket address on the Node server (behind a proxy, limit at the proxy too).
Takedowns: /v1/admin/publications
With VERDICT_ADMIN_KEY. GET /v1/admin/publications?q=&owner=&slug=&limit= lists every publication, withdrawn included, with its tenant. POST /v1/admin/publications/:id/takedown with { "reason": "…" } withdraws it and records the takedown; public reads then answer 410 { "error": { "type": "taken_down", "message": "alice/agent at rv_… was taken down by the hub's operators on 2026-11-09: …" } }, and the author sees the takedown in GET /v1/publications. POST /v1/admin/publications/:id/restore puts back what the operators took down; a publication its author withdrew stays withdrawn (409).
Projects, environments and versions
A project keeps immutable rubric versions under the names services ask for, and environments that say which version of each rubric runs. The server stores rubric documents and pointer changes only: no cases, inputs or outputs. Project, environment and rubric names are lowercase letters, digits and dashes (up to 63 characters).
Projects: /v1/projects
POST { "name": "support-bot" } creates a project (201; 409 already_exists). GET lists this tenant's projects. DELETE /v1/projects/support-bot with { "confirm": "support-bot" } deletes the project and everything in it.
Environments: /v1/projects/:project/environments
POST { "name": "prod" } creates an environment; GET lists them. GET …/environments/prod returns the environment with its pointers:
{"project":"support-bot","name":"prod","createdAt":"…","pointers":[{"name":"intent","versionId":"rv_…","updatedAt":"…"}]}GET …/environments/prod?include=rubrics adds the rubric document to each pointer: everything a service needs to run, in one answer. Both forms answer with an ETag; send it back as If-None-Match and an environment whose pointers did not change answers 304 with no body. Versions never change, so the pointers alone identify the answer.
DELETE …/environments/prod with { "confirm": "prod" } removes the environment and its pointers. Its history stays in the project.
Versions: /v1/projects/:project/rubrics/:name/versions
POST { id, rubric, parentVersionId?, fork? } stores a version under :name. The server freezes the rubric again and answers 400 when it does not freeze to id. 201 for a new version, 200 with the stored one when it is already there, 413 over 1 MB. GET lists a rubric's versions newest first without the rubric documents; GET …/versions/rv_… returns one with its rubric. DELETE answers 409 in_use while an environment points at the version. GET /v1/projects/:project/rubrics lists rubric names with version counts.
Promotions: POST /v1/projects/:project/environments/:env/promotions
{"changes":[{"name":"intent","versionId":"rv_NEW","expected":"rv_OLD"}],"note":"fewer wrong hand-offs"}Each change names the version you saw the pointer at (expected; null when there was none). All changes of a request (at most 20, each rubric once) apply or none do. versionId: null removes a pointer. A version can be promoted only under the name it was uploaded as (400 unknown_version). When a pointer is not where you saw it, nothing is applied and the answer says where the pointers are:
{"error":{"type":"pointer_conflict","message":"prod changed since you read it; nothing was applied","current":[{"name":"intent","versionId":"rv_OTHER"}]}}Rollbacks: POST /v1/projects/:project/environments/:env/rollbacks
{ "name": "intent", "expected": "rv_NEW", "note"?: "…" } puts the pointer back where its latest change found it; rolling back again returns to where it was. 409 nothing_to_roll_back when no earlier version is on record (the first promotion, or an earlier version that was deleted).
History: GET /v1/projects/:project/history?environment=&name=&limit=
Pointer changes, newest first: { id, environment, name, from, to, kind, note?, actor?, at } with kind one of promote, rollback, unset. actor is the API key that made the change (its id and name when keys are stored in a database). History cannot be edited or deleted through the API.
Run reports: /v1/projects/:project/reports/runs
POST stores the numbers of a run made elsewhere (your CI). The request has no field for a case id, an evaluation id, an input or an output; fields it does not know are dropped.
{"name":"intent","run":{"id":"run_…","rubricVersionId":"rv_…","datasetVersionId":"ds_…","status":"completed","startedAt":"…","fingerprint":{"judgmentHash":"…","policyHash":"…","bindingHash":"…","questionsHash":"…","adapterVersion":"…","casesHash":"…","labelsHash":"…","models":[],"scoring":"agreement@1"},"counts":{"cases":40,"completed":40,"pendingHuman":0,"failed":0,"invalidInput":0,"labelled":40},"usage":{"decisionCalls":40,"llmCalls":0,"inputTokens":9200,"outputTokens":0}},"report":{"decided":40,"agreed":37,"criteria":{"wants":{"n":40,"agreed":37,"askedHuman":0}}},"datasetName":"cases.jsonl","source":{"commit":"…","branch":"…","url":"…"},"version":{"id":"rv_…","rubric":{}}}version uploads the run's rubric version under name when the server does not have it; without it an unknown version answers 400 unknown_version. The run must name its version's judgment, policy and binding hashes. 201 for a new report, 200 with the stored one when that run was already reported. Every report is kept as origin: "author_reported": the server cannot repeat the run. GET …/reports/runs?name=&versionId=&dataset=&limit= lists reports newest run first; GET …/reports/runs/run_… returns one.
Comparison reports: /v1/projects/:project/reports/comparisons
POST { name, comparison, source? } stores the counts of a comparison between two reported runs: its kind, the conditions that differed, and per criterion the class counts and both agreement rates. Per-case rows are not part of the request. Both runs must have been reported first (400 unknown_run). The same two runs against the same labels are one comparison (200 on repeat). GET …/reports/comparisons?name=&head=&limit= lists them newest first.