Skip to main content

Eval API

The API reference lists every endpoint with its request and response schema, rendered from the OpenAPI document; this page explains how the calls behave. Eval is in early access and switched on per workspace.

Runs​

An independent MT and LLM bake-off. Give a source (and an optional human reference), then either paste candidate translations or let the server generate them with the engines you pick. The scoreboard ranks candidates by quality and, with prices, by value. Base path /eval/v1.

An engine code is a named engine from the catalog below, di:<model_id> for a DeepInfra model or p:<provider>/<model> for any provider this build supports, so a new model never waits for a release.

POST /eval/v1/runs​

Create and run a bake-off (eval:write). Returns a runId right away; poll the run until it is finalized, or subscribe a webhook to eval.finalized to be told when it is.

Generate mode spends money, so it is capped per workspace: at most 2 generate runs at once and 20 a day. Over either cap the call is a 429 that names the cap it reached. Scoring candidates you send is not capped.

Paste your own candidates​

curl -X POST https://app.stillneedsahuman.com/eval/v1/runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Retail TR product blurbs",
    "target_lang": "tr",
    "source": ["Add to cart", "Free returns"],
    "reference": ["Sepete ekle", "Ucretsiz iade"],
    "candidates": [
      {"name": "Engine A", "targets": ["Sepete ekle", "Bedava iade"], "cost_per_1m": 2},
      {"name": "Engine B", "targets": ["Sepete koy", "Ucretsiz iade"], "cost_per_1m": 5}
    ]
  }'

Generate with engines​

curl -X POST https://app.stillneedsahuman.com/eval/v1/runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Engine bake-off",
    "target_lang": "tr",
    "source": ["Add to cart", "Free returns"],
    "reference": ["Sepete ekle", "Ucretsiz iade"],
    "generate": true,
    "engines": ["gemini_flash", "deepseek_v4", "qwen_2_5_72b"],
    "budget_per_1m": 5
  }'
{ "runId": "ev_5b2c...", "status": "queued" }

GET /eval/v1/runs/{id}​

Poll the run (eval:read): each engine with its scores, cost and latency. Once finalized, rollup holds the ranked candidates and recommendation names the best quality and best value picks. A run of another workspace is a 404.

curl https://app.stillneedsahuman.com/eval/v1/runs/ev_5b2c... \
  -H "Authorization: Bearer YOUR_TOKEN"
{
  "runId": "ev_5b2c...",
  "status": "finalized",
  "rollup": [
    {"rank": 1, "name": "Gemini 3.1 Flash", "quality": 94.2, "chrfpp": 71.0,
     "ter": 28.4, "integrity": "pass", "cost_per_1m": 0.10, "latency_ms": 880},
    {"rank": 2, "name": "DeepSeek V4", "quality": 92.5, "cost_per_1m": 0.50}
  ],
  "recommendation": {
    "best_quality": "Gemini 3.1 Flash", "best_quality_score": 94.2,
    "best_value": "Gemini 3.1 Flash", "best_value_score": 94.2, "best_value_cost": 0.10
  }
}

Each rollup row names the engine (name), its rank, quality (the score the ranking uses), verdict, integrity, findings, cost_per_1m and latency_ms; when the run has a reference it also has chrfpp and ter. ter is the share of words a post-editor would change to reach your reference, as a percentage, lower is better; present only when the run has a reference and never for the human baseline. It counts the fewest word insertions, deletions, substitutions and moves of a word group that turn the candidate into the reference, over the reference's word count, summed across the segments. Words are split on spaces, case is ignored and punctuation stays on its word, so 0 is identical to the reference and a long candidate can go over 100. It is evidence beside quality and never changes the ranking.

FieldNotes
sourceArray of strings, or objects with a source field. Required.
referenceOptional human translation; enables the chrF++ reference-match score and TER.
candidates or generate+enginesPaste candidates, or generate them server-side with the listed engines.
cost_per_1m / budget_per_1mUSD per 1M tokens. Drives the best-value pick. Generated runs use indicative list prices.
template, accountOptional. Resolve the scoring template (quality bar, severities) by id or by account.
target_lang, source_lang, localeThe language pair, and a locale where the target needs one. target_lang is required.
name, division, profile, content_typeOptional. The run's title and where it belongs.
dropbox_pathOptional. Take the source sample from a file in the workspace's Dropbox.
gemba_mqm, gemba_samplesOptional. Score with an MQM-style AI judge, and how many samples it reads.
note

List past runs with GET /eval/v1/runs?limit=&offset= (limit 50 when left out) and download a finalized run as Excel with GET /eval/v1/runs/{id}/export.xlsx (Summary, Scoreboard and Comparison sheets; the Scoreboard has a TER column after chrF++, empty where there was no reference).

Every finalized run of a date range comes as one CSV with GET /eval/v1/export.csv?from=&to= (eval:read; the dates as YYYY-MM-DD on the day a run was finalized, the last 12 months when left out). One row per candidate: run_id, name, created_at, finalized_at, status, account, content_type, source_lang, target_lang, template_id, template_version, engine, rank, quality, verdict, chrfpp, ter, integrity, findings, words, latency_ms, is_baseline and error. The file has the byte order mark and the UTC times of the LQA CSV and counts against the same export rate.

GET /eval/v1/runs/{id}/comparison​

The reviewer's view of a run: every source segment beside every engine's output, with the integrity check and the MQM findings of each cell.

The catalog and the metrics​

What a run can be built from and judged by. All need eval:read.

EndpointReturns
GET /eval/v1/enginesThe engines available in this build.
GET /eval/v1/modelsThe model catalog by vendor. A help for a picker, not a list of what is allowed: an engine code outside it still runs.
GET /eval/v1/metricsThe metric registry, and whether the neural metrics are available here.
GET /eval/v1/analyticsHow each engine did across your finalized runs: average quality, win rate, runs counted, average cost and latency. avg_ter is the average TER over its candidates scored against a reference (null when none was).
POST /eval/v1/source/uploadA multipart file (XLIFF, Excel, CSV, a TMS export) turned into a source sample. For Excel and CSV the answer carries the columns and a preview, so the caller can say which column is source and which target.
POST /eval/v1/calibrateHow a metric's scores per segment follow human MQM scores. Body: {"metric_scores": [...], "human_scores": [...]}. It makes no run and spends nothing.
GET /eval/v1/dropbox/listThe workspace's Dropbox folders, to pick a sample from. Limited to your own workspace's folder.
GET /eval/v1/healthLiveness, with no key.