<!--
  Machine-readable mirror of the Signal API docs (the /docs page).
  Served statically so AI agents, crawlers, and `curl` can read the docs
  without executing the client-side React app. Keep in sync with
  src/pages/DocsPage.tsx when the docs change.
-->

# Signal API documentation

Signal, by Josh Talks. Download your Signal validation set, score your ASR model,
and get OI-WER (Orthographically Informed Word Error Rate) back: overall,
per-language, and per-utterance.

Two ways to integrate:

1. **Python client (`signaljt`)**: recommended. One `pip install`, resumable
   dataset downloads, and one-line scoring that drops into wandb / TensorBoard.
2. **REST API**: three plain HTTP endpoints with API-key auth. Call from any
   stack: curl, Go, JS, Rust.

---

## Overview

Signal is the developer surface for your custom validation set: the focused set
built around your model's failure modes. The workflow is the same whichever
integration you pick:

1. Fetch the ground-truth audio for one or more languages.
2. Run your ASR model over the audio to produce predictions `{index: transcript}`.
3. Submit the predictions and receive an `eval_id`.
4. Poll for the result: overall, per-language, and per-utterance OI-WER.

The Python client wraps all of this (downloads, caching, upload, polling, result
parsing). The REST API exposes the same three operations directly.

## Build with an AI coding agent

These docs are published as agent-readable Markdown, so you can let a coding
agent (Claude Code, Cursor, Copilot, and similar) write the integration for you.
Point it at https://signal.joshtalks.com/llms.txt (a short index) or the full
https://signal.joshtalks.com/docs.md, then paste a prompt like these.

Python client:

```text
Read https://signal.joshtalks.com/docs.md (the Signal ASR eval docs).
Using the signaljt Python client, write a script that:
  1. logs in with the API key in the SIGNALJT_API_KEY env var,
  2. downloads the Hindi ("hi") validation set,
  3. runs my ASR model, assuming a function transcribe(audio_path) -> str,
  4. submits the predictions and prints the overall + per-language OI-WER.
Handle auth/download errors using signaljt's exception types.
```

REST API (any language):

```text
Read https://signal.joshtalks.com/docs.md and use the Signal REST API.
Write a Bash script (curl + jq) that:
  1. fetches and unzips the Tamil ("ta") ground-truth audios,
  2. submits predictions.json to /api/eval/submit/,
  3. polls /api/eval/result/ until status is "completed",
  4. downloads eval_result.json and prints .overall.weighted_oiwer.
The API key is in the $SIGNAL_KEY env var. Use the X-API-Key header.
```

Most agents fetch a URL directly. If yours can't browse the web, download
`/docs.md` and add it to your repo (or paste it into the agent's context). It is
self-contained.

## API keys & authentication

Every request authenticates with a long-lived API key. Create and manage keys
from the Access keys page in the dashboard (`/api-keys`). Each key has an expiry
you choose (1h / 1d / 2d / 7d / never).

Pass it in the `X-API-Key` header on every call:

```http
X-API-Key: f3c9aa1201.MTc4...
```

An expired or invalid key returns `401` with a clear message. Create a fresh key
on the dashboard and swap it in.

## Supported languages

15 Indic languages. Use the ISO-style code as the `language_code` everywhere.

| Code | Language | Code | Language | Code | Language |
|---|---|---|---|---|---|
| `hi` | Hindi | `bn` | Bengali | `ta` | Tamil |
| `te` | Telugu | `mr` | Marathi | `gu` | Gujarati |
| `kn` | Kannada | `ml` | Malayalam | `pa` | Punjabi |
| `as` | Assamese | `or` | Odia | `mai` | Maithili |
| `bho` | Bhojpuri | `hne` | Chattisgarhi | `ur` | Urdu |

## How it works

Scoring is an asynchronous job: submit predictions, the server scores them
against ground truth, then you fetch the result.

```text
fetch-audios  ->  run your model  ->  submit  ->  poll result  ->  download eval_result.json
   (per lang)      (predictions)     (eval_id)   (until "completed")   (overall + per-language + per-chunk)
```

---

# Python client (`signaljt`)

## Install

Python 3.8+. Runtime deps (`httpx`, `filelock`) install automatically.

```bash
pip install signaljt

# with the Jupyter/Colab login widget
pip install "signaljt[notebook]"
```

With [uv](https://docs.astral.sh/uv/):

```bash
uv pip install signaljt
uv pip install "signaljt[notebook]"

# or add it to a uv-managed project
uv add signaljt
```

## Quickstart (5 minutes)

```python
import signaljt

# 1. Authenticate once (or set SIGNALJT_API_KEY)
signaljt.login("your-api-key")

# 2. Download a language's ground-truth audio
ds = signaljt.fetch_dataset("hi")            # Hindi
print(len(ds), "utterances")
print(ds[0].index, ds[0].audio_path)         # absolute path, ready to load

# 3. Run YOUR model over the audio -> {index: transcript}
predictions = {item.index: my_asr_model(item.audio_path) for item in ds}

# 4. Score it, overall + per-language + per-chunk OI-WER
result = signaljt.evaluate(predictions, eval_name="my-model-v1")
print("OI-WER:", result.oiwer)               # e.g. 0.1033
print("Hindi:", result.per_language["hi"].oiwer)

# 5. (optional) log to your experiment tracker
import wandb; wandb.log(result.metrics())
```

The same flow on the command line:

```bash
signaljt login
signaljt download hi
# ... produce predictions.json ...
signaljt eval predictions.json               # prints the signed result URL
```

## Download datasets

`fetch_dataset` fetches the signed URL, downloads, verifies, extracts, and
returns an object with absolute audio paths. It's cached, so the second call
(any process or node) is instant.

### One language, many, or all

```python
ds   = signaljt.fetch_dataset("hi")            # -> a single Dataset
data = signaljt.fetch_dataset(["hi", "bn"])    # -> {"hi": Dataset, "bn": Dataset}
data = signaljt.fetch_dataset("all")           # -> all 15 languages, as a dict
```

Multiple languages download concurrently (bounded by `max_parallel`, default 4);
each keeps its own cache/lock/resume.

### The Dataset object

```python
ds = signaljt.fetch_dataset("bn")

len(ds)                    # number of utterances
ds.language_code           # "bn"
ds.root                    # directory containing the extracted files

for item in ds:
    item.index             # utterance id (str)
    item.audio_path        # ABSOLUTE path to the .flac
    item.extra             # dict of any extra manifest columns

ds[0]                      # by position -> DatasetItem
ds["100825"]               # by index    -> DatasetItem
"100825" in ds             # membership test
ds.indices                 # ["100825", ...]
ds.paths()                 # {"100825": "/abs/.../100825.flac", ...}
```

Paths are computed at load time from wherever the data actually lives, so they
stay correct even if you move the folder. Always go through the `Dataset` object
rather than hand-parsing the manifest.

### Place files where you want

```python
ds = signaljt.fetch_dataset("bn", local_dir="./bn_data")
ds.root            # ./bn_data

# Materialize an already-fetched dataset afterwards
signaljt.fetch_dataset("bn").export("./bn_data", mode="copy")

# For multiple languages, each lands under local_dir/<code>/
signaljt.fetch_dataset("all", local_dir="./data")   # ./data/hi, ./data/bn, ...
```

| `local_dir_mode` | Behavior | Use when |
|---|---|---|
| `auto` | hardlink if same filesystem (instant), else copy | almost always (default) |
| `copy` | a real independent copy | you'll move / rsync / edit it |
| `symlink` | symlink back into the cache | smallest footprint |
| `hardlink` | force a hardlink | guarantee no copy |

### Caching & performance

```python
signaljt.fetch_dataset("bn")                 # instant if cached
signaljt.fetch_dataset("bn", refresh=True)   # re-download only if the remote zip changed
signaljt.fetch_dataset("bn", force=True)     # always re-download

# High-latency link? parallelize:
signaljt.fetch_dataset("bn", connections=8, extract_workers=4, retries=6)
```

Downloads are integrity-checked (MD5/size), resumable (segment-granular, safe
under Slurm preemption), and version-aware.

## Using with ML frameworks

A `Dataset` is just an index -> absolute audio path mapping, so it drops into any
stack. The core eval loop needs no adapter:

```python
# transformers: pipelines accept a file path directly
from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3")
preds = {it.index: asr(it.audio_path)["text"] for it in ds}
result = signaljt.evaluate(preds)
```

Thin adapters when you want a framework-native object (no extra deps in the
package):

```python
# PyTorch DataLoader
import torch, torchaudio
class TorchDS(torch.utils.data.Dataset):
    def __init__(self, ds): self.items = ds.items
    def __len__(self): return len(self.items)
    def __getitem__(self, i):
        wav, sr = torchaudio.load(self.items[i].audio_path)
        return self.items[i].index, wav, sr
loader = torch.utils.data.DataLoader(TorchDS(ds), batch_size=8, collate_fn=list)

# HuggingFace datasets (lazy audio decoding)
import datasets
hf = datasets.Dataset.from_dict(
    {"index": ds.indices, "audio": [it.audio_path for it in ds]}
).cast_column("audio", datasets.Audio())

# fairseq / wav2vec TSV manifest
import soundfile as sf, os
with open("eval.tsv", "w") as f:
    print(ds.root, file=f)
    for it in ds:
        print("%s\t%d" % (os.path.relpath(it.audio_path, ds.root),
                          sf.info(it.audio_path).frames), file=f)
```

## Evaluate (OI-WER scoring)

`evaluate()` submits, blocks until scoring finishes, and returns a parsed result
object.

```python
result = signaljt.evaluate("predictions.json", eval_name="conformer-ckpt3800")
print(result.oiwer)                 # weighted overall OI-WER
```

### Prediction input formats

```python
signaljt.evaluate("preds.json")                          # JSON file path
signaljt.evaluate("preds.jsonl")                         # JSONL file (streamed on upload)
signaljt.evaluate(predictions_url="https://.../p.json")  # a public/GCS URL (no upload)
signaljt.evaluate({"100000": "predicted text", ...})     # in-memory dict {index: hypothesis}
signaljt.evaluate([{"index": "100000", "hypothesis": "..."}, ...])  # in-memory list
```

The prediction set may span any subset of languages; the backend matches each
index to its language's ground truth.

### Non-blocking submit / poll

```python
job = signaljt.submit("predictions.json", eval_name="run-42")
job.eval_id                      # save this
job.status                       # "queued"

# ... later, same or different process ...
result = job.wait()                                    # block until completed
result = signaljt.get_result(job.eval_id, wait=True)   # by id, from anywhere
```

### Logging to wandb / TensorBoard

```python
metrics = result.metrics()
# {"oiwer/weighted": 0.1033, "oiwer/macro": 0.103, "words/errors": 18417,
#  "oiwer_by_language/hi": 0.104, "oiwer_by_language/bn": 0.104, ...}

import wandb; wandb.log(result.metrics())               # Weights & Biases
for name, value in result.metrics().items():            # TensorBoard
    writer.add_scalar(name, value, step)
```

## Result object & schema

Everything is reachable by attribute (`.`) or key (`[...]`); nested dicts are
wrapped lazily.

```python
r = signaljt.evaluate("predictions.json")

r.oiwer                          # shortcut for r.overall.weighted_oiwer
r.eval_id                        # server eval id
r.result_url                     # signed URL to the full result JSON (~2 days)

# Overall
r.overall.weighted_oiwer
r.overall.macro_oiwer
r.overall.word.errors            # errors / count / sub / ins / del
r.overall.total_utterances

# Per language (keyed by ISO code)
r.per_language["hi"].oiwer
r.per_language.hi.coverage.scored, r.per_language.hi.coverage.missing

# Per utterance (lazy list)
c = r.per_chunk[0]
c.index, c.language, c.duration
c.alignment.hyp                  # ["word1", "word2", ...]
c.alignment.ops                  # ["c", "s", "i", "d", ...]
c.tags                           # ["correct"], ["wrong_script"], ...

# Export
r.to_dict()                      # the raw result dict
r.save("eval_result.json")
```

### Result schema

| Section | Fields |
|---|---|
| `overall` | `weighted_oiwer`, `macro_oiwer`, `weighted_by`, `word{errors,count,sub,ins,del}`, `total_utterances`, `total_duration_sec`, `languages_scored`, `extra_predictions` |
| `per_language[iso]` | `iso`, `name`, `oiwer`, `word{...}`, `utterances`, `coverage{scored,missing,extra}` |
| `per_chunk[i]` | `index`, `language`, `duration`, `cps`, `p808_mos`, `native_district`, `native_state`, `gender`, `speaker_id`, `predicted`, `status`, `word{...}`, `alignment{hyp[],ops[]}`, `tags[]` |
| `meta` | `scored_at`, `audio_version`, `lattice_version`, `penalize_missing`, `tag_rows`, `strip_bracketed` |

`alignment.ops` codes: `c` = correct, `s` = substitution, `i` = insertion,
`d` = deletion.

## Scoring options

| Option | Default | Effect |
|---|---|---|
| `penalize_missing` | `True` | Count ground-truth utterances you did NOT predict as full errors. Set False to score only what you predicted (missing ones are reported in coverage but not penalized). |
| `tag_rows` | `True` | Add error-type tags to each per-chunk entry (correct, wrong_script, internal_deletion, ...). |
| `strip_bracketed` | `False` | Strip bracketed tokens like `<noise>` / `[laughter]` from the hypothesis before scoring. |

```python
# "How good is my model only on what it transcribed?"
signaljt.evaluate(preds, penalize_missing=False)

# "How good over the whole benchmark, including gaps?" (default)
signaljt.evaluate(preds, penalize_missing=True)
```

## Distributed training (DDP)

When scoring inside distributed training (say 4 GPUs), the framework runs your
eval/metrics code on every rank, so a naive `evaluate()` submits the dataset N
times, one duplicate entry per epoch. Two flags fix this.

`ddp=True` (recommended): one submission, identical result on every rank. Rank 0
submits; every other rank fetches that same run by `eval_name` (read-only, no new
entry), so all ranks return an identical `EvalResult`:

```python
# inside HuggingFace Trainer's compute_metrics (or any per-rank eval hook)
def compute_metrics(pred):
    predictions = {idx: text for idx, text in zip(indices, decode(pred))}
    result = signaljt.evaluate(
        predictions,
        eval_name=f"my-run-epoch-{epoch}",
        progress=False,
        ddp=True,
    )
    return {"oiwer": result.oiwer}      # identical on every rank; ONE eval submitted
```

Requirements for `eval_name`:

- Identical across ranks: derive it from the epoch/step (it already is the same
  on all ranks).
- Unique per submission, so a re-run doesn't fetch a stale run of the same name.
  Include the epoch, plus a run id if you re-use output dirs:

```python
run_id = os.environ.get("TORCHELASTIC_RUN_ID", "")   # shared across ranks under torchrun/accelerate
eval_name = f"my-run-{run_id}-epoch-{epoch}"
```

Because non-main ranks block until rank 0's eval completes, `ddp=True` also acts
as a natural barrier that keeps the ranks in sync across the eval step.

`main_process_only=True` (lighter): rank 0 submits, others get `None`. No extra
API calls, but you must return matching metric keys yourself so the trainer
doesn't choke on an inconsistent dict across ranks:

```python
result = signaljt.evaluate(predictions, eval_name=name, main_process_only=True)
if result is None:                       # non-main ranks
    return {"oiwer": float("inf")}       # placeholder; non-rank-0 metrics are ignored anyway
return {"oiwer": result.oiwer}
```

Detect the main process yourself with `signaljt.is_main_process()`, which returns
`True` only on rank 0.

Works with any DDP framework. Rank detection uses only the standard `RANK` /
`LOCAL_RANK` environment variables that `torchrun`, `accelerate launch`,
HuggingFace `Trainer`, PyTorch Lightning, and NeMo all set. Multi-node relies on
the global `RANK` (which `torchrun` / `accelerate` export); a launcher that only
sets a per-node `LOCAL_RANK` would submit once per node.

## Command line

```bash
signaljt login                                 # or: export SIGNALJT_API_KEY=...
signaljt languages                             # list the 15 codes
signaljt download hi bn te                     # one or more languages
signaljt download all --connections 8

signaljt eval predictions.json                 # submit + wait, prints result URL
signaljt eval predictions.json --name my-run --no-penalize-missing
signaljt eval predictions.json --no-wait       # submit only, prints eval_id
signaljt result <eval_id> --wait               # fetch an existing eval
```

## Error handling

All exceptions derive from `signaljt.SignaljtError`.

| Exception | Raised when |
|---|---|
| `NotLoggedInError` | no API key found (arg / env / saved token) |
| `AuthenticationError` | the server rejected the key (invalid / expired) |
| `ApiError` | the API returned an unexpected response |
| `DownloadError` | a transfer failed after retries, or a checksum mismatch |
| `ManifestError` | the dataset manifest was missing or malformed |

```python
import signaljt
try:
    ds = signaljt.fetch_dataset("bn")
except signaljt.NotLoggedInError:
    signaljt.login()
except signaljt.SignaljtError as e:
    print("signaljt failed:", e)
```

## API reference

```python
# Auth
signaljt.login(api_key=None, *, validate=True) -> None
signaljt.logout() -> None
signaljt.whoami(api_key=None) -> dict

# Datasets
signaljt.list_languages(api_key=None) -> list[str]
signaljt.fetch_dataset(
    language,                      # str code | list[str] | "all"
    *, api_key=None, cache_dir=None, local_dir=None, local_dir_mode="auto",
    connections=1, extract_workers=1, chunk_size=1048576,
    min_segment=8388608, max_segment=67108864, retries=4,
    force=False, refresh=False, max_parallel=4, progress=True,
) -> Dataset | dict[str, Dataset]

# Scoring
signaljt.evaluate(
    predictions=None, *, predictions_url=None, eval_name=None,
    penalize_missing=True, tag_rows=True, strip_bracketed=False,
    ddp=False, main_process_only=False,
    api_key=None, poll_interval=3.0, timeout=1800.0, progress=False,
) -> EvalResult
signaljt.submit(...) -> EvalJob
signaljt.get_result(eval_id=None, *, eval_name=None, wait=False, ...) -> EvalResult
signaljt.is_main_process() -> bool         # True only on rank 0 (DDP)

# EvalResult
r.oiwer; r.overall; r.per_language; r.per_chunk; r.meta
r.eval_id; r.eval_name; r.result_url
r.metrics(*, per_language=True, words=True, prefix="oiwer") -> dict
r.to_dict() -> dict;  r.save(path) -> None
```

---

# REST API

## Base URL & authentication

Three plain HTTP endpoints. All use API-key auth via the `X-API-Key` header.

```http
Base URL:  https://data-collection-app.joshtalks.org
Header:    X-API-Key: your-api-key-here
```

An expired or invalid key returns `401` on every endpoint.

## 1. Fetch ground-truth audios

```
GET /api/eval/fetch-audios/?language_code=<code>
```

Returns a signed download URL (valid ~1 day) for a language's ground-truth
package: a ZIP containing a manifest plus the audio files. The URL supports HTTP
`Range` requests, so you can parallelize or resume the download.

```bash
curl -s \
  -H "X-API-Key: YOUR_KEY" \
  "https://data-collection-app.joshtalks.org/api/eval/fetch-audios/?language_code=hi"
```

Response `200`:

```json
{
  "language_code": "hi",
  "url": "https://storage.googleapis.com/asr_eval/validation_language_zips/Hindi.zip?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Signature=..."
}
```

Download and unpack it:

```bash
URL=$(curl -s -H "X-API-Key: YOUR_KEY" \
  "https://data-collection-app.joshtalks.org/api/eval/fetch-audios/?language_code=hi" | jq -r .url)
curl -o hindi.zip "$URL"
unzip hindi.zip -d hindi/
```

ZIP structure:

```text
Hindi_manifest.jsonl        # one JSON object per line
audio/
  100003.flac
  100004.flac
  ...
```

Each manifest line has the utterance index and the relative audio path:

```json
{"Index": "100003", "audio_filepath": "audio/100003.flac"}
```

Run your ASR model over these audio files to produce predictions for step 2.

| Status | Cause |
|---|---|
| `400` | no `language_code` (body includes `supported_language_codes`) |
| `404` | unknown language code |
| `401` | bad / expired key |

### Listing supported languages

There is no dedicated endpoint. Call `fetch-audios` without a `language_code` and
read `supported_language_codes` from the `400` body:

```bash
curl -s -H "X-API-Key: YOUR_KEY" \
  "https://data-collection-app.joshtalks.org/api/eval/fetch-audios/" | jq .supported_language_codes
```

## 2. Submit predictions

```
POST /api/eval/submit/
```

Submits your model's predictions. The backend scores them against the ground
truth asynchronously and returns an `eval_id`. Provide one of `file` (upload) or
`predictions_url` (a public/GCS URL).

### Predictions file format

JSON or JSONL, mapping each utterance index to your model's transcript.

JSON: an object of `index -> hypothesis`:

```json
{
  "100003": "आपका मॉडल आउटपुट यहाँ",
  "100004": "अगली प्रतिलिपि"
}
```

JSONL: one object per line:

```json
{"index": "100003", "hypothesis": "आपका मॉडल आउटपुट यहाँ"}
{"index": "100004", "hypothesis": "अगली प्रतिलिपि"}
```

Predictions may span any subset of languages. The backend matches each index to
its language's ground truth and reports per-language scores.

### Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `file` | file (multipart) | n/a | Predictions file (JSON or JSONL). One of `file` / `predictions_url` required. |
| `predictions_url` | string | n/a | Public/GCS URL of the predictions file (no upload). |
| `eval_name` | string | auto | Human-readable name for the run. |
| `penalize_missing` | bool | `true` | Count ground-truth utterances with no prediction as full errors. false skips them. |
| `tag_rows` | bool | `true` | Add error-type tags (correct, wrong_script, ...) to each per-chunk entry. |
| `strip_bracketed` | bool | `false` | Strip bracketed tokens like `<noise>` / `[laughter]` before scoring. |

### Request: file upload (multipart/form-data)

```bash
curl -s -X POST \
  -H "X-API-Key: YOUR_KEY" \
  -F "file=@predictions.json;type=application/json" \
  -F "eval_name=conformer-ctc-ckpt3800" \
  -F "penalize_missing=true" \
  -F "tag_rows=true" \
  -F "strip_bracketed=false" \
  "https://data-collection-app.joshtalks.org/api/eval/submit/"
```

### Request: by URL (application/json)

```bash
curl -s -X POST \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "predictions_url": "https://storage.googleapis.com/your-bucket/predictions.json",
        "eval_name": "conformer-ctc-ckpt3800",
        "penalize_missing": true
      }' \
  "https://data-collection-app.joshtalks.org/api/eval/submit/"
```

Response `202` (queued). Save the `eval_id` and poll step 3:

```json
{
  "eval_id": "9c4b6282720",
  "eval_name": "conformer-ctc-ckpt3800",
  "status": "queued",
  "message": "Your eval is queued. You can see the result on the dashboard or call the get_result API with your eval ID: 9c4b6282720"
}
```

| Status | Cause |
|---|---|
| `400` | neither `file` nor `predictions_url` provided |
| `401` | bad / expired key |

## 3. Fetch eval result

```
GET /api/eval/result/?eval_id=<id>
```

Poll the status of a submitted eval and get the signed URL to the full result
once it's done. `eval_id` takes priority; `eval_name` (latest run with that name)
is a fallback. At least one is required.

```bash
curl -s -H "X-API-Key: YOUR_KEY" \
  "https://data-collection-app.joshtalks.org/api/eval/result/?eval_id=9c4b6282720"
```

Response `200` while processing (`status` is `queued` or `running`):

```json
{
  "eval_id": "9c4b6282720",
  "eval_name": "conformer-ctc-ckpt3800",
  "status": "running",
  "language_code": "",
  "submitted_at": "2026-07-11T10:00:00Z",
  "completed_at": null,
  "message": "Eval is still processing. Please check back shortly."
}
```

Response `200` completed:

```json
{
  "eval_id": "9c4b6282720",
  "eval_name": "conformer-ctc-ckpt3800",
  "status": "completed",
  "language_code": "",
  "submitted_at": "2026-07-11T10:00:00Z",
  "completed_at": "2026-07-11T10:05:32Z",
  "result_file_url": "https://storage.googleapis.com/asr_eval/eval_results/...?X-Goog-Signature=..."
}
```

`result_file_url` is a signed link (valid ~2 days) to the full `eval_result.json`
(schema below). Failed runs return `status: "failed"` with an `error`.

| Status | Cause |
|---|---|
| `400` | neither `eval_id` nor `eval_name` provided |
| `404` | unknown id |
| `401` | bad / expired key |

## Result file schema (eval_result.json)

Top-level keys: `overall`, `per_language`, `per_chunk`, `meta`. In the raw file,
`per_language` is keyed by language name (with an `iso` field inside).

```json
{
  "overall": {
    "weighted_oiwer": 0.1033,
    "macro_oiwer": 0.103,
    "weighted_by": "reference_words",
    "word": { "errors": 18417, "count": 178265, "sub": 9586, "ins": 3181, "del": 5650 },
    "total_utterances": 14714,
    "total_duration_sec": 76784.6,
    "languages_scored": 15,
    "extra_predictions": 0
  },
  "per_language": {
    "hindi": {
      "iso": "hi",
      "oiwer": 0.104,
      "word": { "errors": 1151, "count": 11129, "sub": 625, "ins": 206, "del": 320 },
      "utterances": 1034,
      "coverage": { "scored": 1034, "missing": 0, "extra": 0 }
    }
  },
  "per_chunk": [
    {
      "index": "100003",
      "language": "hindi",
      "duration": 4.68,
      "cps": 8.33,
      "p808_mos": 3.842,
      "native_district": "…",
      "native_state": "…",
      "gender": "M",
      "speaker_id": 106583,
      "predicted": true,
      "status": "scored",
      "word": { "errors": 0, "count": 9, "sub": 0, "ins": 0, "del": 0 },
      "alignment": { "hyp": ["…", "…"], "ops": ["c", "c"] },
      "tags": ["correct"]
    }
  ],
  "meta": {
    "scored_at": "2026-07-12T05:17:58Z",
    "audio_version": "0.1.0",
    "lattice_version": "0.1.0",
    "penalize_missing": true,
    "tag_rows": true,
    "strip_bracketed": false
  }
}
```

| Section | Fields |
|---|---|
| `overall` | `weighted_oiwer`, `macro_oiwer`, `weighted_by`, `word{errors,count,sub,ins,del}`, `total_utterances`, `total_duration_sec`, `languages_scored`, `extra_predictions` |
| `per_language[<name>]` | `iso`, `oiwer`, `word{...}`, `utterances`, `coverage{scored,missing,extra}` |
| `per_chunk[i]` | `index`, `language`, `duration`, `cps`, `p808_mos`, `native_district`, `native_state`, `gender`, `speaker_id`, `predicted`, `status`, `word{...}`, `alignment{hyp[],ops[]}`, `tags[]` |
| `meta` | `scored_at`, `audio_version`, `lattice_version`, `penalize_missing`, `tag_rows`, `strip_bracketed` |

- `weighted_oiwer`: overall OI-WER weighted by reference words (the headline number).
- `macro_oiwer`: unweighted mean across languages.
- `alignment.ops` codes: `c` = correct, `s` = substitution, `i` = insertion, `d` = deletion.
- `coverage.missing`: ground-truth utterances you didn't predict (penalized only when `penalize_missing=true`).

## End-to-end with curl

```bash
KEY="YOUR_KEY"
BASE="https://data-collection-app.joshtalks.org/api/eval"

# 1. Get + unpack the ground truth for a language
URL=$(curl -s -H "X-API-Key: $KEY" "$BASE/fetch-audios/?language_code=hi" | jq -r .url)
curl -o hi.zip "$URL" && unzip hi.zip -d hi/

# 2. Run your ASR model over hi/audio/*.flac -> predictions.json  (your code)

# 3. Submit
ID=$(curl -s -X POST -H "X-API-Key: $KEY" \
  -F "file=@predictions.json;type=application/json" \
  -F "eval_name=my-model-v1" \
  "$BASE/submit/" | jq -r .eval_id)

# 4. Poll until completed
while [ "$(curl -s -H "X-API-Key: $KEY" "$BASE/result/?eval_id=$ID" | jq -r .status)" != "completed" ]; do
  sleep 3
done

# 5. Download the full result
RESULT=$(curl -s -H "X-API-Key: $KEY" "$BASE/result/?eval_id=$ID" | jq -r .result_file_url)
curl -o eval_result.json "$RESULT"
jq '.overall.weighted_oiwer' eval_result.json
```

---

Get an API key from the dashboard (`/api-keys`). Contact sales:
https://ai.joshtalks.com/contact-us