digital-health-clinical-asr-eval

What it does

Scores a NeMo manifest for clinical ASR by transcribing each row with an ASR NIM, computes four metrics (WER, CER, KER, SER) with inlined scoring, and generates a five-section KER leaderboard. It uses per-row fields such as term, entity_category, ipa_source, voice_id, noise_level, and context_type, and outputs artifacts including per_sample.json, results.csv, and leaderboard_cycle<N>.md. It decides the next step via a post-eval decision tree and, depending on results, may route back to build or to finetune stages.

How it works

• Accepts a NeMo-format manifest.jsonl with clinical-extension fields. For each row, transcribes audio_filepath using the chosen ASR NIM (default Parakeet TDT v2) and writes per_sample.json with fields: audio_filepath, ref, hyp, term, entity_category, ipa_source, voice_id, noise_level, context_type.
• Scores four metrics per row: WER, CER, KER, SER, applying normalization steps to ref and hyp (lowercase, NFKD, strip punctuation except hyphen, collapse spaces).
• KER is computed as a contiguous, in-order match of the term words in the normalized hypothesis; a miss of contiguous term words yields KER=1 for that row, otherwise 0, while WER/CER reflect broader errors.
• Builds a five-section leaderboard in this order: Headline (WER/CER/KER/SER), KER by entity_category, KER by ipa_source, KER by noise_level, Per-term KER (worst first).
• Writes artifacts: per_sample.json, results.csv, leaderboard_cycle<N>.md, and uses the post-eval decision tree to route the user (e.g., back to build, to finetune, or stop and harden).

When to use it

Use when you have a manifest.jsonl and want to score it and obtain a structured leaderboard to guide next steps in the Clinical ASR Flywheel, including potential routing decisions after evaluation.

What it can touch

• Reads and writes files per the transcribe and scoring steps, including per_sample.json, results.csv, and leaderboard markdown.
• Interfaces with the ASR NIM via offline_recognize in the provided recipe, and respects environment overrides for ASR model, function-id, and endpoint.

Caveats

• Requires NVIDIA_API_KEY for hosted ASR NIMs and a NeMo-format manifest with clinical-extension fields. All ASR call shapes and scoring recipes are inlined; no sibling agent skill required.
• Audio leaves your environment when sending WAV clips to NVIDIA NVCF Parakeet/Nemotron ASR for each manifest row; synthetic audio should be used per protocol.
• Special routing rules apply for certain ipa_source/terminology cases and the decision tree may route to build or finetune rather than continuing eval.