semantic-consistency-auditor

npx skills add majiayu000/claude-skill-registry --skill semantic-consistency-auditor --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# Skill: Semantic Consistency Auditor **ID:** 212 **Name:** semantic-consistency-auditor **Description:** Introduces BERTScore and COMET algorithms to evaluate the semantic consistency between AI-generated clinical notes and expert gold standards from the "semantic entailment" level. ## Overview Semantic Consistency Auditor is a medical AI evaluation tool used to assess the semantic consistency between AI-generated clinical notes and expert-written gold standards from a semantic level. This tool is not limited to traditional string matching or bag-of-words models, but uses deep learning models to understand semantic entailment relationships, capable of identifying expressions with different wording but similar meaning. ## Algorithms ### 1. BERTScore BERTScore uses pre-trained BERT model contextual embeddings to calculate similarity between candidate text and reference text: - **Precision**: How much semantics in the candidate text is covered by the reference text - **Recall**: How much semantics in the reference text is covered by the candidate text - **F1 Score**: Harmonic mean of Precision and Recall ### 2. COMET (Cross-lingual Optimized Metric for Evaluation of Translation) COME

Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Linux/Mac
Or venv\Scripts\activate  # Windows
Install dependencies
pip install bertscore comet-ml transformers torch
Evaluate single case pair
python scripts/main.py \
Batch evaluation from JSON file
Use specific model