search-benchmark

npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# Search Benchmark Skill Measure semantic search quality against a deployed AI Registry. Generates a ground truth dataset from the registry's own assets, runs queries against the live API, evaluates results using standard information retrieval metrics (NDCG@10, MRR, Recall@10), and produces a markdown report. ## Prerequisites 1. **Registry URL** - The base URL of the deployed registry (e.g., `https://d2xl2zfuhgc4l0.cloudfront.net`) 2. **JWT Token** - A valid admin token in `.token` file (get from "Get JWT Token" button in registry UI) 3. **Registry must have assets indexed** - At least some servers, agents, or skills registered The `.token` file supports both raw JWT format and the full JSON response from the registry UI. ## Input ``` /search-benchmark [REGISTRY_URL] [TOKEN_FILE] ``` - **REGISTRY_URL** - Base URL of the registry to benchmark (default: reads from user or uses `http://localhost`) - **TOKEN_FILE** - Path to the token file (default: `.token`) ## Workflow ### Step 1: Check for Existing Ground Truth Check if a ground truth dataset already exists: ```bash ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/null ``` If the file exists, report how many queries it cont

ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/null
import json
with open('{TOKEN_FILE}') as f:
raw = f.read().strip()
if raw.startswith('{'):
data = json.loads(raw)
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/servers?limit=2000" > /tmp/servers.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/agents?limit=2000" > /tmp/agents.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/skills?limit=2000" > /tmp/skills.json
uv run python scripts/benchmark_search.py \