agentsop-per-model-artifacts

What it does

Operates as a lifecycle SOP for per-model prompt artifacts: it defines when to activate, the core mental model of a three-axis artifact (program × LM snapshot × dataset), and a five-stage workflow to create, register, test, and recompile artifacts when model snapshots or configurations change. It emphasizes that a compiled prompt is not portable and that the identity is tied to the specific LM snapshot and dataset hash. It also prescribes storage, swap-test, gate, and lineage operations to manage artifacts and their registry entries across production deployments. The SOP includes explicit triggers (e.g., adopting compiled prompts, multi-LM production, provider deprecation, framework deprecation) and concrete actions (pin dated LM snapshots, compute dataset sha8, run optimizer, save compiled.json, update REGISTRY, perform regression gates, and emit swap reports). It covers how to handle swaps, recompile decisions, and quarterly maintenance.

How it works

The workflow consists of five stages:

1. Initial compile: define program graph, pin dated LM snapshot, pin dataset hash, run optimizer (MIPROv2 auto="light"), and evaluate on a held-out test set. Exit when held-out test score is recorded but artifact not yet saved.
2. Version: compute artifact path artifacts/<program>/<provider>/<model-snapshot>/<dataset-sha8>/v<n>.json, run compiled.save(path) (or compiled.save(path_dir, save_program=True)), append a REGISTRY entry, and commit both artifacts.
3. Register: CI loads the artifact, identifies parent_artifact, re-runs held-out test, and gates PR if test difference exceeds a threshold; may require artifact-override label.
4. Swap-test: upon new LM snapshot or family swap, run swap-test in both directions, compute Δ-scores, and decide whether to recompile or transfer. Emit a swap-report artifact.
5. Recompile or accept: re-run optimizer under new LM if required, increment v, or keep old/new artifacts and rely on git tags for production deploy. There is a quarterly deprecation scan and canary re-eval workflow. Additionally, it defines storage, swap-test, gate, and lineage operations to track artifact paths, registry entries, and provenance.

When to use it

Activate this SOP when adopting compiled prompts, when supporting multiple LMs in production, when a provider deprecates a model snapshot, or when a framework deprecates a config surface. It should not activate for raw one-off prompt edits or truly model-agnostic system prompts that have been swap-tested. It also specifies triggers for cross-skill bridges and helps manage per-model artifacts across model swaps.

What it can touch

Artifacts are stored under paths like artifacts/<program>/<provider>/<model-snapshot>/<dataset-sha8>/v<n>.json. It uses compiled.save and may use compiled.save(dir, save_program=True). It appends lines to artifacts/REGISTRY.jsonl. It uses a git tag like prompt/<program>/<snapshot>/v<n> for production deployment. It supports swap-test outputs like Δ-score and recompile_required flags, and writes swap-report-<old>-to-<new>.md artifacts.

Caveats

The SOP emphasizes dated LM snapshots, not aliases, to avoid silent behavior drift. It requires staged commits and registry updates, with gating on regression tests. It mandates that artifacts are not deployed if the regression gate fails and supports keeping both old and new artifacts during transitions. It notes that a compiled prompt is LM-conditioned and not portable across LM changes without recompilation. License: MIT.