agentsop-idempotent-ingestion
Re-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — **ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op** — plus the docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` + `DocstoreStrategy`), the delete-propagation problem, and cross-framework equiv
npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Implements an idempotent ingestion protocol for production RAG pipelines. It ensures that re-running an ingestion over a changing corpus results in no-op processing for unchanged documents by hashing content and comparing against a persisted docstore, and by upserting or deleting as dictated by the chosen strategy. It also provides workflow steps, activation conditions, and concrete code patterns to maintain a durable document ledger and consistent vector store state across repeated runs.
How it works
- Activation when an ingestion path will run more than once over a changing corpus; otherwise not activated.
- Core principle: each document is reduced to a stable content hash and a stable doc_id; the doc_id/hash lookup in a persisted docstore decides whether to INSERT (embed + upsert), UPDATE (re-embed and upsert), or SKIP.
- The docstore is a separate ledger from the vector store, holding doc_id → hash mappings used to determine if content has changed.
- Deletes are not automatic; to propagate deletes, the pipeline must compare the set of doc_ids seen this run with the docstore and purge differences using UPSERTS_AND_DELETE or equivalent mechanisms in LangChain.
- SOP workflow includes stages for framing the re-run, attaching a persisted docstore, pinning stable doc IDs, choosing an upsert strategy, proving idempotency with a test, and ongoing persistence/observability.
- Concrete code examples show wiring IngestionPipeline with a docstore and DocstoreStrategy, pinning stable doc IDs via filename_as_id, and testing idempotency by running the pipeline twice and asserting no new processed nodes on the second run.
When to use it
- Any ingestion pipeline that will run more than once over a corpus that can change between runs (cron, CI, webhook, manual).
- When source documents are added, edited, or deleted over time and you need dedup and optional delete propagation.
- When debugging duplicates, stale chunks, or inconsistent retrieval after re-runs.
- When reviewing PRs that call VectorStoreIndex.from_documents or pipeline.run without a docstore.
- When designing the ingestion side of a production RAG before first deploy.
What it can touch
- IngestionPipeline, DocstoreStrategy, SimpleDocumentStore, RedisDocumentStore, MongoDocumentStore, PostgresDocumentStore as the docstore backend.
- Code patterns shown for wiring docstore, setting doc_id stability (filename_as_id=True or explicit doc.id_), and Stage 4 idempotency tests.
Caveats
- Deletes are not propagated by default unless the pipeline is configured with UPSERTS_AND_DELETE and the run includes a diff against the docstore; stale content requires explicit diffing logic.
- A persisted docstore is required for correctness; non-persistent or ephemeral docstores defeat idempotency.
- The approach relies on stable doc_id and stable content hashing to guarantee skip behavior across runs.
# Idempotent Ingestion · Re-Ingest-Correctness SOP > Third-person operating model for a coder agent that owns ingestion > correctness across repeated runs. The audience is the LLM agent writing or > reviewing the pipeline code — not the end user. > **One sentence**: *The interesting run is the second one. A correct pipeline > hashes each document, looks the hash up in a docstore, and inserts / updates > / skips accordingly — so re-running over unchanged docs changes nothing.* This skill is an **ENHANCE overlay** over [[llamaindex]]. The base skill names `IngestionPipeline` with `docstore` + `UPSERTS_AND_DELETE` as a hardening must-have (OP-08, anti-pattern A10) but stops at "use it". This overlay surfaces the *correctness contract* — what idempotency actually means, how the hash decides the action, and how it generalises beyond LlamaIndex. --- ## 1. 何时激活 (Activation Rules) Activate this skill whenever **any** of the following holds: 1. The codebase contains an ingestion call (`VectorStoreIndex.from_documents`, `pipeline.run(documents=...)`, `vectorstore.add_documents`, a custom embed + upsert loop) **and** that call will execute more than once over a corpus that can change between
- 1. 何时激活 (Activation Rules)
- 2. 核心心智模型 (Core Mental Model)
- Principle 1 — Ingestion must be idempotent; the hash is the decision
- Principle 2 — The docstore is a separate ledger, not the vector store
- Principle 3 — Deletes don't propagate for free
- 3. SOP 工作流 (Agentic Protocol)
- Stage 0 — Frame the re-run
- Stage 1 — Attach a persisted docstore
- Stage 2 — Pin stable doc ids and let the pipeline hash
- Stage 3 — Choose the upsert strategy deliberately
- Stage 4 — Prove the second run is a no-op (the gate)
- Stage 5 — Schedule, persist, observe
- 4. 操作模型 (Operation Models)
- OP-01 AttachDocstore
What does the agentsop-idempotent-ingestion skill do?
Re-ingest-correctness SOP for production RAG. Activate when a calling agent builds, reviews, or debugs an ingestion pipeline that runs more than once over a changing corpus — scheduled re-index, incremental updates, CI re-ingest, or a "retrieval has duplicates / shows deleted docs" bug. Encodes the rule — **ingestion must be idempotent: a document's content hash decides insert/update/skip, so re-running over unchanged docs is a no-op** — plus the docstore + doc-hash upsert machinery (LlamaIndex `IngestionPipeline` + `DocstoreStrategy`), the delete-propagation problem, and cross-framework equiv
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-idempotent-ingestion --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.