langchain-cost-tuning

What it does

The skill provides a runbook for tuning LangChain 1.0 usage to manage AI spend. It covers multiple levers: token accounting, retry discipline, agent loop caps, caching behavior, and model tiering. It also defines per-tenant budget enforcement, cache-key considerations for tool binding, and calibration of a semantic cache threshold. It includes a concrete per-tenant budget middleware outline and specific steps to implement and validate these controls.

How it works

• Step 1 reads and reports usage via usage_metadata to ensure streaming-accurate token accounting, avoiding response_metadata tokenUsage alone.
• Step 2 uses astream_events(version="v2") to stream and aggregate on_chat_model_stream events for precise meters.
• Step 3 introduces a retry-aware meter that deduplicates emissions by run_id and optionally by a stable request_id, with guidance to place token accounting above retry middleware or to dedupe on the run_id/request_id.
• Step 4 provides a model-tiering decision framework with a per-1M pricing snapshot for several models, advising when to favor cheaper tiers for drafts vs. more expensive models for final outputs.
• Step 5 aggregates Anthropic cache usage per session/tenant, calculating savings and persisting a CacheLedger per tenant/day for budget checks.
• Step 6 enforces tool-aware cache keys to prevent semantically wrong cache hits when bound tools are used, recommending composite keys with tool fingerprints.
• Step 7 calibrates RedisSemanticCache thresholds (not default 0.95) by collecting labeled pairs and tuning to keep false positives under 2%.
• Step 8 implements per-tenant budget middleware with soft and hard caps, using Redis to track daily usage and to emit alerts or reject calls when hard caps are reached.
• Step 9 caps recursion in agent loops (recommended 5-10 for interactive, 10-15 for batch) and suggests using trim_messages with include_system and start_on to control context.

When to use it

Use when AI spend grows faster than traffic, a cost regression lands, or you need per-tenant budget enforcement. Trigger phrases include: "langchain cost", "langchain token accounting", "langchain per-tenant budget", "langchain model tiering", and "prompt cache savings".

What it can touch

• usage_metadata (token accounting and cache fields)
• astream_events(version="v2")
• Retry-aware meter code (with run_id and optional request_id)
• Redis for per-tenant budget state and soft/hard caps
• InMemoryCache/SQLiteCache/RedisSemanticCache with tool-aware keys
• Per-tenant budget middleware class and Redis schema
• Recursion limit configuration in agent creation

Caveats

• Requires Claude Code compatibility (also compatible with Codex)
• Licensing: MIT
• Calibration steps for cache threshold and model tiering rely on current pricing snapshots and gold-set validation; prices and thresholds are not guaranteed constants.