agentsop-selfhost-decision

What it does

It provides a project-kickoff rubric to decide between self-hosting an LLM platform vs using a managed API, based on two axes: VOLUME (a cost-crossover slider) and COMPLIANCE (a hard gate). It is activated at kickoff to determine where inference should run, or when data-residency pressure prompts reevaluation.

How it works

• It instructs to first apply a COMPLIANCE GATE: verify whether data residency, air-gap, or regulated data requirements forbid leaving your boundary. If yes and no compliant in-region tier exists, self-host; if compliant managed tier exists, allow managed.
• If compliance permits, compute VOLUME crossover: estimate monthly volume, calculate managed_cost = V × $/token and selfhost_cost = (GPU + OPS_LABOR + infra) + V × marginal. Derive V* = fixed / (managed_$tok − marginal_$tok). Below V*, managed wins; above V*, self-host may win if other conditions hold.
• Check HONEST OPS-CAPACITY: confirm there is a DevOps/SRE owner and external components (PostgreSQL/Redis/Vector DB, K8s/Helm) required for self-hosting. If not, managed wins.
• Perform THROUGHPUT-HEADROOM CHECK: calculate replicas = peak_QPS / per-replica_ceiling and re-run the crossover with updated GPU term; if still favorable, proceed to self-host; otherwise reconsider.
• Consider HYBRID options for steady baseline self-hosting with burst traffic to managed API.
• Include PLAN THE FALLBACK steps to flip direction if needed and keep a thin API surface to ease reversibility.

When to use it

• At project kickoff when deciding where an LLM platform runs (managed API/cloud vs self-hosted GPU).
• When monthly API spend climbs and a flip to self-host is contemplated.
• When data-residency or compliance pressure appears mid-project.
• When evaluating self-hostable platform tiers vs own deployment.
• When designing a hybrid topology for steady state plus burst routing.

What it can touch

• The workflow references specific steps and calculations (OP-1 through OP-7) and mentions concepts like GPU term, ops_labor, and replica ceilings. It uses terms like Dify Cloud Pro ($59) and mentions platforms like vLLM and Dify. It does not specify executable scripts or commands beyond described actions.

Caveats

• It requires explicit estimates for tokens per request, monthly volume, and the ops-labor cost; these are not provided and must be supplied by the user. It emphasizes that compliance gating overrides cost and that real-world constraints (ops capacity, throughput ceilings) can invalidate pure cost calculations. It notes potential caveats around replica scaling and hybrid routing, and the need for a reversible flip plan. License/usage implications are discussed in the context of license checks for self-hosted vs managed offerings.