nvidia-cosmos-video

What it does

Instructs the agent to select, run, and govern NVIDIA Cosmos world-video generation across multiple artifacts (Cosmos 3 Generator, Predict2.5, Transfer2.5, downloadable checkpoints, self-hosted NIMs, and hosted previews). It targets text/image/video-to-world tasks, controlled world transfer, multiview or action-conditioned video, local checkpoint pinning, hardware planning, inference approval, and output validation.

How it works

• Route by family and task: choose appropriate Cosmos artifact (Cosmos 3 Generator, Cosmos-Predict2.5, Cosmos-Transfer2.5, etc.) based on input scenario; use the corresponding inputs and outputs as described (e.g., T2V, I2V, edges/depth/seg controls).
• Use self-hosted NVIDIA NIM: operate via POST /v1/infer with a per-artifact container; NIM returns base64 MP4; follow exact schema and field names provided for each artifact (T2V, I2V, Video2World, etc.).
• Maintain precise instrument details: parameters like resolution, num_output_frames, fps, steps, guidance_scale, seed, and specific prompts/negative_prompts are specified for each task.
• Manage local checkpoints and revision control: before fetch, create a deployment manifest with surface, container, model_size, precision, nim_model_profile, metadata_sha256, guardrail, driver info, network policy, and input/output hashes.
• Apply dedicated prompts and controls: for Transfer2.5, supply edge/depth/vis/seg controls; for Predict2.5, supply either image or video (not both) to select Image2World or Video2World; for Generator, provide appropriate prompt, image, and constraints.
• Use specific hardware/precision constraints per surface and ensure correct model-size and checkpoint handling (nano vs. super, FP8 vs BF16).

When to use it

• Use when you need production-style world-generation and world-simulation workflows across Cosmos 3 family artifacts, or when performing controlled world transfer, multiview video, or action-conditioned AI video.
• Use when you require local checkpoint pinning, hardware planning, inference approvals, and output validation prior to deployment.

What it can touch

• Tools: "claude-code", "codex", "copilot", "cursor".
• It references commands and endpoints like POST /v1/infer, specific NIM container references, and several field names (e.g., prompt, image, video, edge, depth, vis, seg, resolution, num_output_frames, guidance, num_steps).

Caveats

• It documents that Cosmos 3 Generator NIM contracts are surface-specific and require exact field values; avoid mixing artifacts or mis-specifying model sizes or resolutions.
• It emphasizes pinning immutable snapshots and not using mutable main branches in production; explains offline-after-cache-warm operation and the need for approved manifests.
• It notes that some paths (Reason/Embed-only) are not intended for generation requests.
• Licensing and NIM security: local checkpoints and safetensors are preferred; DCP vs consolidated .pt formats have specific handling rules; explicit permission and hashes are required for deployment and execution.