twelvelabs-video-understanding

What it does

The skill enables an agent to drive TwelveLabs for real video-understanding work: indexing footage, semantic/visual search across an archive, generating descriptions, summaries, chapters, highlights, and tags from video, producing multimodal embeddings, and wiring TwelveLabs into a media-production pipeline (NLE panels, logging, compliance review, metadata). It covers model families (Marengo for search/embeddings, Pegasus for video-to-text analysis) and the v1.3 Video Understanding API (indexes, assets, tasks, search, analyze, embed), including prompt construction for analysis, capability and format limits, pricing/quota math, output quality review, and privacy/rights obligations. It is not a video pixel generator. It does analysis and retrieval, not generation or editing of pixels.

All volatile facts below carry a verification date. Re-verify model names, limits, and pricing against docs.twelvelabs.io before quoting them to a user as current.

How it works

The skill describes two primary model families and how to use them via the TwelveLabs API:

• Marengo (search + embeddings): you enable Marengo to make videos searchable and to generate embeddings for a vector store. Endpoints include /search and /embed. Outputs are ranked video segments with start/end, video_id, score, confidence, and rank. Embeddings are 512-dimensional and used for retrieval in a vector DB.
• Pegasus (analysis + text generation): you enable Pegasus to analyze and produce natural-language or structured text via /analyze. It can return summaries, descriptions, captions, or structured JSON (via response_format). The synchronous endpoint handles clips up to 1 hour; longer content uses the async analyze task. It supports a structured prompt (prompt/prompt_v2), temperature control, max_tokens, and time-limited segments with start_time/end_time for analysis scope.

The API flow centers on an index configuring which models are enabled and what modalities are analyzed. Uploading/indexing can be done via a legacy single-call route or the newer asset-based flow; indexing is asynchronous with task polling. Searches can be performed per index, with options to match modalities and transcription methods. Embeddings can be generated synchronously or asynchronously and stored in a user-owned vector DB.

When to use it

Use TwelveLabs for:

• Archive / footage search via natural language or visual queries across indexed video.
• Logging & tagging to auto-generate metadata for raw footage or dailies.
• Segmentation to identify chapters, scene breaks, highlights, or speaker changes.
• Video-to-text tasks including summaries, descriptions, captions, Q&A, or structured extraction (e.g., shot lists, compliance flags).
• Embeddings for a custom recommender, deduplication, or RAG over a video store.
• Compliance / brand-safety review to locate logos, on-screen text, phrases, or sensitive content.

Do not use it to generate video pixels or to replace a dedicated ASR for plain transcripts.

What it can touch

• Core API endpoints: /v1.3/search (Marengo), /v1.3/analyze (Pegasus), /v1.3/embed (Marengo).
• Index, asset, and indexing tasks: POST /v1.3/tasks, POST /assets, POST /indexes/{index-id}/indexed-assets.
• Optional prompting constructs: prompt, prompt_v2, response_format, start_time, end_time.
• It references model capabilities and limits (Marengo 3.0, Pegasus 1.5, 1.2 as fallback).

Caveats

• Format/size ceilings differ by model: Marengo up to 4 h / 4 GB; Pegasus up to 2 h / 2 GB; sync analyze up to 1 h. Uploads: URL ≤ 4 GB, local ≤ 2 GB per single-call, or ≤ 10 GB via multipart. Verify the tighter limit when planning tasks.
• Pegasus outputs may be represented with timestamps; for frame-accurate edits, prefer Marengo search for segment boundaries.
• The tool is not for generating video pixels; it performs analysis and retrieval, not media creation.
• Always verify model versions and pricing against current TwelveLabs docs before quoting or implementing in production.