sync-labs-lipsync

What it does

Generates AI lip sync and visual dubbing for talking-head video by selecting a Sync Labs model, constructing a POST /v2/generate request, hosting inputs, and managing async job results via polling or webhooks. It supports batch dubbing/localization, per-segment audio mapping, and output review for sync accuracy and identity preservation. It is intended for re-voicing, dubbing, translation, personalization, or re-timing mouth movement, including animating a still face from audio, and for debugging failed or rejected Sync generations. It also covers consent, likeness, and rights obligations for editing a real person’s face.

How it works

• Choose a model from: sync-3, lipsync-2, lipsync-2-pro, lipsync-1.9.0-beta, or react-1, based on input and quality needs; prepare a request to POST https://api.sync.so/v2/generate with a body containing model, input (one visual item and one audio source, plus optional dubParams or TTS), options (sync_mode, temperature, active_speaker_detection, occlusion_detection_enabled, model_mode, prompt for react-1), outputFileName, and optional webhookUrl.
• The input array must have exactly one visual item (video or image for sync-3) and one audio source; other inputs are rejected. Segment-level overrides and dubParams allow per-segment audio or translation dubbing.
• Use the provided fields to control alignment (sync_mode), motion expressiveness (temperature), and face selection (active_speaker_detection or coordinates). For images, use sync-3 and ignore sync_mode.
• For translations/dubbing, you may supply dubParams with providerName, sourceLang, targetLang, and numSpeakers; the system will extract audio, translate, and re-voice.
• Results are returned with an id and status; poll with GET /v2/generate/{id} or receive webhook deliveries, verifying the Sync-Signature header; if delivery fails, rely on polling for reconciliation.
• Batch/localization supports a JSONL file with multiple /v2/generate bodies and unique request_id values, limited by plan.

When to use it

Use when dubbing/localizing a video, re-timing lips to corrected audio, personalizing a base video, or animating a portrait from audio. Use to debug a failed or rejected Sync generation. Do not use to generate a face from scratch or to swap identities; this tool edits an existing face region only.

What it can touch

• Endpoint: POST https://api.sync.so/v2/generate
• SDKs: Python (pip install syncsdk) importing Sync, and TypeScript (SyncClient) wrapping generations.create()/generations.get()
• Webhook delivery with header Sync-Signature: t={timestamp},v1={hmac}
• Optional assets via Assets API for input items (assetId) rather than direct URLs

Caveats

• Model choice depends on input: lipsync-2 is default for general cases; sync-3 supports still images and 4K inputs; lipsync-2-pro handles fine details; lip-sync-1.9.0-beta is for low-budget throwaway work; react-1 is for acting and has a 15-second input limit.
• Active Speaker Detection is a paid-tier feature; occlusion_detection_enabled is automatic on sync-3 and optional on lipsync-2/2-pro.
• Per-generation duration is plan-bound; input duration and per-generation limits apply (e.g., react-1 <= 15 s, audio up to 300 s, etc.).
• Output pricing is per second of output at 25 fps; exact rates vary by model and plan.