multimodal-ml
Vision-language model patterns including CLIP, LLaVA, cross-modal alignment, and embedding fusion. Use when building or integrating multimodal ML systems.
npx skills add majiayu000/claude-skill-registry --skill multimodal-ml --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Multimodal ML Patterns ## Fusion Strategy Selection | Strategy | Architecture | Pros | Cons | Use When | |----------|-------------|------|------|----------| | **Late Fusion** | Separate encoders, merge at output | Simple, modular, can swap encoders | Limited cross-modal reasoning | Classification, retrieval, CLIP-style | | **Early Fusion** | Concatenate raw tokens, single model | Deep cross-modal interaction | Expensive, needs lots of data | LLMs with image tokens (LLaVA, GPT-4V) | | **Cross-Attention** | Separate encoders + cross-attn layers | Good balance of depth and modularity | More params, harder to train | Flamingo, image captioning, VQA | | **Q-Former** | Learnable queries cross-attend to visual features | Fixed # of visual tokens, efficient | Requires pretraining Q-Former | BLIP-2, InstructBLIP | | **Bottleneck/Perceiver** | Learnable queries attend to both modalities | Fixed compute regardless of input size | May lose fine-grained detail | Long sequences, many modalities | Default recommendation: Late fusion (contrastive) for retrieval/matching. Early fusion (projector + LLM) for generation tasks. ## API-First Vision-Language Models ### Model Selection | Model | Strengt
- Fusion Strategy Selection
- API-First Vision-Language Models
- Model Selection
- SDK Examples
- Open-Source Vision-Language Models
- CLIP-Style Contrastive Training
- Contrastive Training Tips
- Vision Encoder Integration with LLMs (LLaVA Pattern)
- Cross-Attention Fusion (Flamingo Pattern)
- Multimodal Data Loading
- Training Recipes
- Stage 1: Alignment Pretraining (LLaVA pattern)
- Stage 2: Instruction Tuning
- Gotchas and Anti-Patterns
What does the multimodal-ml skill do?
Vision-language model patterns including CLIP, LLaVA, cross-modal alignment, and embedding fusion. Use when building or integrating multimodal ML systems.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill multimodal-ml --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.