Mixture of Experts

SNR switch, not a token router

LLM MoEs often route each token to a subset of experts. Wan2.2’s published design is coarser: the whole denoising step uses one expert. SNR falls as t decreases. At the start, t is large, noise is high, SNR is at SNR_min, and the high-noise expert owns the step. A threshold t_moe is defined at half of SNR_min; when t < t_moe the low-noise expert takes over.

Why two experts

Early steps decide global layout and motion; later steps decide textures and faces. Splitting capacity along that axis enlarges the model without doubling FLOPs per step. The README compares validation loss: Wan2.1 dense, mixed “Wan2.1 + one new expert,” and full Wan2.2 MoE — the full pair converges lowest.

Inference consequences

• VRAM still tracks a 14B DiT plus the second expert’s weights if both stay resident. Offload can page the idle expert.
• LoRAs usually need a high-noise file and a low-noise file.
• TI2V-5B is not MoE — it is a dense 5B with a heavier VAE.
• Community cache (Cache-dit) can skip work inside each expert separately.

FAQ

Do both experts run in one forward?

Not in the published schedule. One expert per timestep, switch once along the SNR curve.

Can I fine-tune only the low-noise expert?

Possible, and the README’s ablation even tests mixed experts — but quality drops versus training the pair.