Chinese
Deep Analysis

What Is LoRA? A Practical Guide to Low-Rank Adaptation for AI Video Generation

LoRA freezes a big model and trains two tiny low-rank matrices. How it works, and how to train and use LoRAs for Wan, HunyuanVideo and LTX-Video.

Jacky Wang 14 min read 193 views

If you have opened a ComfyUI workflow recently, or browsed video model resources on Civitai or Hugging Face, you have probably seen a pile of .safetensors files, tens to hundreds of megabytes each, all with LoRA in the name. One makes Wan videos look like a particular art style. Another keeps the same character recognisable across shots. Another only teaches a single orbit camera move. And some let you cut a 30-plus-step sampling run down to 4 steps.

They are all the same kind of thing. In this post I want to explain it properly: what LoRA is, why the maths makes it so cheap, how it compares with full fine-tuning, adapters and prompt tuning, and how to actually use it in AI video generation, from preparing data to training to loading it in ComfyUI.

What LoRA actually is

LoRA stands for Low-Rank Adaptation. It was introduced by Edward Hu and colleagues at Microsoft in the 2021 paper LoRA: Low-Rank Adaptation of Large Language Models, originally aimed at large language models such as GPT-3. IBM’s explainer covers the same idea in plain language.

The one-line version: leave the original model untouched, attach a very small patch next to it, and only train the patch.

Think of the base model as an illustrator who already knows how to draw. Full fine-tuning is sending them back to art school. LoRA is handing them a thin notebook of style notes they flip through while drawing. You can swap the notebook, stack two notebooks, or take it away, and the illustrator stays the same person.

The maths: W = W₀ + BA

Most of the computation inside a neural network is matrix multiplication. Say a layer has pretrained weights W₀ with shape d×k. Full fine-tuning learns an update ΔW of the same size.

LoRA’s bet is that the update needed to adapt to a new task has a low “intrinsic rank”. So instead of learning all d×k numbers directly, you factor the update into the product of two thin matrices:

W = W0 + ΔW = W0 + B·A
B: d × r      A: r × k      r ≪ min(d, k)
forward pass: h = W0·x + (α / r) · B·A·x

During training W₀ is frozen and gets no gradients. Only A and B are updated. In the paper, A starts with random Gaussian values and B starts at zero, so BA = 0 at step one and the model behaves exactly like the base model before it gradually learns something new.

A quick calculation shows where the savings come from. With d = k = 4096, a full update to that layer is 4096×4096 ≈ 16.8 million parameters. With r = 16, A and B together hold 2×4096×16 = 131,072 parameters, less than 1% of the original.

What rank and alpha mean

Rank (r) is the capacity of the patch. A higher rank can express more complex changes, produces a bigger file, and is more likely to memorise noise in your training set. On GPT-3, the paper found ranks as low as 1 to 4 were already competitive on the tasks they tested. The authors are also clear that a small rank will not suit every task, for example when the target domain is very different from the pretraining data.

Alpha (α) is a scaling factor. The update is multiplied by α/r. The paper fixes α to the first r value tried, so changing r later does not force you to retune the learning rate. In practice people commonly set alpha equal to the rank or half of it. Hugging Face PEFT also offers use_rslora=True, which scales by α/√r instead (rsLoRA); see the PEFT LoRA guide.

Why it saves memory and storage

According to the paper’s abstract, compared with fine-tuning GPT-3 175B using Adam, LoRA cuts trainable parameters by about 10,000 times and GPU memory by about 3 times, while matching or beating full fine-tuning quality on RoBERTa, DeBERTa, GPT-2 and GPT-3. Most of the memory saving comes from not keeping gradients and optimizer state for frozen weights. Adam stores two extra values for every trainable parameter, so freezing almost everything matters a lot.

Deployment is the other win. BA has the same shape as W₀, so after training you can add it straight into the base weights and pay zero extra inference latency. Or you can keep it separate and hot-swap different LoRA files on top of one base model.

How LoRA compares with other fine-tuning methods

  • Full fine-tuning updates every parameter. It has the highest ceiling, but every task needs its own full copy of the model, and memory and storage costs are the highest.
  • Adapters insert small new modules between layers. They are parameter-efficient too, but they sit in the forward path and add some inference latency, which the LoRA paper compares directly.
  • Prompt tuning / prefix tuning leave the weights alone and learn “soft prompt” vectors prepended to the input. Very few parameters, but they eat into sequence length and tend to be less stable than LoRA on harder tasks.
  • QLoRA, from Dettmers et al. 2023, quantises the frozen base model to 4-bit NF4 and trains LoRA on top. The paper shows fine-tuning a 65B model on a single 48GB GPU.
  • DoRA, Weight-Decomposed Low-Rank Adaptation, splits weights into magnitude and direction, learns direction with LoRA and magnitude separately. In PEFT it is just use_dora=True.
  • LoHa, LoKr, LoCon and friends come from the LyCORIS project and use Hadamard or Kronecker products for the low-rank factorisation. They are common in the image community, and according to the ComfyUI docs they load the same way as a regular LoRA.

Why LoRA took over image and video generation

LoRA was designed for language models, but it went mainstream in the Stable Diffusion community. The PEFT docs note it has become a very popular way to train diffusion models because it is so efficient. The reasons are practical:

  1. Video models are 5B to 14B parameters. Full fine-tuning is out of reach for most individuals and small teams, while LoRA puts training within range of a 24GB-class consumer card.
  2. A LoRA file is usually tens to hundreds of megabytes, which makes sharing far easier than passing around full models.
  3. One base model can pair with countless LoRAs, and you can stack them and dial their strength per generation.

How LoRAs are used in AI video generation

Style LoRAs

These push every frame toward one look: film grain, claymation, a specific illustration style, a colour grade. Image data alone often trains a decent style LoRA, because style lives mostly inside each individual frame.

Character and identity LoRAs

The classic video pain point is a character who looks different every time the camera cuts. Train a LoRA on one character across different angles, lighting and expressions, then put the trigger word you used in training into your prompt, and the character holds together across shots. This is the standard approach for short series and brand mascots.

Motion and camera LoRAs

This category only exists for video. It teaches the model a movement or camera move: a 360-degree orbit, a dolly zoom, a dance, a transition effect. Because the thing being learned is change over time, these LoRAs have to be trained on video clips. Still images cannot teach motion.

Speed (distillation) LoRAs

These have been especially popular over the past year. Wan step-distilled LoRAs released by lightx2v, and LoRAs extracted following the CausVid approach, let Wan models that normally need dozens of sampling steps produce usable video in 4 to 8 steps with CFG set to 1, which cuts generation time sharply. The trade-off is that motion can become less dynamic in some scenes and fine detail may not match a full-step run.

I2V versus T2V LoRAs

Text-to-video (T2V) and image-to-video (I2V) are usually separate model weights. Wan 2.2 A14B ships separate T2V and I2V models, and it uses a mixture-of-experts design with a high-noise expert (early steps, layout and motion) and a low-noise expert (later steps, detail). That is why Wan 2.2 LoRAs often come as a high/low pair, one for each branch. A LoRA trained on T2V and loaded onto I2V will often look wrong or fail to load at all, so check which base model a LoRA was made for before you download it.

Hands-on: the workflow for training a video LoRA

Preparing the dataset

  • Size: community guides commonly start character or style LoRAs with a few dozen images or roughly a dozen to a few dozen short clips. Quality and variety matter more than raw count.
  • Clip length and frame count: Wan models use frame counts of the form 4n+1, such as 33, 49 or 81 (81 is Wan’s usual setting). The official LTX-Video trainer uses resolution buckets written as width×height×frames, for example 768x768x49.
  • Captions: each image or clip gets a matching .txt file. For motion LoRAs, describe the movement and the camera move clearly. For character LoRAs, use one consistent trigger word, such as jw_girl.
  • Cleaning: strip watermarks, burned-in subtitles, black bars and hard cuts. For people, cover different angles and lighting, and avoid every sample sharing one background.

LTX-Video-Trainer chains these steps into an automated pipeline: split scenes, auto-caption with Qwen-2.5-VL, precompute latents and text embeddings, train, then convert the LoRA to ComfyUI format.

Picking a training tool

  • diffusers + PEFT: Hugging Face’s official stack, good if you want to write code and control the training loop.
  • musubi-tuner: from kohya-ss, supports LoRA training for HunyuanVideo, Wan, FramePack and other video models.
  • diffusion-pipe: a DeepSpeed-based pipeline-parallel trainer covering HunyuanVideo, Wan, LTX-Video, Flux, SDXL and more, configured with TOML files.
  • ai-toolkit: by ostris, comes with a web UI and supports Wan 2.x, including training both Wan 2.2 high-noise and low-noise experts from the interface.
  • kohya sd-scripts: the long-standing image LoRA tool, heavily used for SDXL and Flux.

Starting hyperparameters

These are common starting points from community guides, not official benchmarks. Treat them as a first attempt:

  • Rank: 16 to 32. Simple styles may only need 8 to 16.
  • Alpha: equal to rank, or half of it.
  • Learning rate: around 1e-4. For identity LoRAs some people go down to about 5e-5 to avoid plastic-looking skin.
  • Steps: a few thousand at most, saving checkpoints along the way so you can pick the best one rather than assuming the last is best.
  • Batch size: often 1 when VRAM is tight, with gradient accumulation to compensate.

VRAM requirements

Video LoRA training is much heavier than image training because every sample is a multi-frame latent. Training against a 14B-class model on a 24GB card usually means fp8 or lower-precision quantisation, block swapping (temporarily moving some layers to system RAM), cached text embeddings and a lower training resolution. Bigger cards (48GB, 80GB) can handle longer clips and higher resolutions. If your hardware is not there, renting a cloud GPU by the hour is often faster than fighting it.

Using LoRAs in ComfyUI

  1. Drop the .safetensors file into ComfyUI/models/loras/.
  2. Image models typically use the Load LoRA (LoraLoader) node, which patches both the diffusion model and the text encoder, with strength_model and strength_clip controls.
  3. Video model LoRAs (Wan, HunyuanVideo, LTX-Video) usually only touch the diffusion model, so the LoraLoaderModelOnly node is the common choice, with a single strength_model.
  4. Wan 2.2 A14B workflows have a high-noise and a low-noise model, so you need two LoraLoaderModelOnly nodes, loading the high and low versions of the LoRA into the matching sampler branches.
  5. Include the trigger word or the same style of description used in training in your prompt.

Strength: most LoRAs look natural between 0.6 and 1.0. Too high and you get broken frames, oversaturated colour and stiff motion. Too low and you can barely see the effect.

Stacking: nodes chain, so you can run a speed LoRA, a character LoRA and a camera LoRA together. Lower each strength a bit when stacking, because their effects add up and can conflict.

Merging: once you settle on a fixed combination, you can merge it into the base weights (save the merged model in ComfyUI, or use merge_and_unload() in PEFT). You save the runtime overhead but lose the ability to tweak strength or remove a LoRA later.

Code examples: PEFT and diffusers

Adding LoRA to a model with PEFT takes a few lines:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                 # rank
    lora_alpha=16,        # scale; effective multiplier is alpha / r
    lora_dropout=0.05,
    target_modules=["to_q", "to_k", "to_v", "to_out.0"],  # attention layers; names depend on the model
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

Loading an existing LoRA for inference with diffusers, using Wan as the example (swap in your real repo and file names):

import torch
from diffusers import WanPipeline

pipe = WanPipeline.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", torch_dtype=torch.bfloat16
).to("cuda")

pipe.load_lora_weights("your-name/your-wan-lora", weight_name="style.safetensors", adapter_name="style")
pipe.set_adapters(["style"], adapter_weights=[0.8])   # same idea as ComfyUI strength

video = pipe(prompt="jw_style, a cat walking on a rainy street at night", num_frames=81).frames[0]

set_adapters accepts several adapters with their own weights, which is LoRA stacking in code. Check the diffusers LoRA inference docs for the current API.

Pitfalls and limitations

  • Overfitting: train too long or on data that is too uniform and the LoRA bakes in the backgrounds, outfits and framing from your dataset, so they show up whatever you prompt. Save more checkpoints, add variety, and lower the learning rate or step count.
  • Strength too high: if you see artefacts, flicker or stiff motion, try dropping strength to 0.6 to 0.8 first.
  • Base model mismatch: LoRAs for Wan 2.1 and 2.2, 5B and 14B, T2V and I2V, or HunyuanVideo and HunyuanVideo 1.5 generally do not cross over. If you see key mismatch warnings or no effect at all, check versions first.
  • Speed LoRA trade-offs: fewer steps can mean less motion and detail. Rerun important shots at full steps.
  • Licensing and ethics: check the base model licence first (the Wan models are Apache-2.0, others may restrict commercial use) and make sure you have rights to your training material. Training a LoRA on a real person’s likeness needs that person’s consent, and should never be used for impersonation, deception or privacy violations.

FAQ

How is a LoRA different from a full checkpoint? A checkpoint is the complete set of weights and runs on its own. A LoRA is a patch and only works on top of the base model it was trained for.

Is a higher rank always better? No. Higher rank means more capacity, a bigger file and more overfitting risk. Most style and character LoRAs are fine at 16 to 32. Start small.

Can I use LoRAs without a high-end GPU? Using someone else’s LoRA only requires being able to run the base model. Training video LoRAs needs much more VRAM, so start with an image-based style or character LoRA, or rent a cloud GPU.

Why does my downloaded LoRA do nothing? Usually one of three things: wrong base model version, missing trigger word, or strength set too low. For Wan 2.2, also make sure the high and low files are on their matching branches.

Does LoRA work for language models too? Yes, that is where it started. PEFT plus QLoRA is one of the most common ways to fine-tune open LLMs on a single GPU today.

My take

The real point of LoRA is not just saving VRAM. It turns “customising a huge model” from training a new model into shipping a small file. For video work, that means style, character, camera movement and speed all become modules you can mix. My advice to anyone starting out is to hold off on training. Get a Wan workflow running in ComfyUI with an off-the-shelf speed LoRA plus one style LoRA, learn how strength and version matching behave, then train your first camera or character LoRA on about 20 clips you shot yourself. In my experience of reading other people’s results, the data and the captions decide the outcome far more than the hyperparameters do.

发表评论