Features
Sana
Training Sana now supported, requires very little config changes.
Example to make multi-training environment:
mkdir config/environment_namewhere environment_name may be something like the model name or concept you were working on.
- Example:
mkdir config/flux
- Move all of your current configurations into the new environment:
mv config/*.json config/flux - Run
configure.pyto create new configs for Sana mkdir config/sanamv config/*.json config/sana
When launching you can now use:
ENV=sana ./train.sh
# or
ENV=flux ./train.shNote: You'll have to adjust the paths to multidatabackend.json and other config files inside the nested config.json files to point to their location, eg. config/flux/multidatabackend.json.
Gradient clipping by max value
When using --max_grad_norm, the previous behaviour was to scale the entire gradient vector such that the norm maxed out at a given value. The new behaviour is to clip individual values within the gradient to avoid outliers. This can be swapped back with --grad_clip_method=norm.
This was found to stabilise training for runs across a range of batch sizes, but noticeably enabled more learning to occur with fewer disasters.
Stable Diffusion 3.5 fixes
The eternal problem child SD3.5 has some training parameter fixes that make it worth reattempting training for.
The T5 text encoder previously was claimed by StabilityAI to use a sequence length of 256, but is now understood to have actually used a sequence length of 154. Updating this results in more likeness being trained into the model with less degradation:
Some checkpoints are available here and the EMA model weights here are noticeably better starting point for use with --init_lora - note, this is Lycoris adapter, not PEFT LoRA. You may have to adjust your configuration to use lora_type=lycoris and --init_lora=path/to/the/ema_model.safetensors
SD3.5 also now supports --gradient_checkpointing_interval which allows the use of more VRAM to speed up training by checkpointing fewer blocks.
DeepSpeed
Stage 3 offload has some experimental fixes which allow running the text and image encoders without sharding them.
Pull requests
- support Sana training by @bghira in #1187
- update sana toc link by @bghira in #1188
- update sd3 seqlen to 154 max for t5 by @bghira in #1190
- chore; log cleanup by @bghira in #1192
- add --grad_clip_method to allow different forms of max_grad_norm clipping by @bghira in #1205
- max_grad_norm value limit removal for sd3 by @bghira in #1207
- local backend: use atomicwrites library to resolve rename errors and parallel overwrites by @bghira in #1206
- apple: update quanto dependency to upstream repository by @bghira in #1208
- swith clip method to "value" by default by @bghira in #1210
- add vae in example by @MrTuanDao in #1212
- sana: use bf16 weights and update class names to latest PR by @bghira in #1213
- configurator should avoid asking about checkpointing intervals when the model family does not support it by @bghira in #1214
- vaecache: sana should grab .latent object by @bghira in #1215
- safety_check: Fix gradient checkpointing interval error message by @clayne in #1221
- sana: add complex human instruction to user prompts by default (untested) by @bghira in #1216
- flux: use rank 0 for h100 detection since that is the most realistic setup by @bghira in #1225
- diffusers: bump to main branch instead of Sana branch by @bghira in #1226
- torchao: bump version to 0.7.0 by @bghira in #1224
- deepspeed from 0.15 to 0.16.1 by @bghira in #1227
- accelerate: from v0.34 to v1.2 by @bghira in #1228
- more dependency updates by @bghira in #1229
- sd3: allow setting grad checkpointing interval by @bghira in #1230
- merge by @bghira in #1232
- remove sana complex human instruction from tensorboard args (#1234) by @bghira in #1235
- merge by @bghira in #1242
- deepspeed stage 3 needs validations disabled thoroughly by @bghira in #1243
- merge by @bghira in #1244
New Contributors
- @MrTuanDao made their first contribution in #1212
- @clayne made their first contribution in #1221
Full Changelog: v1.2.1...v1.2.2
