MFU
- How much FLOPS is translated into model training
- Usually around 20-60 MFU for LM pretraining depending on the quality
Model Weights: N billion parameters
- Standard:
- In PyTorch, by default, floating points are in fp32 — 4 byte each
- It then takes 4N GB memory
- It is almost always possible to use bf16 instead — 2 byte / parameter
- It then takes 2N GB memory
- In PyTorch, by default, floating points are in fp32 — 4 byte each
- QLoRA:
- In QLoRA, it’s possible to do 4bit quantization on model weight, so every parameter takes 0.5 byte
- It then takes 0.5N GB memory
Optimizer States:
- Usually we use AdamW, which requires 12 bytes each parameter
- fp32 copy of the parameter — 4 bytes
- Momentum and variance term — 2 x 4 = 8 bytes
- 8-Bit Optimizer
- It’s also possible to quantize momentum and variance into 8bit
- Therefore in total, 6 bytes / parameter
- Optimizer offloading:
- We can offload the optimizer to CPU, which, in tradeoff for more communication and therefore lower speed, it requires ~ 0GB VRam,
- Parameter-efficient Finetuning:
- We only update the LoRA layers, which is often negligible compared to the total model size. For estimation, just replace that with
Activation
Gradients
Parallelism
-
Tensor-Parallel
#TP
Column Splitting Tensor Parallel
- evenly splits each weight matrix into
#TPaccelerators. Each accelerator therefore only receives1/#TPmodel weight - How about activation?
- evenly splits each weight matrix into
-
Data-Parallel
#DP -
Context-Parallel
#DP -
Pipeline-Parallel
#PP
-
Distributed Optimizer
- You can distribute optimizer weights to each card