FP8 Megatron training question

[chapter544]

Hi,
I am doing experiments with FP8 training using megatron on H100s, but I still have some questions (do not see in the documentations):

1. Does FP8 Megatron training reduces VRAM on GPUs so that I can increase batch-size or increase sequence-length?
2. Does the repo currently support LORA FP8 training and then directly merge/export LORA to FP8 checkpoints (without converting checkpoints from FP8->FP16 model and then FP16->FP8 for inferencing)?
3. Does FP8 training offer speedup in your tests? I setup a quick training run but the training speed is 1.5x-2x slower than FP16, I used the example script llm.sh in the fp8 folder of the repo.

Thanks,

[Jintao-Huang]

FP8 can accelerate large-scale model training (but usually not for small models and LoRA, as it involves additional precision conversion time, although matrix multiplication is faster).
LoRA can be trained by setting lora_rank as a multiple of 128, and supports LoRA export or merge-lora.

FP8 does not result in less VRAM usage.

[Jintao-Huang]

https://github.com/modelscope/ms-swift/tree/main/examples/megatron/fp8