使用megatron 微调FB8模型问题

[yugenlgy]

硬件：80G*8
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
NPROC_PER_NODE=8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
megatron sft
--model /test/weights/Qwen3-Coder-30B-A3B-Instruct-FP8
--train_type lora
--lora_rank 16
--lora_alpha 32
--target_modules all-linear
--load_safetensors true
--save_safetensors true
--fp8_recipe delayed
--fp8_format hybrid
--fp8_param_gather true
--dataset /test/ml/CodeAlpaca-20k.jsonl
--load_from_cache_file true
--tensor_model_parallel_size 1
--expert_model_parallel_size 8
--moe_permute_fusion true
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-6
--micro_batch_size 2
--global_batch_size 16
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 1
--finetune true
--cross_entropy_loss_fusion true
--lr 1e-5
--lr_warmup_fraction 0.05
--min_lr 1e-6
--save megatron_output/Qwen3-Coder-30B-A3B-Instruct-FP8-result
--eval_interval 200
--save_interval 200
--max_length 2048
--num_workers 8
--dataset_num_proc 8
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--moe_expert_capacity_factor 2
--use_precision_aware_optimizer true
--exp_avg_dtype bf16
--exp_avg_sq_dtype bf16
--attention_backend flash
--model_author swift
--main_params_dtype fp16
--model_name swift-robot
--merge_lora false

关键日志
[INFO:swift] [rank0] model_parameter_info: PeftModelForCausalLM: 5253.8388M Params (88.8668M Trainable [1.6915%]), 0.0000M Buffers.

number of parameters on (tensor, pipeline) model parallel rank (0, 0): 5253838848
[after model, optimizer, and learning rate scheduler are built] datetime: 2025-12-10 19:17:57
building train, validation, and test datasets ...
[after dataloaders are built] datetime: 2025-12-10 19:17:57
done with setup ...
(min, max) time across ranks (ms):
model-and-optimizer-setup ......................: (42193.62, 42195.00)
train/valid/test-data-iterators-setup ..........: (0.74, 1.14)
training ...
Overwriting rerun_state_machine.current_iteration from -1 to 0...
[before the start of training step] datetime: 2025-12-10 19:17:57
[INFO:swift] The training of Epoch 0 starts...
WARNING:DotProductAttention:flash-attn v3 may provide important feature support or performance improvement. Please install flash-attn v3 by
(1) git clone https://github.com/Dao-AILab/flash-attention.git
(2) cd flash-attention/ && git checkout 3ba6f82 && git submodule update --init && cd hopper/ && python setup.py install
(3) python_path=python -c "import site; print(site.getsitepackages()[0])"
(4) mkdir -p $python_path/flash_attn_3
(5) cp flash_attn_interface.py $python_path/flash_attn_3/flash_attn_interface.py
WARNING:megatron.core.rerun_state_machine:Result validation enabled
Number of parameters in transformer block in billions: 29.90
Number of parameters in embedding layers in billions: 0.62
Total number of parameters in billions: 30.52
Number of parameters in most loaded shard in billions: 30.5197
Theoretical memory footprints: weight and optimizer=218294.09 MB
[2025-12-10 19:18:34] iteration 1/ 1251 | consumed samples: 16 | elapsed time per iteration (ms): 36464.7 | memory(GiB): 12.5 | elapsed time: 36s | remaining time: 12h 39m 40s | learning rate: 1.598721E-07 | global batch size: 16 | lm loss: 1.463566E+01 | load_balancing_loss: 3.929863E+00 | loss scale: 1.0 | grad norm: 0.000 | number of skipped iterations: 0 | number of nan iterations: 0 |

1、这个 5253.8388M是如何计算出来的？
2、为什么GPU显存占用这么高？每个GPU都在20G左右我的目目标是将Qwen3-Coder-30B-A3B-Instruct-FP8模型进行DP=1，EP=8进行，显存不用占4-5G左右吗？FP8在微调时候，模型加载到显存时候，会变成float16/bfloat16吗？我只想对FP8类型进微调
3、我最终是想在这8张卡上微调Qwen3-Coder-480B-A35B-Instruct-FP8模型，如果仅仅只是修改--model参数，就会报OOM问题，该如何修改训练脚本？

[Jintao-Huang]

5253.8388M 是单个进程中的参数

FP8训练通常比BF16显存消耗更多些

[Jintao-Huang]

blockwise fp8，需要lora_rank为128的倍数

[yugenlgy]

请问：
1、为什么“FP8训练通常比BF16显存消耗更多些”？难道需要数据类型转换吗？有相关文章吗？谢谢。
2、“blockwise fp8，需要lora_rank为128的倍数”，我将 fp8_recipe参数设置为blockwise？是这个意思吗？
fp8_recipe: 用于前向和反向传播中 FP8 张量的 FP8 算法方案。可选为'tensorwise', 'delayed', 'mxfp8', 'blockwise'。默认为'delayed'。其中blockwise fp8需要 cuda129 以上版本。

[yugenlgy]

我现在想单机80G*8 lora微调 Qwen3-Coder-30B-A3B-Instruct-FP8，方案行不通？
通过以上参数 “blockwise fp8，需要lora_rank为128的倍数” 设置可以吗？
目前在模型加载阶段，就报OOM错误了。

[Jintao-Huang]

https://github.com/modelscope/ms-swift/tree/main/examples/megatron/fp8

[yugenlgy]

您发的这个办法我尝试了，只是 CUDA Version: 12.4 ，没有按照要求升级到 cuda12.9
在模型加载阶段就报OOM错误
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 60.00 MiB. GPU 0 has a total capacity of 79.10 GiB of which 56.19 MiB is free. Process 3126122 has 1.90 GiB memory in use. Process 142013 has 76.97 GiB memory in use. Of the allocated memory 74.74 GiB is allocated by PyTorch, and 986.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) when instantiating TEColumnParallelGroupedLinear when instantiating TEGroupedMLP when instantiating MoELayer when instantiating TransformerLayer

以下是训练参数 ，还将训练变成了lora，lora_rank 128，还是报OOM异常。
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
NPROC_PER_NODE=8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
megatron sft
--model /test/weights/Qwen3-Coder-480B-A35B-Instruct-FP8
--load_safetensors true
--save_safetensors true
--train_type lora
--lora_rank 128
--lora_alpha 256
--target_modules all-linear
--merge_lora false
--fp8_recipe blockwise
--fp8_format e4m3
--fp8_param_gather true
--dataset /test/ml/CodeAlpaca-20k.jsonl
--load_from_cache_file true
--tensor_model_parallel_size 2
--expert_model_parallel_size 8
--moe_permute_fusion true
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-6
--micro_batch_size 4
--global_batch_size 16
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 1
--finetune true
--cross_entropy_loss_fusion true
--lr 1e-5
--lr_warmup_fraction 0.05
--min_lr 1e-6
--save megatron_output/Qwen3-Coder-480B-A35B-Instruct-FP8-lora
--eval_interval 200
--save_interval 200
--max_length 2048
--num_workers 8
--dataset_num_proc 8
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--moe_expert_capacity_factor 2
--use_precision_aware_optimizer true
--exp_avg_dtype bf16
--exp_avg_sq_dtype bf16
--attention_backend flash
--model_author swift
--model_name swift-robot

[yinghuozijin]

5253.8388M 是单个进程中的参数

FP8训练通常比BF16显存消耗更多些

FP8占用更多显存？？？ 那我显存不够到底该怎么办？

[yugenlgy]

https://zhuanlan.zhihu.com/p/694352617 这个上边说 GPU 架构下的 FP8 训练与推理，没有看懂是否需要权重转换。
我有个疑问，在训练FP8模型时候，需要模型权重转换，根据torch_dtype这个参数决定？
我猜想Qwen3-Coder-480-FP8 FP8转成bfloat16（权重文件中"torch_dtype": "bfloat16"），FP8训练仅在GEMM计算阶段使用FP8精度，而权重、梯度和优化器状态均以高精度（BF16/FP32）存储？
所以“FP8训练通常比BF16显存消耗更多些”，是因为以上原因？