megatron dpo训练Qwen3-Omni卡死

[Mryangkaitong]

单机8卡训练，卡在下面的位置，部分卡利用率一直为0

training ...
Overwriting rerun_state_machine.current_iteration from -1 to 0...
[before the start of training step] datetime: 2025-12-10 18:23:24
WARNING:DotProductAttention:flash-attn v3 may provide important feature support or performance improvement. Please install flash-attn v3 by
(1) git clone https://github.com/Dao-AILab/flash-attention.git
(2) cd flash-attention/hopper && python setup.py install
(3) python_path=python -c "import site; print(site.getsitepackages()[0])"
(4) mkdir -p $python_path/flash_attn_3
(5) cp flash_attn_interface.py $python_path/flash_attn_3/flash_attn_interface.py
WARNING:megatron.core.rerun_state_machine:Result validation enabled

[Jintao-Huang]

try use torch 2.8 (not torch2.9)

[Mryangkaitong]

try use torch 2.8 (not torch2.9)

降低到2.8后，报错如下，查了一下是说内存不够？我现在是单机8卡，h20

**************************************** [INFO:swift] Patch apply_rotary_pos_emb successfully applied. [INFO:swift] Patch _SplitAlongDim successfully applied. [INFO:swift] Patch FileSystemReader successfully applied. [INFO:swift] Patch peft successfully applied. [INFO:swift] megatron.core.__version__: 0.14.1 [INFO:swift] Successfully registered /data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/ms-swift/swift/llm/dataset/data/dataset_info.json. terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc W1211 15:26:41.374000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23356 closing signal SIGTERM W1211 15:26:41.375000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23357 closing signal SIGTERM W1211 15:26:41.375000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23359 closing signal SIGTERM W1211 15:26:41.375000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23360 closing signal SIGTERM W1211 15:26:41.375000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23361 closing signal SIGTERM W1211 15:26:41.376000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23362 closing signal SIGTERM W1211 15:26:41.378000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 23363 closing signal SIGTERM E1211 15:26:42.138000 23291 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 2 (pid: 23358) of binary: /data/miniconda3/envs/env-3.11/bin/python3.11 Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/run.py", line 905, in <module> main() File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main run(args) File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run elastic_launch( File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 143, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/miniconda3/envs/env-3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

我的启动脚本

`CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-"0,1,2,3,4,5,6,7"}
num_gpus=$(echo "${CUDA_VISIBLE_DEVICES}" | tr ',' '\n' | wc -l)

TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1
NNODES=1 NODE_RANK=0 NPROC_PER_NODE=$num_gpus ENABLE_AUDIO_OUTPUT=1 QWENVL_BBOX_FORMAT=legacy
USE_AUDIO_IN_VIDEO=True VIDEO_MAX_PIXELS=50176 FPS_MAX_FRAMES=12 MAX_PIXELS=1003520 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
megatron rlhf
--rlhf_type dpo
--load ${CONVERTED_MODEL}
--dataset 'data/dpo.json'
--torch_dtype bfloat16
--max_epochs 2
--lr 1e-6
--save ${OUTPUT_PATH}
--split_dataset_ratio 0.01
--train_type full
--load_from_cache_file true
--pipeline_model_parallel_size 2
--expert_model_parallel_size 4
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_permute_fusion true
--moe_aux_loss_coeff 1e-3
--micro_batch_size 1
--global_batch_size 8
--packing true
--freeze_llm false
--freeze_vit true
--freeze_aligner true
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 4
--finetune true
--cross_entropy_loss_fusion true
--lr_warmup_fraction 0.05
--min_lr 1e-7
--eval_interval 400
--save_interval 400
--vit_gradient_checkpointing true
--max_length 8192
--num_workers 8
--dataset_num_proc 16
--no_save_optim true
--no_save_rng true
--sequence_parallel true
--attention_backend flash`