Skip to content

feat(pp): support FusedLinearCrossEntropy under pipeline parallelism#2927

Open
beccohov wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
beccohov:beccohov/feat/fince-under-pp
Open

feat(pp): support FusedLinearCrossEntropy under pipeline parallelism#2927
beccohov wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
beccohov:beccohov/feat/fince-under-pp

Conversation

@beccohov

@beccohov beccohov commented Jul 5, 2026

Copy link
Copy Markdown

What does this PR do ?

Makes FusedLinearCrossEntropy actually run under pipeline parallelism instead of
silently falling back to MaskedCrossEntropy on every PP run (#2681).

Changelog

  • recipes/llm/train_ft.py: run the loss-capability check on the real last-stage
    module, not the AutoPipeline wrapper (which defines no forward, so the check
    always failed and silently swapped FusedLinearCrossEntropy for
    MaskedCrossEntropy). Extracted into _maybe_downgrade_loss_fn.
  • recipes/llm/train_ft.py: gate fused CE under PP to the generic patched forward —
    MoE/custom-forward models (e.g. deepseekv3) fall back to MaskedCrossEntropy with a
    clear warning instead of crashing; flag the last stage to emit hidden states when
    fused CE is active.
  • distributed/pipelining/hf_utils.py: the last stage skips the lm_head projection
    and returns hidden states when flagged; mark the model fused-CE-capable only when the
    generic forward is installed.
  • distributed/pipelining/functional.py: size the last-stage output as
    [mb, seq, hidden] instead of [mb, seq, vocab] on the fused path.
  • components/loss/mtp.py: PipelineCausalLMLoss treats a bare last-stage tensor as
    hidden states for FusedLinearCrossEntropy (running the fused lm_head+CE) and raises
    a clear error for the unsupported MTP-tuple + fused-CE combination.
  • Tests: forward emits hidden states; stage output shape switches to hidden; pipeline
    loss routes the bare tensor to hidden states; capability/gate truth table.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?

Additional Information

…nder PP

Signed-off-by: Arkadii Be <beccohov@gmail.com>
@beccohov beccohov requested a review from a team as a code owner July 5, 2026 13:23
@copy-pr-bot

copy-pr-bot Bot commented Jul 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@beccohov

beccohov commented Jul 5, 2026

Copy link
Copy Markdown
Author

Hey @akoumpa,
as promised -- here is the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants