feat(nn): add configurable norm_layer to Mlp#1702
Conversation
Add a norm_layer parameter supporting LayerNorm, TE-aware norms, and custom factories while keeping use_batchnorm backward compatible. Closes NVIDIA#1451 Signed-off-by: Thabhelo <50872400+Thabhelo@users.noreply.github.com>
Greptile SummaryThis PR adds a
Important Files Changed
Reviews (1): Last reviewed commit: "feat(nn): add configurable norm_layer to..." | Re-trigger Greptile |
Split layernorm (PyTorch LayerNorm) from te_layernorm (requires TE and CUDA), store norm_layer on the module, and extend tests accordingly. Signed-off-by: Thabhelo <50872400+Thabhelo@users.noreply.github.com>
|
Okay cool, I have addressed the feedback in the latest push:
Added tests for the TE string path and the stored attribute, too. |
|
Greptile feedback from the first review is addressed in 8896888. Replies are on the inline threads. docs come from the updated Mlp docstrings via Sphinx automodule. |
|
@loliverhennigh gentle ping when you have a moment. Greptile feedback is addressed in 8896888. I will rebase onto main shortly to clear the merge conflict. Thanks! |
PhysicsNeMo Pull Request
Description
Adds a
norm_layerparameter tophysicsnemo.nn.Mlpso normalization type is configurable instead of being limited toBatchNorm1dviause_batchnorm.Supported options:
norm_layer="batchnorm","layernorm", or"te_layernorm"norm_layer=nn.LayerNormorget_layer_norm_class()for TE-aware LayerNormnorm_layer(out_features)use_batchnorm=Trueremains backward compatible and is mutually exclusive withnorm_layer.Closes #1451
Checklist
physicsnemo.nnlayer change, notphysicsnemo.models.)Notes
norm_layeris documented in theMlpclass docstring and doctest examples; Sphinx exposes it viadocs/api/nn/layers/fully_connected_mlp.rstautomodule.Dependencies
None.
Test plan
pytest test/nn/module/test_mlp_layers.py -vpytest --doctest-modules physicsnemo/nn/module/mlp_layers.py