Library / SDK
Fine-tuning & Training
Megatron-LM
1Train 2giant 3transformers 4at 5massive 6scale.
the six Ws · specification
W1
Who
Built by NVIDIA.
W2
What
Implements tensor, pipeline, and sequence parallelism for large transformer training.
W3
Where
A Python framework on multi-GPU clusters.
W4
When
When pretraining very large models across many GPUs.
W5
Why
Reference implementation for large-scale model parallelism.
W6
With
PyTorch, CUDA, and NCCL.
W7
Watch
NVIDIA open-source, permissive license; needs a GPU cluster, no credentials.
NVIDIAParallelismGPU