Library / SDK
Fine-tuning & Training
GPT-NeoX
1EleutherAI's 2library 3for 4large-scale 5model 6training
the six Ws · specification
W1
Who
EleutherAI develops GPT-NeoX as a large-scale model-parallel training library.
W2
What
GPT-NeoX trains autoregressive transformer language models across hundreds of GPUs using Megatron- and DeepSpeed-based 3D parallelism.
W3
Where
Hosted on GitHub under the EleutherAI organization.
W4
When
First released in 2021 and used to train models such as Pythia and GPT-NeoX-20B.
W5
Why
Built to give academic and industry labs a scalable, open path to training frontier-scale LLMs.
W6
With
Runs on Slurm, MPI, or Kubernetes clusters, built on PyTorch, Megatron-LM, and DeepSpeed.
W7
Watch
Open source under Apache 2.0, self-hosted, no credentials required; maintained by EleutherAI.
model-parallelismDeepSpeedlarge-scale-trainingMegatronautoregressive-LM