ORIGINAL REDDIT POST
Does anybody have a favorite training system?
I am a college student, and I am currently experimenting with making smaller models more efficient and useful, but I am currently using a slow custom system and need something more professional.
I am a college student, and I am currently experimenting with making smaller models more efficient and useful, but I am currently using a slow custom system and need something more professional.
Collected discussion
Can you add any details? Purpose, budget, etc etc I would assume pytorch for most things though for training.
I'm mainly looking for a training framework rather than just a library. My goal is training small language models (roughly 10M–1B parameters) with features like mixed precision, gradient accumulation, checkpointing, distributed training, and good logging. I'm already using PyTorch underneath, but I'm interested in hearing what people use on top of it (Lightning, Hugging Face Accelerate, DeepSpeed, MosaicML Composer, etc.) and what they've had the best experience with.
Also a student, but not in the LLM field. I use base pytorch most of the time, sometimes HF accelerate. For "mixed precision, gradient accumulation, checkpointing, distributed training", base pytorch or HF Accelerate works well. For good logging, tensorboar works well enough for me.
For very simple training, TRL is short and sweet. It takes more manual effort to format the training data than it does to actually apply it. For anything more complicated, I like Unsloth.
i like axolotl or trl
Unsloth generally has nice workbooks to start from as examples, so I go with those most of the time.
Depends on what field you're working in? Most non-hobbyist training I've worked alongside has been via ArcGIS Pro. That is probably not what most people in here are using and was not originally not even designed to handle LLMs at all, but these days it allows for some extremely interesting applications, particularly with vision-enabled models.
Back in the day, axolotl and unsloth were very appreciated. I think they are still sound.
Are you training LLMs? Then I would recommend unsloth studio
depends whether you're fine-tuning or training from scratch, and people conflate those a lot. fine-tuning: axolotl or unsloth. axolotl is a yaml config and you're running, handles most of the annoying stuff for you. unsloth if you're on one consumer GPU and hitting memory walls, the speed and VRAM savings are real. llama-factory if you want a UI. torchtune is cleaner code if you actually want to read what's happening instead of trusting a config. from scratch on small models: nanoGPT. it's tiny, readable, and you'll learn more from it than any framework. still what i'd point someone at for understanding. underneath all of these it's just huggingface accelerate or deepspeed doing the distributed part, so knowing that layer helps when something breaks. honestly though, keep your custom system around. once you switch to a framework you stop seeing the parts that were teaching you things. the slow version being slow is usually informative about where the time goes.