The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).
On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters…
On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters per token" (ternary weights and a granular MoE), the total capacity can grow without effecting the speed. Now the new catch: if speed is not the problem (a 105M and a 206M model run at the same tok/s (predicted 739–1309 tok/s)), will the model capacity scale with the number of params or the model will go "dumber" because of the lack of routing capacity with more experts? My measurements: On a Ryzen r5 3600X (single thread), the engine went from 176 tok/s to 848 tok/s on a 8.3M sandbox model with ternary LUT MLP, activation skip, deterministic SSM scan, two pool MoE with only a +0.00004 BPB quality cost. (Here the model is cache resident). I launched the full training on a 30M (11M active) model on the 2x T4 on Kaggle. Before the run I pushed 5 gates. Four passed and one failure: distilling from a bigger teacher using a different tokenizer lost against simply cross entropy (-0.0116 BPB, about 2.3 sigma worse), so I flipped the recipe to CE-primary. This is my idea of a 100% transparent project. (nothing above the 8.3M sandbox is trained yet, the 10B is the target) Repo in the comments
已收录讨论
Fair, you are right, but that was not the point. The difference is not the principle, is the fit. On GPU you must spend several thousands to achive what 300$ of DRAM holds
Thanks man, appreciate it
I honestly don't know what you're trying to do. your post's wording and architecture seems quite different to your repo.
Yes, but in text it keeps getting emphasized, thus implying behavior on GPU is different (it's not).
Fair, hte repo is a reseach jurnal, not a produbct page, so there is old stuff mixed in. The current thread is indocs/SCALEUP_ARCHITECTURE.md and the docs/PAHSE64_* docs. The post is simplified, I led with the bandwidth angle and olpy talked about the SSM + MoE half, The full design in the repo is a two tier split: a smalll SSM thinking core always in L3 and a streamed "knowing" tier. Is that the gap you saw or something else?
Great framing, the five sparsities are a cleaner split than what I had in my head, thanks. Where I sit: mostly #1 and #2, but a sharper version of #2. I'm not just keeping the dense head in fast memory, I'm co-designing it to fit in L3 cache (I measured a hard bandwidth cliff right at 16MB on Zen2), since on CPU the real wall is L3 vs DRAM, not RAM vs VRAM. The routed experts (your #3) stream from DRAM in bulk. And it's both: quant and sparsity are multiplicative here. Ternary shrinks the bytes per weight, the MoE shrinks the active weights per token, and neither one alone hits the target. One data point back on your #4, since it might interest you. At my toy scale (8 to 30M, TinyStories) I measured expert activations close to i.i.d., no hot or cold power law yet. So I deliberately don't bet on predicting hot experts. I treat any random per token gather as the CPU worst case and keep everything either resident or streamed in bulk. My guess is that your power law is a frontier scale or domain effect that just doesn't show up at my size, and I'm curious where the crossover is. I'll read the essay properly this weekend. And yeah, Colibri's expert streaming sits right next to my "knowing" tier, good pointer.
Dude. Just copy and paste it as-is from the llm without going back and mashing random typos into it to try and cover your tracks.
That’s exactly right. On average the experts activate with equal or near equal probability, by design/training constraint. But nobody sends a median prompt equally weighted to all of human knowledge. Within a prompt/domain they don’t. A big moe model is really a few dozen smaller specialized moe models traveling together in a pack. You need the specialization to get the IQ per weight advantage (to my understanding ).
Blliriant work! I'm really looking forward to seeing further progress of the project.
On CPU Why do you need to emphasize this? It is the same on GPU.
I'm interested in hearing more about hot experts. I expect that if you gave the training set back to the LLM, all the experts would have roughly the same duty cycle. If they didn't it would imply that the density of weights is misallocated across the training set, which should be impossible unless there are flaws in the training method. But I wondered if, for a particular prompt, the same experts tended to be used repeatedly such that an MRU on VRAM would produce reasonable speeds. Is that what you've found, essentially?
oh. I think I got what you mean. you're saying experts are retrieved from approximate nearest neighbors methods like ivfpq, and you would like to scale the amount of experts. I think if you treat experts as memory then you might want to look into modern attention based hopfield network, because that line of research is more established. my 2 cents.
Good work. This is the kind of "I wonder" research we need. Deep down, I think we all realize models aren't running as efficiently as they possibly can (or as densely as they can). Explore those edges and boundaries!
The repository: https://github.com/WildPino/SiliconLLM
Translation (I think): Thinner experts but more active at the same time to get less active parameters overall to use idle compute for smarter routing
I’ve become fascinated with the physics of moe inference the last few months. I’ve written up what I refer to as the “five sparsities” on my GitHub repo. That section is approximately a self contained essay: https://github.com/tedtieken/Speedwagon-Inference Each of the sparsities has unique opportunities for engineering. It’s not clear to me which sparsity you’re focused on, (and how much your work depends on quantization vs sparsity) but I think all 5 will be relevant to you. It’s useful to have them named and disaggregated, or at least it was for me. My library code is still late prototype and rough, so not trying to promote that yet, but I haven’t written up the essay anywhere else. Colibri is doing similar work on “cpu inference for moe” and might be relevant to you as well.
Looks like you are deep in it. I will need a discussion later as I am trying to increase the speed of generation mobile phones. Saving this post so I can comeback with my material.
bro i rather translate what you give then hear what the AI has to say.