REDDIT 原始帖子
I want a service that can remotely start and stop a model before and after use
local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this…
local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.
已收录讨论
Llama swap?
That’s not true on many nvidia cards (especially data center cards). On my v100 cards if they have a model (or comfyui context) loaded, it prevents then from going into the lowest power state / slowed clocks. Idle power goes from 35+ watts to the 26ish range. With multiple cards in my machine it makes a difference at true idle. Whole rig goes from 170 watts to 140watts when my scripts enable complete model offload (I set a quiet time of the day from 2am to 9am). They will load on the first request but it will make that first time to token pretty slow but it’s a good trade off when the rig won’t be used much at those times. Have measurements here: https://github.com/bradrlaw/ai-server
interesting! thanks for that. every bit starts to add up.
wow i didn't know it was already inside llama.cpp. That's awsome thanks.
interresting. does it use llama.cpp as a backend? Its not immediately clear "llama-server" could be it.
llama.cpp can do this natively with it's router mode. You can set up a config.ini with all the model settings, it'll load the model when the API requests, and you can set a timeout period for unloading it.
This works great for me too
You can also set the maximum number of models in memory to 1 as a runtime flag, no more oomkills or CPU offloaded models!
This is build in into LM Studio. Start a server, it will expose all models available. You can configure the model to release itself after some idle time.
While it’s in memory it barely uses more energy. Your overall system idle draw could be very high. Even if it’s around 70w you look at 50kW a month without inference. So Wake on lan could be an option. Also turning on by time when everyone is sleeping
Model uses no energy if it is not processing prompt.
Llama swap works.
GPU's don't consume much power when idle. I think my RTX 6000 Pro uses like 4 watts at idle and RTX 5090 uses only like 6-7 watts at idle, this is with processes like llama.cpp and comfyui attached and have memory reserved on the GPU.
Sounds like you should copy this post, paste it in your agent’s chat box, hit enter and see what happens… But in short: JIT loading in most runtimes can either evict a model after idle time or evict on loading another model. This is configurable in almost all runtimes you may use, from vLLM to lmstudio. FWIW, auto-restarting after crashes and constant serving is what I needed, almost the opposite. I went ahead and asked an agent, who set up some nice automation scripts and watchdogs to do this.
llama-server exposes endpoints through the http api, i generated a script for this. Its rather simple, so i did not bother to install a tool for this
llama-router can unload a model after it's not in use for a while automatically. Also being loaded doesn't mean it's cranking the GPU, it's just resident in memory.
Llama-server does that just fine
Llama swap. You can click a button to start, and click a button to unload. But I need to tell you that there is no noticeable energy consumption if the model sits unused on your GPU. Unless you talk about physically turn off the desktop. In that case, I saw some people running a raspberry pi 4 as a relay to turn on turn off electricity to the desktop. There should be a safe way to not break of the OS on the desktop with this design.
How much dos your gpu consume? I am just curious. My 5060 Tis consume 4-6 watt no matter if a model is loaded.or not.
ollama
llama.cpp /load /unload API endpoints?..
Do you mean load/unload/swap models or energy saving by suspending the server when it's idle? Load/swap models is relatively easy and included in llama.cpp, LM Studio server, and others. Suspending your inference server on idle is far more complicated. I started working on this on the Ubuntu side of my LLM server. It looks at GPU utilization, ssh and http connections, and probably others I don't recall atm. If utilization is idle, it notes the running llama.cpp arguments, saves them, stops (unload) llama.cpp, and suspends the server. Note: GPUs don't 'suspend', they power off so a loaded model would be copied to disk, if anything. On the client, send a WoL packet when needed, the server starts, loads the last llama.cpp instance back again. It's been a few months since I tried this so I'm likely forgetting details. It wasn't easy to setup, especially the part about detecting 'idle'. IIRC, I suspend after 60 mins of idle.
You can run Mastra locally with your local models, then use Signals and Inbox to start and stop the agent remotely. https://mastra.ai/docs/agents/signals
You're only using significant electricity when you run inference on the model, otherwise the GPU idle consumption isn't much for consumer cards like the 30/40/50 series ones. Holding the model in VRAM is effectively free.
I built QikLM exactly for your use case. It's free. it makes it easy to use your own llama.cpp build, manage your models, has an API endpoint so you can use it with anything and good built in chat interface. It has always on mode and it has Auto session so after 10 minutes inactivity, it will stop the llama server automatically. you can change the time out length. it even supports vllm. https://qiklm.ai