ORIGINAL REDDIT POST
Would I be able to talk to an image generation model’s text encoder like a normal LLM?
Coming from the r/stablediffusion community, I’ve collected a bunch of models that are sitting on my drive like: Qwen_3_8b.safetensors qwen3vl_fp8_scaled.safetensors Mistral_3_small_flux2_fp8.safetensors Would be able to hook them up to a local harness and…
Coming from the r/stablediffusion community, I’ve collected a bunch of models that are sitting on my drive like: Qwen_3_8b.safetensors qwen3vl_fp8_scaled.safetensors Mistral_3_small_flux2_fp8.safetensors Would be able to hook them up to a local harness and make use of them like for light work like sillytavern or are these models only used for image generation?
Collected discussion
Yes those models are full LLMs. The image generation itself only uses the part of the LLM responsible for text encoding. Some image generation frontends allow you to run the LLM as an LLM to do things like generating/expanding T2I prompts. Most notably ComfyUI added a Generate Text node not long before Krea 2 released, and it's REALLY slow (10 tok/s decode from Qwen3-VL-4B BF16, mind you I have a mid-range Blackwell GPU which should probably do 4x that). But it saves you several gigabytes of VRAM because the same backend uses the same loaded LLM to do both the prompt expansion followed by text encoding for image generation. Before that node, people would (and STILL) use a stupid OllamaGenerate node or similar that would load a completely separate LLM into an external inference engine.
I'm pretty sure there are many prompt generators already. Basically, you just hook up something like qwen3.5-9b or similar sized model and explain to it in system prompt how to generate prompt for your image generation model, it'll take user prompt and generate prompt for you. With more sophisticated harness you can even feed a generated image back through image-to-text model (same qwen3.5-9b should be fine for that) to make loop to verify result and recreate if needed.