REDDIT 原始帖子
Best local model for analyzing video
I’m trying to make a tool that edits my shorts for me and from my understanding, last time I tried to use llama and Gemma it didn’t let me upload video anyone know the best local options for uploading video. So far the best llms I tested for video…
I’m trying to make a tool that edits my shorts for me and from my understanding, last time I tried to use llama and Gemma it didn’t let me upload video anyone know the best local options for uploading video. So far the best llms I tested for video understanding has been Gemini but I see some people saying qwen could be a good option , any tips or setups ?
已收录讨论
The thing that tripped me up: almost none of the local models "take a video". Gemini does the preprocessing for you server side, which is why it feels like it just works and llama/gemma feel broken. Locally you have to do that step yourself: pull frames on scene change, not fixed fps. fixed fps either misses cuts or floods your context run ASR for a timestamped transcript (whisper / faster-whisper) hand the frames plus the transcript to whatever VLM you like Qwen-VL is a reasonable pick for step 3, but step 3 is not where your quality comes from. For editing shorts specifically your cut points come almost entirely from audio: VAD boundaries and where sentences actually end. A VLM looking at 30 keyframes cannot tell you where to cut on a 200ms beat, a VAD can. One thing worth deciding before you write any of it: keep the real source timestamp attached to every frame you keep. If your pipeline extracts, dedups, then renames to frame_001.jpg, you have thrown away the mapping back into the source video, and the model can describe a shot but cannot tell you where it is. Parse ffmpeg showinfo on the same select pass and carry the PTS through every stage. frame_number/fps arithmetic drifts on variable frame rate footage, which most screen recordings and phone video are. For an auto-editor that mapping is the whole product, since the output you actually want is a cut list in source time.
Thanks this was insightful have you made an auto editor
Sort of, a rough one. The analysis half works: scene-change frames + timestamped whisper transcript go to the LLM, and it returns a list of segments worth keeping as in/out timecodes. Then ffmpeg just cuts from that list, no model involved in the actual cutting. The honest caveat: the LLM picks good moments but its timecodes are sloppy, often off by a second or landing mid-word. What fixed it was snapping every cut to the nearest silence gap in the ASR output and the nearest scene boundary. Before that step the output was unwatchable, after it it's decent. So if you build one, treat the LLM as the "what to keep" brain and let dumb signal processing decide the exact frames.
Qwen 3 Omni
Yolo26. Very efficient even on nano model