How do you do offline wake word detection when the user picks the phrase himself?
Been working on a local voice assistant for a while and the wake word is the part I keep getting stuck on. Not the easy version where you pick one of the pretrained models, I mean the user types in whatever phrase he wants and it just has to work. Offline,…
Been working on a local voice assistant for a while and the wake word is the part I keep getting stuck on. Not the easy version where you pick one of the pretrained models, I mean the user types in whatever phrase he wants and it just has to work. Offline, CPU only, since most machines I care about have no usable GPU. Right now I run a Vosk recognizer with a grammar that only knows my phrase and unk. That part is cheap and fires fine. The problem is the second step where I check if the hit was real. At first I made the normal decoder spell the phrase back and threw the hit away if it didn't match. Zero false wakes, looked perfect. But it also killed about a third of my real wakes, because a small offline model has no clue what your name is. I say Hey Nova and it writes back herum or hey oben or erhoben. Recall was around 32 percent, so you basically say your wake word four times before anything happens. What fixed most of it was to stop looking at the words at all. Now I only look at stuff like how many words came back, how long the speech actually was, and whether the decoder sounded sure that it heard some other normal word. If it doesn't know a word its confidence drops, if you say google or engineering it comes back super confident. That took me from around 50 to 74 percent on real recordings, and another phrase went from 36 to 66 with the same settings. False wakes went from 1 to 3 in about 1650 windows of room audio, which I can live with. But 74 is still not good. Hey Google works the first time and mine doesn't, and there's also a 0.6 second wait before it confirms that you can actually feel. So what do people here actually use for this. I keep reading that the real answer is to train a tiny model per user from a bunch of TTS clips of their phrase and run it as onnx on CPU. Does that really work with synthetic audio only, and how many voices do you need before it's usable. The other idea I had is to go phoneme level, so turn the phrase into phonemes and match on that instead of words, which would get rid of the whole out of vocabulary problem. Has anyone got something like that running offline? Also curious what hit rate you guys consider fine. Under 90 percent feels broken to me but I honestly don't know what's normal.
已收录讨论
Openwakeword is probably still the thing that people rely on for this. They have a link to a Google training recipe for custom wakewords which is pretty reliable in my experience.
Yeah vosk surprised me too, way faster than whisper at the same size. Do you do any second check on the grammar hit or just take it, mine false fires on room speech without a confirm step.
Yeah that's what keeps me off it, every user would need their own access key and a trip through their console just to set a wake word. Kind of kills it for something people are supposed to install and have working offline.
Yeah, room noise is the biggest problem, and sadly the main reason I don't actually use it much (I almost always have something playing on the TV). I initially had a wake word on the phone but ended up just sending all text to the desktop for processing since it's always going to be connected on the LAN and it's only text after all. I guess a headset mic or lapel mic would help but I really wanted the star trek experience, and I'm not there yet :)
microwakeword as well
Haven't looked at that one yet, thanks. That's the ESP32 one right, does it run fine off device on a normal desktop CPU too?
Yep, that's the one. You make the word phonetically, so as long as you can use English sounding phonemes for the word you want, you should be okay
Thank you for that
Porcupine works nicely, even runs on tiny embedded systems and has been well supported over the years. The drawback is that they've locked it down quite a bit by now. It probably requires a (quick) roundtrip through their backend to create a new wakeword, which would also be possible on-device - when not forced into that flow.
Good to know, that's the Colab recipe with the synthetic TTS clips right? Wondering how it holds up for a non English name, that's where my current setup falls apart.
Once in a while a random redditor stumbles on a question that just happens to land squarely in their realm of expertise and we get posts liked yours. Bravo.
Maybe make a base trigger nonoptional, like have the Hey/Hello/Bonjour/Dear or some other list of these as mandatory start words even if the rest of the trigger phrase is custom. Before trying to process Hey Nova, check if the user uttered "Hey", or one of the other mandatory start words which would be guaranteed to be in your vocabulary. Meanwhile, take the typed user phrase and generate N TTS variations of it and call them your templates. (N depends on compute as you will later use these to run pairwise comparisons to a new utterance) Use a frequency domain transform to breakdown the TTS templates, and later, a user's new utterance into features that represent the audio window. As features, you can use MFCC, LPCC coefficients or even just fourier coefficients although this with may suck compared to the other two which are meant for human voice frequency range. During use, calculate a distance between the uttered phrase's coeffs/features and the TTS template feature values. If the average distance is low, it's your phrase, if the average distance is high, it's something else. If you have the previous mandatory startword filter and the number of templates is small, the pairwise distance calculation between your template features and the new utterance will be fast. Test out different threshold values for the average pairwise distance check that give you the best result. (You can also create negative TTS templates for the user phrase, i.e. TTS of phrases of user phrase length with randomly swapped words of a certain edit distance. Then in the distance comparison step, you compare the user utterance to the positive templates vs negative templates. If it's low distance to both sets then it's ambiguous and don't trigger. basically synthetic KNN. You can also use SMOTE or similar oversampling techniques to generate more frequency domain features for each template set on the fly. Might as well train a model at some point though. edit: https://github.com/OHF-Voice/micro-wake-word seems to do a bunch of this and more
It is so neat to get taught things. Thanks!
I'm also using vosk but on android to avoid loading it locally when I'm running qwen, so my phone acts as the microphone, and it sends it to a service running on my desktop. I only use it in the opencode gui, which is an electron app so it was easy to inject text directly the right text boxes (it can do things like change models, agents, etc in the gui without any changes to opencode itself) Vosk is impressively fast and better than the whisper2 model of the same size in my tests.
On your last question, since nobody's answered it directly: I don't think 90 is unreasonable as a bar, and I think that bar is exactly why a lot of people quietly stop. Hey Google has one fixed phrase, an enormous amount of tuning, and hardware built around it. Arbitrary user-chosen phrase, offline, CPU only, non-English names is a much harder problem than the thing you're measuring yourself against. Being straight with you, we hit the same wall building a local voice assistant and ended up going push to talk instead. That's a goal change, not a solution, and I know it's the opposite of the Star Trek thing you and tinny66666 are both after, so take it for what it's worth. But it is a real data point on your "what's normal" question: the false-wake-from-the-TV problem never fully went away for us either, and at some point a held key was one less unsolved thing between us and the rest of the project. If the wake word IS the project, ignore all that and keep going, the per-user TTS template route people have pointed you at is the right next step. If it's a means to an end, it's worth knowing up front how much of your time it will keep eating. Your confidence-heuristic writeup is genuinely one of the better things I've read on this, for what that's worth. Publishing the false wake counts over 1650 windows instead of just a recall number is more rigor than most people bring to it.
这条评论已被删除。
I have to correct something: right now, you can't use it fully locally, but the wake-up word is local. I'm integrating Olaama right now, and the update will be shipped soon, in 1 to 2 days, so don't worry. The wake-up word is always local. That was the main reason.