Help making my voice puppet better?

I made this voice puppet for my kids and it works, but not very well. I took a couple audio example files and tried to modify them to make it work. I'm trying to do something like what is possible with Adobe Character Animator. I'm not expecting it to work that well, but maybe something a little better than what I have now. I've been working on it for awhile now, but I hit a wall. I really don't know what I'm doing. Can anyone help me?

Here's my puppet test.

Ok, so I made a small break through, and it works a little better now, but the head is completely off screen until you click on the "Get microphone input" button for some reason, and the movement is still a little jarring. I'm sure it's something simple, but I'm not smart enough to figure it out. Anyone help me with it?

Here's my latest version.

How can I compare the decibel level of the UserMedia mic? I don't see a way in the UserMedia conditions.

I think this is the best I can do with my limited knowledge and skills.

Test 8

If I could somehow Tween the height of the head to the decibel level changes, I think the voice puppet would work much better.

dop2000 I know you are smart enough to figure this out. Probably only take you a minute or two. I'll buy you some coffee if you'll help me.

Whoa, that's too much coffee! Thanks! :)

Let me know if you need help with anything else!

That works well. It makes sense the way you explained it, and looks so simple, but it's definitely something I would not have figured out on my own. Thank you! (Coffee sent.)

The effect I am going for is to have the puppet open his mouth to the cadence of the voice. Open and close with each syllable or word. A lot like the muppets. I have several ideas for it, one is to have a newscaster puppet react to random images that pop up on the screen as breaking news. I think hat would be fun for kids.

A more advance version would recognize each sound you make into the mic and replace the mouth shape of the puppet. I know C3 has a plugin for speech recognition, but I imagine that would be a lot of work to figure out how to make it all work. Do you think it's possible? Not that I could afford it, but if it is possible using the plugin, how much would that cost me?

A more advance version would recognize each sound you make into the mic and replace the mouth shape of the puppet.

I don't have much experience with speech recognition plugin, but I think this will be a nearly impossible task. The main purpose of this plugin is to recognizes words and phrases, not individual sounds. And it's not instantaneous, the recognized text may change several times while you continue speaking, so the puppet will not be able to "lip sync". Not to mention that many letters are pronounced differently in different words in English..

So reacting to volume change is probably the only option.

The puppet works good enough for me now. Thanks for all the help on it. Too bad the speech recognition plugin won't work for this. People have made auto lip sync plugins for other apps like After Effects before. Maybe someone will eventually make one for C3?

Thanks again!

You can analyze sound frequencies and match them with different puppet animations. Or there may be a JS library that does something similar..