I've made a quick update, please see the link in the video description!
Python only connects the different components together. Even if you convert my script to C you won't get much of a speed improvement, because it's not the bottleneck. The speech processing is, which is done by Julius, which *is* written in C.
You'll have better luck enabling hardware floating-point and gcc optimizations for the entire system at compile time, which is what the community is doing now.