Maybe game developers could introduce more shake into their games to make them feel more real?
Other than that: The tech is great. The videos certainly get a cinematic feeling with that kind of image stabilization.
I'm entirely impressed with some of the sample videos - I can think of numerous videos I've taken that would have benefited greatly from luma.io's stabilization.
It is pretty neat stuff.
I'd love to see a side-by-sire as well though.
Taken in aggregate, all of these pointers in the compressed data stream effectively show you which way the image "shook" relative to the previous image (and how far it shook) saving you CPU cycles to determine this yourself. You can even detect rotation. So all you need to do now is compensate for the shake by rewriting the compressed stream, "panning" (and perhaps rotating) in the opposite direction of the shake. In order to have room to pan, you need to emit a smaller rectangle than the original video.
This isn't sufficient for advanced stabilization, but it's a quick first pass.
I took a stab at writing this for the iPhone in 2010, but by that time the writing was clearly on the wall: Apple was soon going to offer this functionality in hardware (and they do, on the iPhone 4S and 5), and they would only do a better job with every phone refresh. The only way one can hope to compete is to perform global optimizations across the entire video clip that the hardware encoder can't do (e.g. dynamic programming), or else apply fancier transforms which are so CPU intense they kill the mobile experience. Good job on the part of the developers; the video looks great. As iPhone GPUs get more powerful stabilization algorithms will only get better.
One business angle here is to give the app. away for free and charge a dollar per video to deshake clips as a web service in the cloud.
It seems to take the approach of a post-processing frame shift rather than real-time stabilisation. In other words it lets the phone capture and compress using hardware, but keeps the sensor data separately and shifts the x,y of the video to stabilise. Unlike Final Cut etc. that has to analyse each frame, Luma has the raw data as the video was being shot. So the battery hit would come on export when the video has to be re-rendered I'm guessing.
I think this is the method because the Luma video appears more zoomed in than the original, i.e. they leave some space around so the video is always visible. This means that the Luma output will be lower res than the original hardware captured video I'm guessing.
Still, if it avoids walking around with your phone in a steadicam rig then it's a huge win. I have a steadicam smoothee but the ridicule from friends and family mean I don't use it, the slightly lower res output with Luma is the price my family will pay for their insensitive mockery!
Also, are there any plans for motion estimation for achieving super-slow motion, like Twixtor or Timewarp? That would be a neat effect for mobile video.
Digital stabilization will inevitably lose resolution because it needs to crop some data to even out the shake, and also because there is varying motion blur in the images that needs to be cleaned up somehow.
Like with zoom, optical is always a better solution because it fixes the problem before light hits the sensor, but of course it comes with its own tradeoffs involving physical size and production cost.
There's an example of it in action here, but it requires a login: https://developer.apple.com/videos/wwdc/2012/?include=520#52...
You can think of AVFoundation's as dampening handshake. We retarget the motion to be along the smoothest path possible. So we can often completely eliminate handshake. Luma supports both native and cinematic stabilization modes.
Here's an example of an actual algorithm used for digital IS: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.148....
Also, another thing that goes with shaky videos is bad or stuttering sound. If you guys handle both, I can easily see you becoming the go-to solution for filming on cell cameras.
