Bokehlicious Selfies (opens in new tab)

This is really cool, but it isn't bokeh, it's just blur.

To make the Gaussian blur look better, it needs to be applied to linear lightness, not to gamma-encoded colors, as it is done here. In reality, the blur is induced by the lens, on linear lightness, and only gamma-encoded afterwards.

The effect being that blurred colors mix wrongly, as you can see where the red from the flower mixes with the green. The transition looks wrong, and not how a lens would render it.

This is a quick proof of concept of an idea. This won't work for every use-case but it worked surprisingly well for a use-case I had in mind. Happy to answer any questions.

I just discovered that Zoom teleconferencing software has the ability to detect the background and replace it with an arbitrary video in real time. I have no idea how they do it, but it's as impressive as heck.

It would be nice if there was some more information about where the mask comes from. We want a segmentation map for people, so this technique basically takes the layer activation map for the "person" class - which is id 0 in COCO - and you can threshold that to get foreground/background. If you changed the mask index, this would respond to other object types (so as written this code will only work for humans).

--

By the way, Google does some absolutely nuts stuff with this on the Pixel 3 and 4 - they actually calculate a stereo depth map using the two autofocus sites in individual pixels. Essentially some modern CMOS sensors use a technology called dual pixel autofocus (DPAF), by measuring the response from two photodiodes in the same pixel, you adjust focus until each pixel has the same intensity (more or less). If the camera is out of focus, the two photodiodes will have different intensities.

However what this gives you is two separate images with an extremely small (but detectable) parallax which can be used to give coarse 3D reconstruction, and you can segment foreground and background. It's nice because you get a strong physical prior, rather than having to worry about using a convnet to identify fore/background regions. (They of course apply a convnet anyway to refine the result).

https://ai.googleblog.com/2018/11/learning-to-predict-depth-...

https://ai.googleblog.com/2019/12/improvements-to-portrait-m...

Awesome! Computational photography is fascinating. A while back, I wrote some OpenGL shaders to do the same thing, and got stuck when I wanted variable bokeh based on scene depth. This is easy when you have the depth map (say, in a renderer). I recall reading about how Apple achieved this using stereo disparity mapping, which is great if you have the hardware. I hadn't considered a neural net approach like yours for auto-generating a depth map, although I now that I look into it, Facebook recently published research on just that!

https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-p...

One problem with bokeh is that it disappears when you scale the image down. Perhaps this technique can help solve that (but I suppose there would be better ways to do this).

Anyway, the entire technique is also very easy to do manually with Photoshop.

One step further would be to incorporate the blurring, masking, and combining operations into the network itself, so you can leverage the GPU to do the whole computation in one fell swoop.

Ain’t no lens that would blur the front brim of her hat and her face is clear like that. Disregarding the nature of the optics you are emulating ruins the value of the effect

OT: I can't understand why this website _want_ to looks like a mobile web __and__ hides scrolling bar. I'm on desktop and it irritates me a bit.

I think I'm missing the part of my brain that lets me understand why bokeh selfies are even a thing. If you're going to remove your surroundings from the picture, why even bother taking a new picture?

His original photo looks way nicer compared to the processed one.

Page is broken with no scrollbar.

if we're using deep learning anyways, might as well use a u-net (ie. pix2pix) to add the bokeh directly - this would require pair training data but would probably look a lot better visually.

website has no scrollbar??

Wow this looks ugly :D. But still nice work. It's just that the girl is totally over-processed and the "bokeh", well I hate to break it to you, but this isn't bokeh, its called "blur".

This is bokeh: https://cdn.mos.cms.futurecdn.net/JgrDuxQPCAvgd5VzqiKN5a-650...

And bokeh is 3 dimensional and surrounds the focus area, increasing with distance: https://www.flickr.com/photos/scottcartwrightphotography/145...

I suggest to brush up a bit on photo skills, although I agree that the instragram and iPhone culture can shift your perception of "bokehlicious" quite a bit.

I don't want to make you feel bad, so I also tell you that Apple's "bokeh" in portrait mode looks disgusting, doesn't make any sense and breaks the image in a lot of trivial cases... and they spent a lot more money & effort on it than you :).