Object-recognition dataset stumped the world’s best computer vision models (opens in new tab)

The thing about AI/CV and other interpretation simulators is that there is always a quantization of nature in the end result. This is why the Uncanny Valley exists, "Uncanny Valley" being a term that refers to the difference between nature and the model itself, and I'm pretty sure it can't be engineered out of the technology. That is, the simulator can not picture an object from any angle, only an angle greater than its mathematical-precision limits. Analog vs. digital doesn't always have to be about audio. ;)

So, "construct an internal 3D model of [something in the natural world]" will always be deficient, and any conclusions derived from these models will always have inherent errors (even before you get to algo bias). Self-driving cars, airport face-reading gates, Pixar blockbusters...their models can never represent reality in anything but a temporarily-convincing way. Those that affect policy and peoples' lives (aka not-entertainment) will always come up with the wrong conclusion sometimes, sometimes fatally.

https://en.wikipedia.org/wiki/Uncanny_valley

I recently saw a paper (which I reposted here on HN a while back) that perhaps could help with your problem. The paper proposes an approach that sort of induces models to learn representations of objects that are good at predicting "the most agreed-upon" rotations of the inputs. I'm not explaining it well. Anyway, a model in the paper achieved SOTA on a change-of-viewpoint dataset with a really tiny number of parameters. Might be worth a look: https://arxiv.org/abs/1911.00792

Decades ago, computer vision was primarily thought of as a kind of inverse graphics, the opposite of rendering: image goes in, 3D shapes with material properties and lighting properties go out. Of course 3D reconstruction is still a huge thing, but object recognition split off onto a different path with the realization that "superficial", 2D-based features (like SIFT and HOG) work very well for recognizing image content, when combined with powerful classifiers and regressors of the time (like the SVM). It was common for lecturers to say that "You may think we need complicated internal 3D representations of everything, but another approach seems more fruitful: ..." Nowadays there's a lot of buzz around merging the two branches back together to unify explicit 3D geometric reasoning and 3D modeling with deep learning.

I think humans use both strategies. Sometimes we really rely on superficial visual information, like yellow/black stripes -> time to get away! No need to first perfectly match the visual input to a mental tiger model rotated at the correct orientation. I think split-second recognition is usually like this. Or perhaps we use different strategies for different objects, I could imagine for example that facial recognition in the brain is more 2D-feature based pattern matching, rather than 3D reconstruction.

For contrast, horse brains seem to not do this. They'll often see the same object from a different angle as a new stimulus.

> It seems to be how the human works. I can rotate an object in my mind and picture it from any angle.

You can, but that doesn't seem to me to be how the mind works. If it looked at it from every angle simultaneously there would be no speed difference in my recognising an object regardless of the orientation.

But in some cases there are a huge speed difference. I can be staring at it for many seconds and then snap! - oh it's upside down. As soon as I realise this my mind immediately adapts and what was unrecognisable is suddenly as plain as day.

The only way I can account for this is when I finally twig the image is upside down I re-route it through a different path in my brain that does a rotation before feeding it to the recogniser. But normally that path is shut off - it's not constantly scanning the input.

I suppose what happens is in most cases some pre-processor uses other clues in the picture to tell me it's rotated from it's normal position and engages the correct path without conscious intervention. That would explain why most of the time you say you don't notice it.

Nonetheless the two mechanisms are very different. A sequential path that rotation -> recognition will be slightly deeper and slightly slower than that does both in the one step, but far smaller. Nonetheless, to looks to me modern designs do attempt to do it in one step, which is to say they attempt to recognise the object in all possible orientations simultaneously.

You can, but I don't think that's what happens when looking at stuff. When I look at the hammer, I don't rotate a hammer to see what orientation matches what I'm seeing, I see the handle and... metal bit (don't know the term, sorry), realize those look like they belong to a hammer and go "oh yeah, it's an upside down hammer".

I think that's the idea behind capsule networks. But I haven't heard anything more about them since they were announced so maybe they don't work too well.

"I can rotate an object in my mind and picture it from any angle"

Except there was a wonderful thread about Aphantasia a while ago https://news.ycombinator.com/item?id=20267445

Several HN readers chimed in to say they have this feature. It would be interesting to know if this dataset would stump them.

The model hasn't ever picked up or held a hammer. How is it ever supposed to recognize a hammer in an unfamiliar context (e.g., when given a new image that is not representative of what it was trained against?). You can't train intuition.

They show in section 4.3 that fine-tuning the last layer of ResNet-152 on half of ObjectNet (25k images) and testing on the other half increases the top-1 accuracy from 29% to 50%, while the corresponding accuracy on ImageNet is ~67%.

Nevertheless, I agree with you. Given a huge dataset with millions of of unconventional images may be enough. Who knows.

Things kind of go in cycles in machine learning (similar to other fields). There is nowadays growing dissatisfaction of having to use so much (labeled) data, and people want the models to be better "primed" to capture the variations and structures existing in the real world. Partially because labeling a lot of data is just very expensive, but partially it's also seen as inelegant and "black-boxy" or it's just not in their scientific taste.

Other people argue that learning it all from data is fine and this kind of robustness shouldn't have to be baked in to models. Rather they should/could be learned from vast amounts of unlabeled data instead (Yann LeCun seems to be in this group.), with unsupervised/self-supervised methods.

But human can take more time and get it correct. Computers cannot simply take more time and be more accurate (I guess AlphaZero can, but that is a different problem entirely).

... it's a chair. Wow that took a minute. Some of these are devilishly tricky; clearly they're designed to hit all the difficult edge cases within the domain. What a fun dataset!

1) Training with synthetic data is definitely a thing in computer vision, exactly the way you describe. You can even throw a GAN on top of the results to make the renderings look less artificial. 2) They do something like this, in section 4.3 by splitting ObjectNet in half. They fine-tune (the last layer of) ResNet-152 (I wonder what happens if you fine-tune more layers) on half and test on the other half. This pushes results up by about 15% points. There's still a gap, but it can be plausibly argued that the gap would close up if we scaled things up by one or two orders of magnitude. The question is whether there's a better way.

It's clearly delusional to think that computer vision will go from soft computing (erroneous) to hard computing in less than a decade, at the rate of current incremental improvements. We will soon hit an accuracy wall that only breakthrough research will allow to beat. The problem being: there's too much research exploring the search space in the same direction and not enough foundational research.

Is there an end here? Just google "optical illusion photos" or something similar.