Also, while it is laudable that they attempt to learn slow higher-level features, the result of contrastive loss functions is still very much detail-focussed, it just is so in a translationally invariant way.
A common problem for image classification is that the AI will learn to recognize high-level fur patterns, as opposed to learning the shape of the animal. Using contrastive loss terms like in their example will drive the network towards having the same features vector for adjacent pixels, meaning that the fur pattern detector needs to become translation-invariant. But the contrastive loss term will NOT prevent the network from recognizing the fur, rather than the shape, as is claimed in this article.
I did not get the second part unfortunately, could you elaborate more and clarify if you are talking about a specific paper?
I would instead suggest to train a categorization classifier unsupervised, too, for example using mutual information loss with the correct number of categories, as suggested in [2]. Afterwards, one can then deduct the mapping between the categories learnt unsupervised and the groundtruth categories to allow evaluation. That way, good results clearly prove a good unsupervised training method.
The problem that I mean in the second part was that most networks trained for object recognition work on low-level features such as colors and textures, as shown in [3]. The turtle clearly has a turtle shape and arrangement and looks overwhelmingly like a turtle to humans. But its high-frequency surface details are those that the neural network associates with a rifle, which is why those networks are fooled even on photos from varying perspectives.
Training a network with a loss to ensure that the local area of an image produces features that are highly correlated to the global features of the same image does not avoid this problem, because the high-frequency patters that the AI erroneously uses for detection are present both in the local as well as in the global scale. Sadly, I don't have any idea on how to improve that either.
[1] What's Hidden in a Randomly Weighted Neural Network? https://arxiv.org/abs/1911.13299
[2] Invariant Information Clustering for Unsupervised Image Classification and Segmentation https://arxiv.org/abs/1807.06653
[3] Synthesizing Robust Adversarial Examples https://arxiv.org/abs/1707.07397
I've discussed these ideas here, for those that are interested in learning more: https://www.fast.ai/2020/01/13/self_supervised/
I recently created a little dataset that is specifically designed to allow for testing out self-supervised techniques, called Image网 ("Imagewang"). I'd love to see some folks try it out, and submit strong baselines to the leaderboard: https://github.com/fastai/imagenette#image%E7%BD%91
I've a good amount of experience playing with autoencoders but this is the first I've heard of contrastive learning.
Agree that using them in conjunction would make sense, since generative methods could capture some features better and vice versa.
