Elasticsearch Based Image Search Using RGB Signatures (opens in new tab)

Recent research in deep manifold traversal may interest you [1]. This method uses deep neural networks to approximate the manifold of natural images, learns transformations that traverse the manifold from one image class to another (i.e. from images of young people to images of old people), and then allows you to transform any source image into a new, automatically-generated image by mapping it to the manifold and performing the transformation. So, for example, the paper's authors learn the transformation from images of young people to old people, and then are able to generate realistic-looking images of the celebrities with older facial features (as well as darken hair colors, change skin tones, and even colorize black and white images). This is somewhat analogous to the way that word2vec vectors allow you to do things like queen = king + (man - woman), with an image's projection onto the approximated manifold being analogous to a word's word2vec vector.

[1] http://arxiv.org/pdf/1511.06421v3.pdf

I think in addition to histograms and features like SIFT, SURF, and others like DAISY[0] that would permit searching with images as a query it would be beneficial just to use a neural network to text index using the classes at very top of a neural network, although you are right some features in layers just below that could be used too as I understand it.

You could then use classical text indexing on the text, perhaps with a topic model like LDA. Then an image with a plane in it will be indexed by "plane" via the output of the neural network but would also come first, or in the top results, when using "flight" as the query via a topic model.

Ditto for word2vec or para2vec over those words, the benefit being you can bring the knowledge of relations contained in the textual training data, Wikipedia or something else, to bear on the problem. I.e. a golf club and a baseball glove might not be correlated in the neural network that annotated the images but might be correlated in the text based knowledge model trained on Wikipedia and so a query of "sport" might bring both images up.

The big players like Google already have caption generation that's capturing relationships between objects.[1]

[0]http://scikit-image.org/docs/dev/auto_examples/features_dete... [1]https://arxiv.org/pdf/1411.4555.pdf

I'm also pretty ignorant here, but if I understand word vectors correctly they are a trained model that results in the ability to predict the next word in a sequence. I can imagine extending that idea to images, i.e. in such a way that the color of a target pixel can predict the colors of surrounding pixels. I wonder if tools like Photoshop and Gimp don't already have similar algorithms for some advanced effects. In any case, it seems several layers of abstraction down the stack from the problem of attaching meaning to images. Isn't that more what the BOW approach tries to do?

I think depending on the need Neural Network feature detection/description might be an overkill.

What is great about stuff like SIFT and more modern ORB, BRISK and AKAZE is that they are fast and given appropriate implementation they would work just as well as Neural Network would. I haven't researched NN computer vision whole lot, but it seems like it might be slower in feature detection/description compared to traditional approach. If that's the case then for live/near-live video processing NN won't work that well.

Images don't necessarily map directly onto a word2vec-like solution but the closest thing is pre-trained deep networks, for example: http://caffe.berkeleyvision.org/model_zoo.html

You're right that bag of words with SIFT is not state of the art, with deep learning dominating computer vision approaches these days.

There're ton more local features proposed since SIFT but SIFT, arguably, still among the best. Deep feature, on the other hand, is more like global feature. May be a combination of these two could give the best?