Indexing iCloud Photos with AI Using LLaVA and Pgvector (opens in new tab)

I would love to see some benchmark on that

vit h and g are fine I wouldn't use b anymore.

I learned it the hard way while trying to index few TBs of photos. I couldn't have it finished, always got stuck after 15 ish hours.

> has been using CLIP models for smart search for a while and anecdotally seems to work really well [..] results are of similar quality to the giant SaaS providers

I'm not super familiar with how the results for the "giant SaaS providers" are, but the demo instance of Immich doesn't seem to do it very well.

Example query for "airplane": https://demo.immich.app/search?q=airplane&clip=true

Even the fourth result seems to rank higher than photos of actual airplanes, and most of the results aren't actually airplanes at all.

Again, not sure how that compares with other providers, but on Google Photos (as one example I am familiar with), searching for "airplane" shows me photos taken of airplanes, or photos taken from the inside of an airplane. Even lego airplanes seems to show up correctly, and none of the photos are incorrectly shown as far as I can tell.

We use CLIP internally (large US tech company) and it works very well at a large scale

Both Google and Apple have implemented models that aim to take the "best" picture from a video sequence, like a live photo.

Have a look at MUSIQ and NIMA.

https://github.com/google-research/google-research/tree/mast...

https://blog.research.google/2022/10/musiq-assessing-image-a...

So I think some sort of hybrid between object recognition (like being discussed here as part of the workflow) and standard image processing stuff could be helpful there. E.g. it's not absolute sharpness that you're looking for it's the subject being sharp (and possibly sharper than in other photos from the same time period of the same subject).

Try LAION Aesthetics: https://laion.ai/blog/laion-aesthetics/

Going through a LLM may improve the performance. From my experience working with Stable Diffusion 1.*, clip is not very intelligent and a 7B quantised LLM could help a lot.

I haven't used it for search, but I believe Insightface's embeddings can be used for this purpose. https://insightface.ai/

One long winded way could be using the Lightroom for that. It finds and groups faces. Also, maybe it can save that info into fmthe file itself (with xmp)

that is exactly what I wanted to do for my Apple Photos lib but have not yet got the time to spend on it. Apple Photos search is just bad, very bad.

Object detection models have human-comprehensible outputs. You can feed in a picture and it'll tell you that there's a child and a cat, and it'll draw bounding boxes around them. You can pass that info into an LLM if you want.

The downside to that approach is the LLM can't tell whether the cat is standing in front of the child, or sitting on the child, or the child is holding the cat; the input just tells it there's a child, and a cat, and their bounding boxes overlap.

In contrast, LLaVA feeds feeds the image into a visual encoder called 'CLIP' which doesn't output anything human-comprehensible - it just gives out a bunch of numbers which have something to do with the contents of the image. But the numbers can be fed into the LLM along with text - and they can train the image encoder and the LLM together.

If the training works right, and they have enough training data for the model to figure out the difference between a cat sitting on a lap and one being held, they end up with a model that can figure out that the child is holding the cat.