Svelte CLIP Image Search

A SvelteKit UI for a CLIP-based image search backend.

My first attempt at a SvelteKit based UI.

This project uses the same FastAPI backend for CLIP-based image search as my earlier Vue-based UI found here.

Demo

A demo of this UI, using Wikimedia Commons images, can be seen here.

Try searching for somethign like skiing -summer +winter to see sports that are kinda-like skiing, but happen in summer instead of winter.

Features:

• Incremental loading of thumbnails, with support for throttled image sources like Wikipedia (that only lets a single client load a few thumbnails a second).

• Flicker-free previews based on Svelte deferred transitions to not render partially-downloaded images.

• Fast previews based by initially showing a zoomed thumbnail, then fading into the full res image when its available.

• A search syntax that lets you do simple math on CLIP embeddings with prefixes like "-" to subtract CLIP vectors and "+" to add them.

  • zebra -stripes +spots - Animals that look kinda like zebras but with spots instead of stripes.
  • zebra -mammal +fish - Animals that look like zebras but fish instead of mammals.
  • zebra -animal +car - Objects colored like zebras but more cars than animals.
  • zebra -"black and white" - Baby zebras (brown & white) and a Greater Kudu (a brown & white striped 4-legged animal). Of course you could also find the same baby zebra searching for zebra -big +small or even more simply, just baby zebra.
  • furry black and white striped animal - zebras, lemurs, and other furry black and white animals.
  • striped horse-like animal - more zebras (and horses with stripes)
  • zebra habitat -zebra - places that look like somewhere a zebra might live

• Search for similar images

  • It can also do a search based on the difference between the CLIP embeddings of two images directly. For example, CLIP considers this image of a spider on a purple flower minus this image of the same kind of spider on a white flower to be this set of pictures which is mostly purple flowers without the spider.

Thanks

Special Thanks to:

• Github user @yurijmikhalevich for his rclip CLI tools that inspired this project and are now used to manage all my home photos.
• Github user @berkinakkaya for his svelte-image-gallery project that made me interested in porting the front-end to Svelte, (and whose code I used, though I need to double-check with him about licensing.... doing that now)

TODO

• push some of the heavy lifting to the browser; especially to process images from webcams; possibly using this: https://github.com/josephrocca/openai-clip-js