.. _ImageSimilaritySearch: Image Similarity Search ======================= Image similarity search combines vector search with graph traversal. A descriptor represents an image as a vector; a connection links that descriptor to the stored image. A query finds nearby descriptors and follows their connections to retrieve the corresponding images. This example uses five JPEGs and four-dimensional synthetic vectors. The vectors encode sample order, so their distances illustrate the query flow without measuring visual similarity. All input images are provided below. The `image similarity search application `_ covers vectors generated from image content by a model. .. _id2: .. _ImageSimilaritySearch_Ingest: Ingest Images and Descriptors into ApertureDB --------------------------------------------- A :doc:`vector index <../commands/CreateIndex>` on a descriptor class defines the vector dimensions, search engine, and distance metric. This request creates an HNSW index with four dimensions and the L2 metric. Use an unused descriptor class name for ``docs_similarity_example`` throughout the example. .. TESTED_EXAMPLE: ImageSimilaritySearch_1 .. code-block:: json [{ "CreateIndex": { "target": "descriptor", "class": "docs_similarity_example", "property": "_blob", "kind": "hnsw", "params": { "dimensions": 4, "metric": "L2" } } }] Each descriptor and its image share a ``sample_id`` property. The following query inserts sample 0 and connects the image to the descriptor through ``has_descriptor``. Supply two input blobs in command order: the vector ``[0.0, 1.0, 0.0, 0.0]`` encoded as four float32 values, then the JPEG bytes. See :ref:`blob arrays ` for the wire representation. .. TESTED_EXAMPLE: ImageSimilaritySearch_2 .. code-block:: json [{ "AddDescriptor": { "class": "docs_similarity_example", "properties": { "sample_id": 0 }, "_ref": 1 } }, { "AddImage": { "class": "Images", "format": "jpg", "properties": { "sample_id": 0, "doc_fixture": "image_similarity_example" }, "connect": { "class": "has_descriptor", "ref": 1 } } }] The remaining samples use the same insertion pattern with sample IDs 1 through 4 and the inputs below. The ``doc_fixture`` property identifies the images belonging to this example. Each JPEG is available as a download. .. list-table:: Sample images and vectors :header-rows: 1 * - sample_id - Vector - JPEG * - 0 - ``[0, 1, 0, 0]`` - :download:`input.jpg <../images/ImageSimilaritySearch/input.jpg>` * - 1 - ``[1, 1, 0, 0]`` - :download:`result_0.jpg <../images/ImageSimilaritySearch/result_0.jpg>` * - 2 - ``[2, 1, 0, 0]`` - :download:`result_1.jpg <../images/ImageSimilaritySearch/result_1.jpg>` * - 3 - ``[3, 1, 0, 0]`` - :download:`result_2.jpg <../images/ImageSimilaritySearch/result_2.jpg>` * - 4 - ``[4, 1, 0, 0]`` - :download:`result_3.jpg <../images/ImageSimilaritySearch/result_3.jpg>` .. _id3: .. _ImageSimilaritySearch_Query: Query for Similar Images ------------------------ Pass sample 0's descriptor, ``[0, 1, 0, 0]``, as the single input blob. ``k_neighbors: 5`` selects the source and all four neighbors. The descriptor ``_ref`` lets the second command retrieve their connected images. The image constraint excludes sample 0, and ``blobs: true`` returns the other JPEGs. Images are explicitly sorted by ``sample_id``; their order does not implicitly inherit descriptor distance order. .. TESTED_EXAMPLE: ImageSimilaritySearch_3 .. code-block:: json [{ "FindDescriptor": { "engine": "HNSW", "_ref": 1, "with_class": "docs_similarity_example", "k_neighbors": 5, "distances": true, "results": { "list": ["sample_id"] } } }, { "FindImage": { "is_connected_to": { "ref": 1 }, "constraints": ["$sample_id", "!=", 0], "results": { "list": ["sample_id"] }, "sort": { "key": "sample_id", "order": "ascending" }, "blobs": true } }] For these vectors, ``L2`` reports squared Euclidean distances of 0, 1, 4, 9, and 16. Each ``_blob_index`` identifies an image in the returned blob array. .. TESTED_EXAMPLE: ImageSimilaritySearch_3_response .. code-block:: json [{ "FindDescriptor": { "entities": [{ "_class": "docs_similarity_example", "_distance": 0, "sample_id": 0 }, { "_class": "docs_similarity_example", "_distance": 1, "sample_id": 1 }, { "_class": "docs_similarity_example", "_distance": 4, "sample_id": 2 }, { "_class": "docs_similarity_example", "_distance": 9, "sample_id": 3 }, { "_class": "docs_similarity_example", "_distance": 16, "sample_id": 4 }], "returned": 5, "status": 0 } }, { "FindImage": { "blobs_start": 0, "entities": [{ "_blob_index": 0, "_class": "Images", "sample_id": 1 }, { "_blob_index": 1, "_class": "Images", "sample_id": 2 }, { "_blob_index": 2, "_class": "Images", "sample_id": 3 }, { "_blob_index": 3, "_class": "Images", "sample_id": 4 }], "returned": 4, "status": 0 } }] The four returned images appear in ``sample_id`` order: .. image:: ../images/ImageSimilaritySearch/result_0.jpg :width: 48 % .. image:: ../images/ImageSimilaritySearch/result_1.jpg :width: 48 % .. image:: ../images/ImageSimilaritySearch/result_2.jpg :width: 48 % .. image:: ../images/ImageSimilaritySearch/result_3.jpg :width: 48 % With model-generated embeddings, stored and query vectors share the same model and preprocessing. The descriptor class's dimensions and distance metric must match that model. A useful relevance threshold depends on those embeddings; the synthetic distances above do not define one. The example images can be removed with a ``DeleteImage`` constraint on ``doc_fixture == "image_similarity_example"``. To clean up the descriptor class, delete descriptors with ``with_class: "docs_similarity_example"``, remove the vector index with RemoveIndex, then remove the empty class with RemoveClass. Removing an index alone preserves descriptor nodes. Deleting the images also removes their connections.