Semantic search integration

[sergiimk]

According to this design document we want to implement a semantic search capability in kamu.

Given a free-form text user prompt, the search API should return N most relevant datasets.

The relevance will be based on vector distance between the prompt and the dataset metadata in the embedding space.

Metadata should initially include:

• Dataset name
• Schema column names
• Description
• Tags
• Readme and other textual attachments

Design document should include:

• Proposed model for generating embeddings (low-cost, low-footprint)
• Deployment proposal of where the embedding model will run
• Eventually consistent mechanism for synchronzing dataset metadata with the vector DB
• Semantic search API
• How kamu search command will interact with the new system
• How WebUI will interact with the new system

Scope:

• English-only embeddings model is acceptable at this stage

Based on preliminary research we would like to use Qdrant vector DB for storing the embeddings and performing search.

See also:

• https://www.reddit.com/r/rust/comments/15qsd7m/blazing_fast_opensource_semantic_searchasyoutype/