PGVector integration #222
PGVector integration #222
Conversation
Co-authored by: Doug Ortiz <[email protected]> Sarah Conway <[email protected]>
| 'INSERT INTO embeddings (id, embedding) VALUES (%s, %s::vector) ON CONFLICT (id) DO UPDATE SET embedding = EXCLUDED.embedding', | ||
| (id, embedding.tolist()) | ||
| ) | ||
| self._conn.commit() |
There was a problem hiding this comment.
I am not sure if this is the right place but somewhere you need to create an HNSW index. I think the default values for index density are fine. You should probably ensure that working memory is temporarily set larger than the default. See here
https://github.com/thesteve0/arvix-query/blob/main/sql_commands.sql#L23
Cosine is also a fine distance metric to have as the default.
There was a problem hiding this comment.
Looks like the elasticsearch has an explicit method for index creation
|
|
||
| _SUPPORTED_METRICS = ("cosine", "dotproduct", "euclidean") | ||
|
|
||
| class PgVectorSimilarityConfig(SimilarityConfig): |
There was a problem hiding this comment.
If you look at the elasticsearch integration they allow for defining the distance metric. We should do the same:
We will also need the SSL certs and other connection types that people use with PostgreSQL
There was a problem hiding this comment.
Still need an implementation for
get_embeddings -
remove_from_index -
Hi!
These files still need to be polished up and the documentation + similarity test added. I'm opening this draft PR so existing work can be reviewed if desired and so the current progress is known.
Additional changes will be pushed up to this PR ASAP.
The existing code will have a pre-commit hook performed on it before this PR is marked as ready to merge.