feat: Adding vector search for sqlite (#4176)

* feat: Adding vector search for sqlite

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding the sqlite_vss dependency

Signed-off-by: Francisco Javier Arceo <[email protected]>

* linter

Signed-off-by: Francisco Javier Arceo <[email protected]>

* latest progress

Signed-off-by: Francisco Javier Arceo <[email protected]>

* uploading latest progress

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated function

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding configuration

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding current progress

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updating requirements files

Signed-off-by: Francisco Javier Arceo <[email protected]>

* moving to sqlite-vec

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updating sqlite.py

Signed-off-by: Francisco Javier Arceo <[email protected]>

* checking in progress

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated test type

Signed-off-by: Francisco Javier Arceo <[email protected]>

* got the initialization working, nice

Signed-off-by: Francisco Javier Arceo <[email protected]>

* checking in progress from last night

Signed-off-by: Francisco Javier Arceo <[email protected]>

* removing unnecessary stuff

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fixing merge conflicts

Signed-off-by: Francisco Javier Arceo <[email protected]>

* removing files changed accidentally]

Signed-off-by: Francisco Javier Arceo <[email protected]>

* uploading current progress...things run but need to update the virtual table insertion

Signed-off-by: Francisco Javier Arceo <[email protected]>

* linted

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding working notes

Signed-off-by: Francisco Javier Arceo <[email protected]>

* found a bug, original feature_store.py was only grabbing first feature view, adjusted

Signed-off-by: Francisco Javier Arceo <[email protected]>

* cant use a string have to verify it is a proper FeatureView object

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated got it working, need to fix some other stuff still

Signed-off-by: Francisco Javier Arceo <[email protected]>

* working

Signed-off-by: Francisco Javier Arceo <[email protected]>

* linter

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fixing some type issues

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fixed typing and lint issues

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated dependencies

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fix for pixi and updating requirements

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fixed type

Signed-off-by: Francisco Javier Arceo <[email protected]>

* linter

Signed-off-by: Francisco Javier Arceo <[email protected]>

* testing sqlite_vec import

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding minimal example test

Signed-off-by: Francisco Javier Arceo <[email protected]>

* lint

Signed-off-by: Francisco Javier Arceo <[email protected]>

* testing raw sqlite

Signed-off-by: Francisco Javier Arceo <[email protected]>

* Printing package version

* printing version

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated requirements

* rebuilding requirments

Signed-off-by: Francisco Javier Arceo <[email protected]>

* only going to run this on 3.10 for now

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated docs for sqlite caveats

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding reason

Signed-off-by: Francisco Javier Arceo <[email protected]>

* skipping

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated tests

Signed-off-by: Francisco Javier Arceo <[email protected]>

* removing print

Signed-off-by: Francisco Javier Arceo <[email protected]>

* added method call

Signed-off-by: Francisco Javier Arceo <[email protected]>

* added prubt

Signed-off-by: Francisco Javier Arceo <[email protected]>

* added print

Signed-off-by: Francisco Javier Arceo <[email protected]>

* removing print

Signed-off-by: Francisco Javier Arceo <[email protected]>

* adding check in sqlite

Signed-off-by: Francisco Javier Arceo <[email protected]>

* missed an =

Signed-off-by: Francisco Javier Arceo <[email protected]>

* still running on 3.11

Signed-off-by: Francisco Javier Arceo <[email protected]>

* typo

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fix

Signed-off-by: Francisco Javier Arceo <[email protected]>

* fix

Signed-off-by: Francisco Javier Arceo <[email protected]>

* updated setup and docs

Signed-off-by: Francisco Javier Arceo <[email protected]>

* renamed things

Signed-off-by: Francisco Javier Arceo <[email protected]>

---------

Signed-off-by: Francisco Javier Arceo <[email protected]>

docs/reference/alpha-vector-database.md

@@ -13,7 +13,9 @@ Below are supported vector databases and implemented features:
 | Elasticsearch   | [x]       | [x]      |
 | Milvus          | [ ]       | [ ]      |
 | Faiss           | [ ]       | [ ]      |
+| SQLite          | [x]       | [ ]      |
 
+Note: SQLite is in limited access and only working on Python 3.10. It will be updated as [sqlite_vec](https://github.com/asg017/sqlite-vec/) progresses.   
 
 ## Example
 
@@ -108,4 +110,20 @@ def print_online_features(features):
         print(key, " : ", value)
 
 print_online_features(features)
+```
+
+### Configuration
+We offer two Online Store options for Vector Databases. PGVector and SQLite.
+
+#### Installation with SQLite
+If you are using `pyenv` to manage your Python versions, you can install the SQLite extension with the following command:
+```bash
+PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" \
+    LDFLAGS="-L/opt/homebrew/opt/sqlite/lib" \
+    CPPFLAGS="-I/opt/homebrew/opt/sqlite/include" \
+    pyenv install 3.10.14
+```
+And you can the Feast install package via:
+```bash
+pip install feast[sqlite_vec]
 ```