Zero-downtime reindexing

[kbond]

My app has a simple search system that I fully re-index in a cron job. This project looks very interesting! Currently, I'm using elasticsearch but looking to switch to algolia.

To avoid downtime during the re-index, I have two indexes that I swap between with an alias.

Algolia has a similar feature: https://www.algolia.com/doc/api-reference/api-methods/replace-all-objects/

Curious if there is interest in this feature in SEAL. Maybe a ReindexableSchemaManagerInterface?

[alexander-schranz]

Hello @kbond,

That is sure an interesting idea and aliases are something I already had on my list because I also think it make reindexing easier. The algolias replaceAll looks a little bit strange for me atleast how I understand that it works internally the same I'm curious that there is no batch processing here and all objects need to be send via a single request which can be a lot of documents and a huge request which definitely ends in some cases in memory limit error in big amount of documents. Another thing which is not listed in the docs of replaceAll if it works with replicas as they are require for sorting, the moveIndex which is mention.

I need to let this go to my head if you have more information how this is supported in Meilisearch, Redisearch, Solr or Typesense, let me know.

Maybe a ReindexableSchemaManagerInterface?

Curious why you see this as part of SchemaManager and not part of the Indexer?

---

PS: Currently a migration with Zero-downtime can also be achieved manually via a read-write adapter configuration currently some more configuration and manual switching required but still possible:

Step 1: Configure different read write indexes

schranz_search:
    index_name_prefix: '%env(TEST_INDEX_PREFIX)%'
    schemas:
        default:
            dir: '%kernel.project_dir%/config/schemas'
            engine: default
    engines:
        read:
            adapter: '%env(SEAL_READ_DSN)%'
        write:
            adapter: '%env(SEAL_WRITE_DSN)%'
        # multi: # would be possible to use as write adapter if still want to write into both indexes
        #   adapter: 'multi://read?adapters[]=write'
        default:
            adapter: 'read-write://read?write=write'

Step 2: Configure different read write ENV

SEAL_READ_DSN=elasticsearch://127.0.0.1:9200
SEAL_WRITE_DSN=algolia://ALGOLIA_APPLICATION_ID:ALGOLIA_PRIVATE_KEY

Step 3: Reindex

Then reindex bin/console schranz:search:reindex --engine default

Step 4: Use same DSN to read and write form same source

SEAL_READ_DSN=algolia://ALGOLIA_APPLICATION_ID:ALGOLIA_PRIVATE_KEY
SEAL_WRITE_DSN=algolia://ALGOLIA_APPLICATION_ID:ALGOLIA_PRIVATE_KEY

It's not yet documented, as I'm not yet fan of the configuration, but this way a zero downtime migration from elasticsearch to algolia is currently possible from one adapter to another. What I sure want to support is a index_name_postfix on adapter DSN level so same adapter could be reindexed.

Still doesn't solve your problem if you want todo a complete reindex as a cronjob, just wanted to mention it here that kind of zero migration from one to another DSN is supported over a manually mechanism.

But this manually mechanism is only require if the Schema was changed or moving to another adapter, if really just a reindex is required the following command does reindex your documents without dropping the existing index:

bin/console schranz:search:reindex --engine default --index blog

So that also could at current state also be used as a cronjob as it does not drop the index.

---

I need a little bit experiment around with the aliases and replaceAll and check the other engines to get then back what the best way for reindex interface here. If you have any more hints and experience here how you currenlt ydid implement it let me know.

Its maybe also related to bulk inserts which should make reindex in some engines a lot faster: #24.

[kbond]

Thanks for the detailed reply! I'm excited to dig into this library!

The algolias replaceAll looks a little bit strange for me atleast how I understand that it works internally the same I'm curious that there is no batch processing here and all objects need to be send via a single request which can be a lot of documents and a huge request which definitely ends in some cases in memory limit error in big amount of documents.

Fair point. I believe the manual way would be to create a new (temporary) index (ie blog__temp), configure/populate it like normal, then call moveIndex().

I need to let this go to my head if you have more information how this is supported in Meilisearch, Redisearch, Solr or Typesense, let me know.

Unfortunately, I have no experience with these others.

Curious why you see this as part of SchemaManager and not part of the Indexer?

I just quickly looked though the code and saw SchemaManagerInterface included the create/dropIndex methods that the re-index command's use.

just wanted to mention it here that kind of zero migration from one to another DSN is supported over a manually mechanism.

Interesting! When I make the switch from elasticsearch to algolia I'm not concerned about downtime. More just the cronjob as discussed.

[alexander-schranz]

I could currently found this for the other engines:

• Redisearch: FT.ALIASADD: https://redis.io/commands/ft.aliasadd/
• Apache Solr: Aliases: https://solr.apache.org/guide/8_1/aliases.html
• Typesense: Aliases: https://typesense.org/docs/0.24.1/api/collection-alias.html#create-or-update-an-alias
• Opensearch: Aliases: https://opensearch.org/docs/latest/im-plugin/index-alias/
• Elasticsearch: Aliases: https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html
• Meilisearch: SwapIndexes: https://www.meilisearch.com/docs/reference/api/indexes#swap-indexes
• Algolia: replaceAllObjects: https://www.algolia.com/doc/api-reference/api-methods/replace-all-objects/

So in most cases it is working via an Alias the SwapIndexes of Meilisearch seems to be little bit kind of alias but also not. Think after swap the old index can then be removed. Algolia I will experiment a bit what the save: true flags means and if maybe over that we can suceed some batch-processing of the documents.

I just quickly looked though the code and saw SchemaManagerInterface included the create/dropIndex methods that the re-index command's use.

Yeah maybe that make sense 🤔 first thought it as part of the Indexer but maybe as aliases, swapIndexes, ... would be more part of what SchemaManager is responsible for and a Interface on the SchemaManager make sense, think like the idea to have it there.

A question which pops in is I'm thinking in which case the reindexing should be do the reindex on the current live index and in which cases we should do it over alias mechanism. Maybe connected to the dropIndex flag here: https://github.com/schranz-search/schranz-search/blob/4217a41ddaebe852f7ecfc0ef3f5232183a96917/packages/seal/src/Engine.php#L127

$reindexProvidersPerIndex = [];

// ...

if ($dropIndex && $this->schemaManager instanceof ReindexableSchemaManagerInterface) {
    foreach ($reindexProvidersPerIndex as $index => $reindexProviders) {
        $this->schemaManager->reindex($this->schema->indexes[$index], $reindexProviders, ?callable $progressCallback = null);
    }
    
    return;
}

// else fallback to the current mechanism

What do you think about this?

EDIT: Not sure if I want to give reindexProviders in the SchemaManager or instead give just a \Generator of documents in it. Think that maybe better from interface point of view 🤔.

I'm not concerned about downtime. More just the cronjob as discussed.

I'm curious about the usecase that a normal reindex update documents from source is not working in your case? As that would also be zero downtime when you just update the documents in the existing index? Does in your case the Index get out of Sync that documents are not removed when related entity is removed and so you need build the index always from new that this old documents are not longer part of the index?

[kbond]

Does in your case the Index get out of Sync that documents are not removed when related entity is removed and so you need build the index always from new that this old documents are not longer part of the index?

Yes, exactly, this is just a process I've used for a long time. Admittedly, it perhaps needs to be re-thought.

What do you think about this?

I have converted over to algolia (using their sdk directly) and these are the commands I've created:

1. search:index:delete <name>: Delete's the <name> index if it exists
2. search:index:create <name>: Create the <name> index. If it exists, fail
   • --populate option to add data from providers after creation
3. search:index:repopulate <name>: create temp index, populate from providers, move to live
   • I opted not to use replaceAllObjects because if the issue you mentioned above. Upon later review, it looks like it can be used with a generator and breaks up the requests

Just wanted to note some algolia-specific features I created:

1. search:index:pull-settings <name>: saves settings, synonyms, and rules to local json files (in config/algolia/* that you commit to your repo)
2. search:index:push-settings <name>: push settings, synonyms, and rules saved locally to algolia
   • this is automatically done during search:index:create and search:index:repopulate

What's nice about this is you can create all your settings/synonyms/rules in Algolia's UI (maybe in a development index), then save them to your project. Part of your deployment process could be to push the settings back up. In my case, my nightly repopulate job ensures the latest settings are pushed.

I'm not sure exactly how this feature could/should be tied into this package...

For searching my algolia index, I've heavily customized the query options:

$results = $index->search($query, [
    'X-Forwarded-For' => $request->getClientIp(), // "unique" search tracking (for non-logged in users)
    'userToken' => $user->id(), // "unique" search tracking for logged in users
    'facetFilters' => [
        ['locale:'.$_locale], // this is already handled by this package
    ],
    'clickAnalytics' => true, // required for algolia events
    'attributesToHighlight' => [], // remove extra stuff from response I don't need
    'attributesToRetrieve' => [ // restrict result data to just the fields I need
        'url',
        'status',
        'image',
        'title',
        'subtitle1',
        'subtitle2',
    ],
]);

Again, this is just demonstrating my current use-case/need.