[Feature] Zero-downtime full re-index of Elasticsearch

[Nek-]

A common need while working with ES is to rebuild the index from scratch.

Let's take an example to highlight an issue. I have an index of Products. My products database moves a lot, for stock updates or creation of new products/disabling some others: it changes, a lot. I have 1 000 000 products in my database.
I want to rebuild my products index: this process will take a while, and in the meantime, I need to be consistent with updates on the new and the old index.

To stay consistent, it would be convenient if the IndexationRequest could update many indexes simultaneously, with an extra argument it would be possible. Something like this:

$request = new IndexationRequest($document, ['indexes' => [
    $liveIndex->getName(),
    $newIndex->getName()
]);

[damienalexandre]

Hello 👋

As much as I know, there are no bulletproof method at the moment to rebuild an entire index while keeping updates running in the background.

I have two great source of knowledge when it come to tacking reindexing without downtime:

• https://www.codecentric.de/wissens-hub/blog/elasticsearch-zero-downtime-reindexing-problems-solutions 😍
• https://betterprogramming.pub/how-i-reindex-elasticsearch-without-downtime-6e8a6a512070

As you can see, there are pros and cons to each method, and every application is different.

What I did one time was to tweak the Indexer to ALWAYS push to a Schrödinger alias next_index and ignore the 404, that way all my updates where always sent to the new index I was building. But that was working only because I didn't to DELETE, and didn't care about performances.

I would love for Elastically to provide an actual implementation of a real zero downtime reindex!

In the meantime, about your suggestion - how does your application knows about the $newIndex? For example if I update a Product from the Admin? 🤔 The new index would have to be stored somewhere?

One alternative proposal could be to tell the Indexer "there is a new Index in progress", and tweak

elastically/src/Indexer.php

Line 82 in c82e787

public function flush(): ?Bulk\ResponseSet

to add the new Index in all the Bulk operations 😋