-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Remote Store] Add multipart upload integration for translog and segment files #7119
Merged
gbbafna
merged 18 commits into
opensearch-project:main
from
raghuvanshraj:multi-part-upload-core-2-pr
Jul 11, 2023
Merged
[Remote Store] Add multipart upload integration for translog and segment files #7119
gbbafna
merged 18 commits into
opensearch-project:main
from
raghuvanshraj:multi-part-upload-core-2-pr
Jul 11, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
linuxpi
reviewed
Apr 21, 2023
server/src/main/java/org/opensearch/common/blobstore/BlobContainer.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 2, 2023 09:58
716c219
to
33f3a69
Compare
Gradle Check (Jenkins) Run Completed with:
|
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/BlobContainer.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/common/blobstore/stream/write/UploadResponse.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
gbbafna
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
reviewed
May 19, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
ashking94
requested changes
May 19, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have done high level review, will review once more.
gbbafna
reviewed
May 23, 2023
server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/transfer/BlobStoreTransferService.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/opensearch/index/store/RemoteSegmentStoreDirectoryTests.java
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 24, 2023 15:52
33f3a69
to
4827a91
Compare
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 25, 2023 02:15
4827a91
to
74518bc
Compare
raghuvanshraj
changed the title
Add multipart upload integration for translog and segment files
[Remote Store] Add multipart upload integration for translog and segment files
May 25, 2023
Gradle Check (Jenkins) Run Completed with:
|
Failure from gradle CI logs:
This seems to be a flaky test as it is passing when I run it locally. I've opened a bug for the same: #7755 |
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 29, 2023 01:30
74518bc
to
55a31ac
Compare
Gradle Check (Jenkins) Run Completed with:
|
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
May 29, 2023 12:46
55a31ac
to
c48cc33
Compare
Gradle Check (Jenkins) Run Completed with:
|
gbbafna
reviewed
Jun 1, 2023
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/exception/ChecksumCombinationException.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/checked/TranslogCheckedContainer.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/translog/checked/TranslogCheckedContainer.java
Outdated
Show resolved
Hide resolved
raghuvanshraj
force-pushed
the
multi-part-upload-core-2-pr
branch
from
June 5, 2023 06:32
c48cc33
to
51ba892
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
6 tasks
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 11, 2023
…ect#7119 Signed-off-by: Ashish Singh <[email protected]>
raghuvanshraj
added
backport 2.x
Backport to 2.x branch
and removed
backport 2.x
Backport to 2.x branch
labels
Jul 11, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jul 11, 2023
…ent files (#7119) Signed-off-by: Raghuvansh Raj <[email protected]> (cherry picked from commit 5f98d67) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale
pushed a commit
that referenced
this pull request
Jul 12, 2023
--------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
6 tasks
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Ashish Singh <[email protected]>
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: Ashish Singh <[email protected]>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Ashish Singh <[email protected]>
ashking94
pushed a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: Ashish Singh <[email protected]>
ashking94
added a commit
to ashking94/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Ashish Singh <[email protected]>
vikasvb90
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]>
vikasvb90
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
raghuvanshraj
added a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]>
raghuvanshraj
pushed a commit
to raghuvanshraj/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
sachinpkale
pushed a commit
that referenced
this pull request
Jul 12, 2023
* [Remote Store] Add multipart upload integration for translog and segment files (#7119) Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: Ashish Singh <[email protected]> * Fix remote segments sync retry regression introduced in PR #7119 (#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Ashish Singh <[email protected]> * Empty-Commit Signed-off-by: Ashish Singh <[email protected]> --------- Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Raghuvansh Raj <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
dzane17
pushed a commit
to dzane17/OpenSearch
that referenced
this pull request
Jul 12, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]>
dzane17
pushed a commit
to dzane17/OpenSearch
that referenced
this pull request
Jul 12, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
buddharajusahil
pushed a commit
to buddharajusahil/OpenSearch
that referenced
this pull request
Jul 18, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: sahil buddharaju <[email protected]>
buddharajusahil
pushed a commit
to buddharajusahil/OpenSearch
that referenced
this pull request
Jul 18, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: sahil buddharaju <[email protected]>
baba-devv
pushed a commit
to baba-devv/OpenSearch
that referenced
this pull request
Jul 29, 2023
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]>
baba-devv
pushed a commit
to baba-devv/OpenSearch
that referenced
this pull request
Jul 29, 2023
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…ent files (opensearch-project#7119) Signed-off-by: Raghuvansh Raj <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…-project#7119 (opensearch-project#8632) --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Credits: @vikasvb90 and @itiyamas for the design and core implementations of the feature.
Description
Adds multipart parallel uploads in segment and translog upload path for remote store utilizing utilities and models added in #7000.
Issues Resolved
Part of #6703
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.