Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Add multipart upload integration for translog and segment files #7119

Merged

Conversation

raghuvanshraj
Copy link
Contributor

@raghuvanshraj raghuvanshraj commented Apr 12, 2023

Credits: @vikasvb90 and @itiyamas for the design and core implementations of the feature.

Description

Adds multipart parallel uploads in segment and translog upload path for remote store utilizing utilities and models added in #7000.

Issues Resolved

Part of #6703

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from 716c219 to 33f3a69 Compare May 2, 2023 09:58
@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

Copy link
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have done high level review, will review once more.

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from 33f3a69 to 4827a91 Compare May 24, 2023 15:52
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from 4827a91 to 74518bc Compare May 25, 2023 02:15
@raghuvanshraj raghuvanshraj changed the title Add multipart upload integration for translog and segment files [Remote Store] Add multipart upload integration for translog and segment files May 25, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj
Copy link
Contributor Author

Failure from gradle CI logs:

* What went wrong:
Execution failed for task ':qa:mixed-cluster:v2.8.0#mixedClusterTest'.
> There were failing tests. See the report at: file:///var/jenkins/workspace/gradle-check/search/qa/mixed-cluster/build/reports/tests/v2.8.0%23mixedClusterTest/index.html

This seems to be a flaky test as it is passing when I run it locally. I've opened a bug for the same: #7755

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from 74518bc to 55a31ac Compare May 29, 2023 01:30
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: 55a31ac
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from 55a31ac to c48cc33 Compare May 29, 2023 12:46
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: c48cc33
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-core-2-pr branch from c48cc33 to 51ba892 Compare June 5, 2023 06:32
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 11, 2023
@raghuvanshraj raghuvanshraj added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jul 11, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 11, 2023
…ent files (#7119)

Signed-off-by: Raghuvansh Raj <[email protected]>
(cherry picked from commit 5f98d67)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Jul 12, 2023
---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
vikasvb90 pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
vikasvb90 pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
raghuvanshraj added a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
raghuvanshraj pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
sachinpkale pushed a commit that referenced this pull request Jul 12, 2023
* [Remote Store] Add multipart upload integration for translog and segment files (#7119)

Signed-off-by: Raghuvansh Raj <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>

* Fix remote segments sync retry regression introduced in PR #7119 (#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>

* Empty-Commit

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Raghuvansh Raj <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Raghuvansh Raj <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023
dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023
buddharajusahil pushed a commit to buddharajusahil/OpenSearch that referenced this pull request Jul 18, 2023
buddharajusahil pushed a commit to buddharajusahil/OpenSearch that referenced this pull request Jul 18, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: sahil buddharaju <[email protected]>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants