Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-29003 Proper bulk load tracking #6506

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

HBASE-29003 Proper bulk load tracking #6506

wants to merge 1 commit into from
+226 −123

Conversation

DieterDP-ng
Copy link
Contributor

Note that this build on top of my older PR: #6134

@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@DieterDP-ng
HBASE-29003 Proper bulk load tracking
44307bf 
The HBase backup mechanism keeps track of which HFiles
were bulk loaded, so they can be included in incremental
backups.

Before this ticket, these bulk load records were only
deleted when an incremental backup is created. This
commit adds 2 more locations:

1) after a full backup. Since a full backup already
captures all data, this meant that unnecessary HFiles
were included in the next incremental backup.

2) after a table delete/truncate. Previously, if an
HFile was loaded before a table was cleared, the next
incremental backup would effectively still include the
HFile. This lead to incorrect data being restored.

This commit also completely refactors & simplifies the
test for this functionality.
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for branch
+1 💚 mvninstall 4m 27s master passed
+1 💚 compile 9m 51s master passed
-0 ⚠️ checkstyle 1m 25s /buildtool-branch-checkstyle-root.txt The patch fails to run checkstyle in root
+1 💚 spotbugs 12m 42s master passed
+0 🆗 refguide 2m 43s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 spotless 0m 53s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for patch
+1 💚 mvninstall 3m 56s the patch passed
+1 💚 compile 9m 21s the patch passed
+1 💚 javac 9m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 10s /buildtool-patch-checkstyle-root.txt The patch fails to run checkstyle in root
+1 💚 spotbugs 9m 49s the patch passed
+0 🆗 refguide 2m 3s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 hadoopcheck 11m 36s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 27s The patch does not generate ASF License warnings.
80m 1s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6506
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless refguide
uname Linux 281a14a6808b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 44307bf
Default Java Eclipse Adoptium-17.0.11+9
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-6506/3/yetus-general-check/output/branch-site/book.html
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-6506/3/yetus-general-check/output/patch-site/book.html
Max. process+thread count 188 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/3/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 29s Docker mode activated.
-0 ⚠️ yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for branch
+1 💚 mvninstall 3m 16s master passed
+1 💚 compile 2m 16s master passed
+1 💚 javadoc 2m 49s master passed
+1 💚 shadedjars 5m 53s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for patch
+1 💚 mvninstall 3m 8s the patch passed
+1 💚 compile 2m 14s the patch passed
+1 💚 javac 2m 14s the patch passed
+1 💚 javadoc 2m 38s the patch passed
+1 💚 shadedjars 5m 55s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌ unit 258m 25s /patch-unit-root.txt root in the patch failed.
293m 14s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6506
Optional Tests javac javadoc unit compile shadedjars
uname Linux 43ef4f83d281 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 44307bf
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/3/testReport/
Max. process+thread count 7620 (vs. ulimit of 30000)
modules C: hbase-server hbase-backup . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6506/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DieterDP-ng @Apache-HBase