update files in distributed mode to use hash #761
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Eshanatnight
force-pushed
the
file-with-hash
branch
from
0ff1e51 to
bd647f9
Compare
nitisht
reviewed
Apr 16, 2024
| @@ -105,6 +105,7 @@ hashlru = { version = "0.11.0", features = ["serde"] } | |||
| path-clean = "1.0.1" | |||
| prost = "0.12.3" | |||
| prometheus-parse = "0.2.5" | |||
| sha2 = "0.10.8" | |||
There was a problem hiding this comment.
why not use ulid that we use for deployment ids?
There was a problem hiding this comment.
I am hashing the timestamp, so I needed sha2
Eshanatnight
force-pushed
the
file-with-hash
branch
from
bd647f9 to
50caaf9
Compare
|
We should add a check in querier for hash collision. I mean querier should
stop if it detects a collision- because this will lead to data loss
otherwise
…On Tue, 16 Apr 2024 at 5:34 PM, Eshan ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In server/Cargo.toml
<#761 (comment)>
:
> @@ -105,6 +105,7 @@ hashlru = { version = "0.11.0", features = ["serde"] }
path-clean = "1.0.1"
prost = "0.12.3"
prometheus-parse = "0.2.5"
+sha2 = "0.10.8"
I am hashing the timestamp, so I needed sha2
—
Reply to this email directly, view it on GitHub
<#761 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHK2KZEZS5RCXALBTERO6TY5UHTLAVCNFSM6AAAAABGJATVWSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDAMBTGQZTINJSGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Eshanatnight
force-pushed
the
file-with-hash
branch
2 times, most recently
from
4bcbc13 to
7ce300f
Compare
Eshanatnight
force-pushed
the
file-with-hash
branch
from
7ce300f to
866308a
Compare
nikhilsinhaparseable
added a commit
to nikhilsinhaparseable/parseable
that referenced
this pull request
Apr 20, 2024
This PR ensures all metadata and data files (json and parquet) use a simple sha256 based hash name mechanism. Each ingestor allocates itself a unique hash which is used in all file names relevant to that ingestor. This hash is persisted in metadata file content also and is supposed to be the same for the lifecycle of the ingestor. --------- Co-authored-by: Nikhil Sinha <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Needs to be rebased after #760 merge