Fetch schema and stream from obj store #745
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tern in get_objects method
fetch stream info from store if stream info is not present in memory. error if stream info does not exist in s3 and memory
nikhilsinhaparseable
approved these changes
Apr 8, 2024
nikhilsinhaparseable
pushed a commit
to nikhilsinhaparseable/parseable
that referenced
this pull request
Apr 20, 2024
* Refactor object storage to use filter_func instead of starts_with_pattern in get_objects method * Refactor fetch_schema method to use object storage instead of HTTP requests * Refactor metadata.rs and storage.rs * refactor ingest logic * fetch stream info from store if stream info is not present in memory. error if stream info does not exist in S3 and memory
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Schema now is fetched from the Object Store
And Stream Info for ingesters is no longer synced/updated at creation time, rather at ingest time, if stream info does not exists
Ingester Updates the stream info from store
This PR has: