Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read parquet metadata with opendal range reader #3191

Closed
evenyag opened this issue Jan 18, 2024 · 0 comments · Fixed by #3199
Closed

Read parquet metadata with opendal range reader #3191

evenyag opened this issue Jan 18, 2024 · 0 comments · Fixed by #3199
Labels
A-storage Involves code in storage engines C-enhancement Category Enhancements

Comments

@evenyag
Copy link
Contributor

evenyag commented Jan 18, 2024

Now we use AsyncFileReader::get_metadata() to read the metadata, which requires seek() and downloading the whole file. We'd better implement it using opendal's range reader instead.

We could also store some information in the FileMeta (e.g. file size, footer len, or position of the parquet file meta) to eliminate seek().

@evenyag evenyag added this to mito2 Jan 11, 2024
@evenyag evenyag converted this from a draft issue Jan 18, 2024
@evenyag evenyag added C-enhancement Category Enhancements A-storage Involves code in storage engines labels Jan 18, 2024
@evenyag evenyag moved this from Todo to In Progress in mito2 Jan 18, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in mito2 Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Involves code in storage engines C-enhancement Category Enhancements
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants