Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement to ERA5 Data Retrieval and Download Process #397

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

yndevops2
Copy link

@yndevops2 yndevops2 commented Oct 28, 2024

This update introduces an optimized approach for data retrieval and caching for ERA5 data from the Climate Data Store (CDS). Key changes include:

  1. Caching Mechanism: Added a caching mechanism to prevent repeated downloads for identical data requests. The cache files are named based on a unique hash of the request parameters, making subsequent retrievals faster by using pre-downloaded data.

  2. Custom Download Function: Integrated a custom download function with a progress bar to enhance user experience. The function uses chunked downloading with error handling and retry mechanisms for a robust download process.

  3. Progress Bar: A dynamic progress bar displays the download status of multiple files, with completed files removed from the display to improve readability.

These improvements aim to make data retrieval more efficient and user-friendly.

Closes # (if applicable).

Changes proposed in this Pull Request

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@yndevops2 yndevops2 changed the title Update era5.py Enhancement to ERA5 Data Retrieval and Download Process Nov 3, 2024
@fneum fneum requested a review from lkstrp November 4, 2024 11:03
Copy link
Member

@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yndevops2 for the contribution!
In general you can contact us via GitHub and PRs, you don't need to send any emails.
Still, I haven't heard back from you.

A caching feature like this adds a lot of overhead and I do not know if it is needed. Also, the CDS API already provides caching and there is no real use case for re-downloading data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants