Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set encoding/compression parameters upon DataArray creating #397

Open
bdestombe opened this issue Dec 24, 2024 · 0 comments
Open

Set encoding/compression parameters upon DataArray creating #397

bdestombe opened this issue Dec 24, 2024 · 0 comments

Comments

@bdestombe
Copy link
Collaborator

bdestombe commented Dec 24, 2024

Hi all,
Maybe worth a discussion first. In ./dims/attributes_encodings.py the tools are located to set encoding and compression parameters for netCDF. Currently, it is up to the user to set those encoding/compression parameters right before writing it all to disk. I like to believe that those encoding/compression parameters are sane defaults for all and we might as well set them upon the creation of the dataarrays that are part of ds.

My proposal would be to set those encoding/compression parameters at the creation of the dataarrays that form ds, such as kh, kv, top, botms, but also the stresses and later also when the model results are returned.

A few considerations:

  • One of the better compression techniques is to store floats as ints, which is allowed if the precision loss is smaller that an acceptable set with dval_max. Which could result in
    • Invalidating cache when precision is lost. But I think we could solve this in several ways.
    • Running the model twice, first with all the retreived files without precision loss and the second time from cache with precision loss results in different model outputs. The differences will be small as we can set dval_max to be sufficiently small. Nonetheless, the results can be slightly different.
  • The ds can have a parameter that prevents setting encoding/compression parameters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant