You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi all,
Maybe worth a discussion first. In ./dims/attributes_encodings.py the tools are located to set encoding and compression parameters for netCDF. Currently, it is up to the user to set those encoding/compression parameters right before writing it all to disk. I like to believe that those encoding/compression parameters are sane defaults for all and we might as well set them upon the creation of the dataarrays that are part of ds.
My proposal would be to set those encoding/compression parameters at the creation of the dataarrays that form ds, such as kh, kv, top, botms, but also the stresses and later also when the model results are returned.
A few considerations:
One of the better compression techniques is to store floats as ints, which is allowed if the precision loss is smaller that an acceptable set with dval_max. Which could result in
Invalidating cache when precision is lost. But I think we could solve this in several ways.
Running the model twice, first with all the retreived files without precision loss and the second time from cache with precision loss results in different model outputs. The differences will be small as we can set dval_max to be sufficiently small. Nonetheless, the results can be slightly different.
The ds can have a parameter that prevents setting encoding/compression parameters
The text was updated successfully, but these errors were encountered:
Hi all,
Maybe worth a discussion first. In
./dims/attributes_encodings.py
the tools are located to set encoding and compression parameters for netCDF. Currently, it is up to the user to set those encoding/compression parameters right before writing it all to disk. I like to believe that those encoding/compression parameters are sane defaults for all and we might as well set them upon the creation of the dataarrays that are part of ds.My proposal would be to set those encoding/compression parameters at the creation of the dataarrays that form ds, such as kh, kv, top, botms, but also the stresses and later also when the model results are returned.
A few considerations:
dval_max
. Which could result inds
can have a parameter that prevents setting encoding/compression parametersThe text was updated successfully, but these errors were encountered: