-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Content Size (self.size) remaining at 0 for certain zarr-related assets #2067
Comments
@bendichter -- just tagging for visibility -- as this specific dandiset contained |
The size of a zarr being zero is only significant if the zarr has been finalized. If the zarr is not finalized, it simply means that the upload was not "finished" and the checksum computed. The zarr checksum computation is how the >>> ZarrArchive.objects.filter(size=0, status=ZarrArchiveStatus.COMPLETE).count()
0 |
@jjnesbitt good to know, thanks for looking into this -- seems that it isn't so much a bug then, I think? -- albeit Zarrs that don't have the |
It may be a good idea to not show any zarrs that haven't been finalized "at least once". Technically you can further upload data to a zarr that's already been finalized, so in that case we wouldn't want to "disappear" the zarr during that secondary upload, even though the size gets reset to zero. We've had this idea in the past, but it's not been fleshed out or implemented in any way yet. We've also considered email reminders on un-finished (blob or zarr) uploads. |
while thinking about zarr "redesign" (#1892), but may be even before that -- I agree with @jjnesbitt that we should disallow minting an assets until asset was finalized. In new design (whatever it would be) it would mean that it gets a manifest, and thus that version would already be accessible etc. A complimentary behavior we might want to add (or not) is |
Issue:
Some zarr assets are registering with a size of 0 even though memory size is well above -- this issue was noticed when attempting to call
dandi download
upon Dandiset 000719 -- still investigating if other zarr-containing datasets have similar issues.Seems that there are 119 ZarrArchive objects registered with size of 0:
@waxlamp @satra @kabilar -- any idea if this could be intentional behavior? Seems that the dandisets that contain these ZarrArchives of size 0 have variable dates of posting on DANDI Archive.
e.g. https://api.dandiarchive.org/api/dandisets/000108/versions/draft/assets/db2fe61f-2874-444d-b952-02234d00f2ba/ is from a few years ago and contains a similar size of 0 on
sub-SChmi53/ses-20220114h19m16s46/micr/sub-SChmi53_ses-20220114h19m16s46_sample-13_stain-LEC_run-1_chunk-3_SPIM.ome.zarr
Cc @yarikoptic @jwodder -- is this something that I should also cross-post in
dandi-cli
repo for reference?Steps to replicate initial bug noticed
dandi download
command:dandi download DANDI:000719/draft
0 Bytes 31.0 MB 0% downloading 1256 done
Exec into Django shell --
python manage.py shell
-- then:Notice that some assets have content size of 0
The links below correspond with that assets
dandiarchive
API and UIhttps://api.dandiarchive.org/api/dandisets/000719/versions/draft/assets/23182373-62a6-4747-b6cd-ac7e37f0bb15/
https://dandiarchive.org/dandiset/000719/draft/files?location=ophys_DirectoryStore_9_29_24&page=1
The text was updated successfully, but these errors were encountered: