You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Before each operation that potentially downloads enough data to fill the target storage, the service should check the available storage.
There are three steps where this would be useful:
git clone (gin get)
git annex get (gin get-content)
zip
Step 1 might be an issue since it's not straightforward to know the size of the repository before cloning. We might have to solve that by adding the functionality to report repository sizes on GIN Web (Gogs).
Step 2 is pretty straightforward. For one, git annex will refuse to download a file if there is not enough local storage. That said, if several files are being downloaded, we might reach near-capacity before the limit is hit, so the service should check for the entire download size before running get-content. Git annex provides this information via git annex info (which also supports --json output).
When we add the repo size reporting functionality to GIN Web, this step could be merged into step 1.
Step 3 essentially means we would require twice the space required for step 2. Assuming no compression (worst case), the zip file would take up the same amount of storage as the repository, so the storage needed to clone & get-content will be doubled when creating the zip file.
Essentially, if we can know ahead of time the storage requirements for a repository, we can safely clone and zip if there is twice as much available space.
NOTE: This becomes tricky when multiple repositories are being registered simultaneously. We should consider having workers report the storage space they're planning to use to the other workers.
The text was updated successfully, but these errors were encountered:
Before each operation that potentially downloads enough data to fill the target storage, the service should check the available storage.
There are three steps where this would be useful:
Step 1 might be an issue since it's not straightforward to know the size of the repository before cloning. We might have to solve that by adding the functionality to report repository sizes on GIN Web (Gogs).
Step 2 is pretty straightforward. For one, git annex will refuse to download a file if there is not enough local storage. That said, if several files are being downloaded, we might reach near-capacity before the limit is hit, so the service should check for the entire download size before running
get-content
. Git annex provides this information viagit annex info
(which also supports--json
output).When we add the repo size reporting functionality to GIN Web, this step could be merged into step 1.
Step 3 essentially means we would require twice the space required for step 2. Assuming no compression (worst case), the zip file would take up the same amount of storage as the repository, so the storage needed to clone & get-content will be doubled when creating the zip file.
Essentially, if we can know ahead of time the storage requirements for a repository, we can safely clone and zip if there is twice as much available space.
NOTE: This becomes tricky when multiple repositories are being registered simultaneously. We should consider having workers report the storage space they're planning to use to the other workers.
The text was updated successfully, but these errors were encountered: