Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use pooch for data management #51

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nstarman
Copy link

@nstarman nstarman commented Nov 20, 2024

Fixes #50.

Pooch works for all shell types (because it doesn't use shell commands).
Pooch will try multiple times if the connection is interrupted. It does file caching and re-downloading. The cache location is customizable. I made target_dir be a symlink, so the file is only ever downloaded once.

The Pooch registry is called strato because "Strato is a minor character in William Shakespeare's play Julius Caesar. Although he doesn't have a significant role, he serves as a loyal and trustworthy friend to Brutus" link

@nstarman
Copy link
Author

nstarman commented Nov 20, 2024

I didn't implement it because I didn't want to change the API, but IMO strato should be imported into the top namespace, so one could do

from brutus import strato
from brutus import seds
from brutus import filters

isochrone = seds.Isochrone(
    filters=filt,
    nnfile=strato.fetch("nn_c3k.h5"),  # will auto-download if necessary
    mistfile=strato.fetch("MIST_1.2_iso_vvcrit0.0.h5"),
)

@nstarman
Copy link
Author

I also didn't do it here, but with strato there's also not much point to all the fetch_X utilities... removing them would be another nice simplification. It does mean removing the symlink generation, but that's an easy single utility function

def fetch(name, symlink_dir):
    fpath = strato.fetch(name)
    fpath = pathlib.Path(fpath)
    
    target_path = pathlib.Path(symlink_dir).resolve() / name
    target_path.mkdir(parents_ok=True, exists_ok=True)
    target_path.symlink_to(fpath)
    return target_path

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: download scripts only work in bash
1 participant