Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(#580) fix failing github actions tests by moving data files to Zenodo #581

Merged
merged 2 commits into from
Aug 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ build/phantom-version.h
*~
*/*~
*/*/*~
eos_binary_table.dat
helm_data.tab
forcing.dat
*.bindata
*.pyc
#*
Expand Down
29 changes: 24 additions & 5 deletions docs/developer-guide/datafiles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,29 @@ For large data files, the procedure is as follows:

2. Add the name of the file to the **.gitignore** file in the root-level
phantom directory
3. Then, send your file to [email protected] (e.g. via Dropbox) to
store on the phantom website
4. Implement the call in the code as previously using the
3. Then, upload your file(s) to a repository on zenodo.org, obtain
the DOI for the repository, and submit it in the phantom zenodo community

https://zenodo.org/communities/phantom/

This will allow the file to be downloaded
by Phantom users at runtime
4. Edit the datafiles.f90 module to include the URL for the file in the
**map_dir_to_web** routine. This routine maps the directory where the
file should be located to the URL where the file can be downloaded
from. For example, if the file is in the
**star_data_files/red_giant** directory, you would add a case to the
routine like so:

::

case('data/star_data_files/red_giant')
url = 'https://zenodo.org/records/12345678/files/'

where the URL is the URL of the repository on zenodo.org

5. Implement the call in the code as previously using the
find_phantom_datafile routine. This will automatically retrieve the
file from the web into your phantom/data directory at runtime (using
wget). Alternatively you can manually place the file in the
file from the web into your phantom/data directory at runtime.
Alternatively you can manually download the file to the
appropriate folder
2 changes: 1 addition & 1 deletion scripts/test_analysis_ce.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ rm -f *.ev *txt
# grab the data file from the server if it doesn't already exist
file=binary_01000
if [ ! -f $file ]; then
curl -k https://users.monash.edu.au/~dprice/phantom/data/tests/test_analysis_ce/binary_01000 -o binary_01000; err=$?;
curl -k https://zenodo.org/records/13163487/files/binary_01000 -o binary_01000; err=$?;
if [ $err -gt 0 ]; then
exit $err;
fi
Expand Down
48 changes: 43 additions & 5 deletions src/main/datafiles.f90
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
module datafiles
!
! Interface to routine to search for external data files
! This module just provides the url and environment variable
! settings that are specific to Phantom
! This module just provides the url and environment variable
! settings that are specific to Phantom
!
! :References: None
!
Expand All @@ -19,11 +19,14 @@ module datafiles
! :Dependencies: datautils, io, mpiutils
!
implicit none
character(len=*), parameter :: data_url = &
'https://users.monash.edu.au/~dprice/phantom/'

contains

!----------------------------------------------------------------
!+
! Find a datafile in the Phantom data directory
!+
!----------------------------------------------------------------
function find_phantom_datafile(filename,loc)
use datautils, only:find_datafile
use io, only:id,master
Expand All @@ -34,7 +37,8 @@ function find_phantom_datafile(filename,loc)

search_dir = 'data/'//trim(adjustl(loc))
if (id == master) then ! search for and download datafile if necessary
find_phantom_datafile = find_datafile(filename,dir=search_dir,env_var='PHANTOM_DIR',url=data_url)
find_phantom_datafile = find_datafile(filename,dir=search_dir,env_var='PHANTOM_DIR',&
url=map_dir_to_web(trim(search_dir)))
endif
call barrier_mpi()
if (id /= master) then ! find datafile location, do not attempt to download it
Expand All @@ -44,4 +48,38 @@ function find_phantom_datafile(filename,loc)

end function find_phantom_datafile

!----------------------------------------------------------------
!+
! Find the web location for files that are not in the Phantom
! git repo, which need to be downloaded into the data directory
! at runtime
!+
!----------------------------------------------------------------
function map_dir_to_web(search_dir) result(url)
character(len=*), intent(in) :: search_dir
character(len=120) :: url

!print*,' search_dir=',trim(search_dir)
select case(search_dir)
case('data/eos/mesa')
url = 'https://zenodo.org/records/13148447/files/'
case('data/eos/shen')
url = 'https://zenodo.org/records/13163155/files/'
case('data/eos/helmholtz')
url = 'https://zenodo.org/records/13163286/files/'
case('data/forcing')
url = 'https://zenodo.org/records/13162225/files/'
case('data/velfield')
url = 'https://zenodo.org/records/13162515/files/'
case('data/galaxy_merger')
url = 'https://zenodo.org/records/13162815/files/'
case('data/starcluster')
url = 'https://zenodo.org/records/13164858/files/'
case default
url = 'https://users.monash.edu.au/~dprice/'//trim(search_dir)
end select
!print*,'url=',trim(new_url)

end function map_dir_to_web

end module datafiles
5 changes: 0 additions & 5 deletions src/main/utils_datafiles.f90
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,6 @@ function find_datafile(filename,dir,env_var,url,verbose) result(filepath)
! try to download the file from a remote url
!
my_url = url
if (present(dir)) then
if (len_trim(dir) > 0) my_url = trim(url)//'/'//trim(dir)//'/'
else
my_url = url
endif
call download_datafile(trim(my_url),trim(mydir),trim(filename),filepath,ierr)
if (ierr == 0) then
inquire(file=trim(filepath),exist=iexist)
Expand Down
Loading