Gentoo based Dockerfile (presumably reproducible) to provide bids-app #41
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It is not yet finished - did not even try FSL installation - AFNI installation relies on incomplete fixes proposed in ReproNim/neurodocker#601 - moved away attempt to install bids-validator (would be useful) due those failing too ATM
|
FWIW there is an effort by @TheChymera to provide gentoo support in neurodocker:
|
It is based on WIP done in ReproNim/neurodocker#611 to add support for Gentoo. But it is just a dirty draft so produced docker file is not good.
yarikoptic
changed the title
|
@yarikoptic thank you for your contribution! I'll be traveling tomorrow through the weekend but I'll review it as soon as I can. @ericearl thanks for the ping and following up on this! |
|
might not be good enough yet lead to a quick generation of bunch of filed a dedicated I might add patching directly to this PR... we will see... |
otherwise it was just placing everything, including the RUN into value of gentoo_hash (I guess) and not really installing anything
…all fail etc from git relevant info https://bugs.gentoo.org/824782#c17
…upported) and using --depth 1
|
8b72b07 seems like a good idea, tested it and it appears to also be faster: [deco]/tmp/test_gentoo_old ❱ time bash script.sh
mkdir: cannot create directory ‘repo’: File exists
/tmp/test_gentoo_old/repo /tmp/test_gentoo_old
Initialized empty Git repository in /tmp/test_gentoo_old/repo/.git/
warning: filtering not recognized by server, ignoring
remote: Enumerating objects: 4926206, done.
remote: Counting objects: 100% (53175/53175), done.
remote: Compressing objects: 100% (4372/4372), done.
remote: Total 4926206 (delta 49271), reused 48803 (delta 48803), pack-reused 4873031 (from 1)
Receiving objects: 100% (4926206/4926206), 1.18 GiB | 14.22 MiB/s, done.
Resolving deltas: 100% (3568243/3568243), done.
From https://anongit.gentoo.org/git/repo/gentoo
* branch 2d25617a1d085316761b06c17a93ec972f172fc6 -> FETCH_HEAD
Updating files: 100% (88342/88342), done.
HEAD is now at 2d25617a1d08 sys-kernel/gentoo-sources: drop 6.7.8
/tmp/test_gentoo_old
real 2m5.778s
user 2m14.753s
sys 0m16.466s
[deco]/tmp/test_gentoo_old ❱ cat script.sh
mkdir repo
pushd repo
git init .
git remote add origin https://anongit.gentoo.org/git/repo/gentoo.git
git fetch --filter="blob:none" origin 2d25617a1d085316761b06c17a93ec972f172fc6
git reset --hard 2d25617a1d085316761b06c17a93ec972f172fc6
rm -rf .git
popdvs [deco]/tmp/test_gentoo_new ❱ time bash script.sh
Cloning into 'repo'...
remote: Enumerating objects: 106861, done.
remote: Counting objects: 100% (106861/106861), done.
remote: Compressing objects: 100% (92429/92429), done.
remote: Total 106861 (delta 21371), reused 66212 (delta 13400), pack-reused 0 (from 0)
Receiving objects: 100% (106861/106861), 52.58 MiB | 12.38 MiB/s, done.
Resolving deltas: 100% (21371/21371), done.
Updating files: 100% (88772/88772), done.
/tmp/test_gentoo_new/repo /tmp/test_gentoo_new
remote: Enumerating objects: 27260, done.
remote: Counting objects: 100% (27249/27249), done.
remote: Compressing objects: 100% (15637/15637), done.
remote: Total 17400 (delta 7719), reused 6267 (delta 1743), pack-reused 0 (from 0)
Receiving objects: 100% (17400/17400), 8.97 MiB | 7.35 MiB/s, done.
Resolving deltas: 100% (7719/7719), completed with 6078 local objects.
From https://anongit.gentoo.org/git/repo/gentoo
* branch 2d25617a1d085316761b06c17a93ec972f172fc6 -> FETCH_HEAD
HEAD is now at 2d25617a1 sys-kernel/gentoo-sources: drop 6.7.8
/tmp/test_gentoo_new
real 0m26.527s
user 0m5.100s
sys 0m2.680s
[deco]/tmp/test_gentoo_new ❱ cat script.sh
git clone --depth 1 https://anongit.gentoo.org/git/repo/gentoo.git repo
pushd repo
git fetch --depth 1 origin 2d25617a1d085316761b06c17a93ec972f172fc6
git reset --hard 2d25617a1d085316761b06c17a93ec972f172fc6
rm .git -rf
popdI'll upstream this to my Containerfile template as well. |
…v python3
=== Do not change lines below ===
{
"chain": [],
"cmd": "sed -i -e 's,/usr/local/bin/python3,/usr/bin/env python3,g' src/deface.py src/generate_mappings.py src/prepare_shareable.py src/prepare_to_share.py src/register.py src/run.py",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
It is a hardcoded assumption on a configuration of the environment, should not be there!
Someone could add a CLI option like --on-error=error,continue,ignore so that if continue it would continue but error at the end with some summary etc, and ignore just exit normally
Although does not matter much for this particular case, aiming for NeuroDocker - we do need to have produced Dockerfile self contained. Hence here we are practicing on how to make it happen
yarikoptic
commented
Jun 5, 2024
Dockerfile
Outdated
| COPY --from=portage /var/db/repos/gentoo /var/db/repos/gentoo | ||
|
|
||
| ARG gentoo_hash=2d25617a1d085316761b06c17a93ec972f172fc6 | ||
| ARG science_hash=73916dd3680ffd92e5bd3d32b262e5d78c86a448 |
There was a problem hiding this comment.
@TheChymera could you please provide new hashes needed here for the new/fixed up build and also add a patch for AFNI to reflect change like https://github.com/afni/afni/pull/650/files ? or could you just incorporate it to AFNI package so we could build/test?
There was a problem hiding this comment.
I was planning to, but after testing the fix doesn't seem to work. collecting more logs so I can report in the upstream issue exactly what's going on how it was built etc.
There was a problem hiding this comment.
Report here → afni/afni#639 (comment)
There was a problem hiding this comment.
PR here (tested with this repo, for some reason the image takes a long time to build compared to the neurodocker test) → yarikoptic#1
Commot on the gentoo+neurodocker PR: ReproNim/neurodocker@758951b
Updated hashes to include AFNI nifti-clib fix from upstream
|
With @TheChymera tune ups to AFNI packages we finally have something which works! ended with saving a number of files .duct/logs/2024.06.09T10.58.56-1623046_info.json | 1 +
.duct/logs/2024.06.09T10.58.56-1623046_stderr | 0
.duct/logs/2024.06.09T10.58.56-1623046_stdout | 95 ++++++++++++
.duct/logs/2024.06.09T10.58.56-1623046_usage.json | 50 +++++++
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0001/ses-01/anat/defacing_pipeline.log | 165 +++++++++++++++++++++
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0001/ses-01/anat/sub-0001_ses-01_acq-MPRAGEXp3X08mm_T1w.json | 839 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0001/ses-01/anat/sub-0001_ses-01_acq-MPRAGEXp3X08mm_T1w.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0133/ses-01/anat/defacing_pipeline.log | 165 +++++++++++++++++++++
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0133/ses-01/anat/sub-0133_ses-01_acq-MPRAGEXp3X08mm_T1w.json | 839 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0133/ses-01/anat/sub-0133_ses-01_acq-MPRAGEXp3X08mm_T1w.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/defacing_QC/sub-0001/ses-01/acq-MPRAGEXp3X08mm/T1w/defaced.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/defacing_QC/sub-0001/ses-01/acq-MPRAGEXp3X08mm/T1w/orig.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/defacing_QC/sub-0133/ses-01/acq-MPRAGEXp3X08mm/T1w/defaced.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/defacing_QC/sub-0133/ses-01/acq-MPRAGEXp3X08mm/T1w/orig.nii.gz | 1 +
derivatives/dss-defacing-0.0.5-1/logs/anat_unavailable.txt | 3 +
derivatives/dss-defacing-0.0.5-1/logs/t1_unavailable.txt | 0
derivatives/dss-defacing-0.0.5-1/primary_to_others_mapping.json | 14 ++
17 files changed, 2177 insertions(+)which I am yet to digest how to take advantage of (in particular for QC). Meanwhile would file some "operational" issues. |
|
FWIW, generally the image works, so I am taking out of draft. It doesn't work for my dataset due to various |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODOs
gentoo/filesafnipackage in gentoo--versionjust causes--helpprintout. github releases mention 1.0.1 ...