Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overwritten nodes in configurations #43

Open
JimCircadian opened this issue Aug 21, 2024 · 0 comments
Open

Overwritten nodes in configurations #43

JimCircadian opened this issue Aug 21, 2024 · 0 comments
Assignees

Comments

@JimCircadian
Copy link
Collaborator

Due to the Batch defaults that are set via namedtuple in config.py, run specific values are overwritten when specified as default in the root vars of configurations. We should identify when these are being replaced in the batch context by None and ensure they aren't overwritten, as a placeholder for some clearer work in the future.

17:20:29                 DEBUG            TASK CTX: {'batch': 4,
 'cluster': 'dummy',
 'dataset': 'test_net_ds',
 '**email**': '[email protected]',
 'epochs': 100,
 'filter_factor': 1,
 'gpus': 1,
 '**length**': '1-00:00:00',
 'mem': '128gb',
 '**nodes**': 1,
 '**ntasks**': 2,
 'preload': 'test_net_ds',
 'strategy': 'default',
 'symlinks': ['../../../data',
              '../../../ENVS*',
              '../../../loader.test.json',
              '../../../test_net_ds',
              '../../../network_datasets',
              '../../../processed',
              '../../../results']}
17:20:29                 DEBUG            TASK FUNC: Task(name='execute', args={'cmd': 'mkdir -p ./results/networks'}, value=None)
17:20:29                 DEBUG            Calling with cmd = mkdir -p ./results/networks
17:20:29                 INFO             Running command: mkdir -p ./results/networks
17:20:29                 DEBUG            Executing command mkdir -p ./results/networks, cwd unset
17:20:29                 DEBUG            Command successful
17:20:29                 INFO             Start batch: 2024-08-21 16:20:29.854505
17:20:29                 DEBUG            Batch(name='test_ensemble', templates=['icenet_train.sh.j2'], templatedir='../template', job_file='icenet_tra
in.sh', basedir='./ensemble/test_ensemble', runs=[{'seed': 42}, {'seed': 46}], maxruns=2, maxjobs=2, repeat=False, cluster=None, **email**=None, **nodes**=None
, **ntasks**=0, **length**=None, pre_batch=[], pre_run=[], post_run=[], post_batch=[{'name': 'execute', 'args': {'cmd': '/usr/bin/echo "No postprocessing in pl
ace for training ensemble"'}}])
17:20:29                 INFO             Running cycle 1
17:20:29                 INFO             Start run test_ensemble-0 at 2024-08-21 16:20:29.859752
17:20:29                 DEBUG            Run(batch='4', cluster='gpu', dataset='test_net_ds', **email**='[email protected]', epochs='10', filter_factor='1.
44', gpus='1', **length**=None, mem='128gb', **nodes**=None, **ntasks**='8', preload='test_net_ds', strategy='default', symlinks=['../../../data', '../../../ENVS*'
, '../../../loader.test.json', '../../../test_net_ds', '../../../network_datasets', '../../../processed', '../../../results'], name='test_ensemble', te
mplates=['icenet_train.sh.j2'], templatedir='../template', job_file='icenet_train.sh', basedir='./ensemble/test_ensemble', maxruns=2, maxjobs=2, repeat
=False, seed=42, idx=0, id='test_ensemble-0', dir='/data/hpcdata/users/jambyr/icenet4/pipeline.monthly/ensemble/test_ensemble/test_ensemble-0')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant