Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory resource definitions #953

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Memory resource definitions #953

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5a7e4d7
Revise mem_mb definitions to always result in an integer > 0
tsibley May 25, 2022
acb198e
Pass the tree rule's mem_mb through to IQ-TREE
tsibley May 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion workflow/snakemake_rules/export_for_nextstrain.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import re
import requests
import json
from math import ceil
from workflow.lib.persistent_dict import PersistentDict, NoSuchEntryError

ruleorder: dated_json > finalize
Expand Down Expand Up @@ -80,7 +81,7 @@ rule export_all_regions:
# Memory use scales primarily with the size of the metadata file.
# Compared to other rules, this rule loads metadata as a pandas
# DataFrame instead of a dictionary, so it uses much less memory.
mem_mb=lambda wildcards, input: 5 * int(input.metadata.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(5 * (input.metadata.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand Down
18 changes: 10 additions & 8 deletions workflow/snakemake_rules/main_workflow.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
from math import ceil

rule sanitize_metadata:
input:
metadata=lambda wildcards: _get_path_for_input("metadata", wildcards.origin)
Expand Down Expand Up @@ -803,13 +805,13 @@ rule tree:
# Multiple sequence alignments can use up to 40 times their disk size in
# memory, especially for larger alignments.
# Note that Snakemake >5.10.0 supports input.size_mb to avoid converting from bytes to MB.
mem_mb=lambda wildcards, input: 40 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(40 * (input.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
augur tree \
--alignment {input.alignment} \
--tree-builder-args {params.args} \
--tree-builder-args {params.args}' --mem {resources.mem_mb}M' \
{params.exclude_sites} \
--output {output.tree} \
--nthreads {threads} 2>&1 | tee {log}
Expand Down Expand Up @@ -839,7 +841,7 @@ rule refine:
# Multiple sequence alignments can use up to 15 times their disk size in
# memory.
# Note that Snakemake >5.10.0 supports input.size_mb to avoid converting from bytes to MB.
mem_mb=lambda wildcards, input: 15 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(15 * (input.size / 1024 / 1024))
params:
root = config["refine"]["root"],
clock_rate = config["refine"]["clock_rate"],
Expand Down Expand Up @@ -893,7 +895,7 @@ rule ancestral:
# Multiple sequence alignments can use up to 15 times their disk size in
# memory.
# Note that Snakemake >5.10.0 supports input.size_mb to avoid converting from bytes to MB.
mem_mb=lambda wildcards, input: 15 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(15 * (input.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand Down Expand Up @@ -924,7 +926,7 @@ rule translate:
# Multiple sequence alignments can use up to 15 times their disk size in
# memory.
# Note that Snakemake >5.10.0 supports input.size_mb to avoid converting from bytes to MB.
mem_mb=lambda wildcards, input: 15 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(15 * (input.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand Down Expand Up @@ -1055,7 +1057,7 @@ rule clades:
"benchmarks/clades_{build_name}.txt"
resources:
# Memory use scales primarily with size of the node data.
mem_mb=lambda wildcards, input: 3 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(3 * (input.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand All @@ -1081,7 +1083,7 @@ rule emerging_lineages:
"benchmarks/emerging_lineages_{build_name}.txt"
resources:
# Memory use scales primarily with size of the node data.
mem_mb=lambda wildcards, input: 3 * int(input.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(3 * (input.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand Down Expand Up @@ -1125,7 +1127,7 @@ rule colors:
# Memory use scales primarily with the size of the metadata file.
# Compared to other rules, this rule loads metadata as a pandas
# DataFrame instead of a dictionary, so it uses much less memory.
mem_mb=lambda wildcards, input: 5 * int(input.metadata.size / 1024 / 1024)
mem_mb=lambda wildcards, input: ceil(5 * (input.metadata.size / 1024 / 1024))
conda: config["conda_environment"]
shell:
"""
Expand Down