Magma via Conda Env

[foamy1881]

I'm new to this community and could use some help. How exactly does one execute Magma after installing it through Conda? I've followed the instructions and installed both magma-env-1 and magma-env-2. However, I'm at a loss regarding the next steps, specifically how to incorporate the sample sheet and execute the YAML file. Could anyone offer some guidance?

[abhi18av]

Hi @foamy1881 , thank you for your interest in the pipeline!

As this is not a feature/bug report, I am converting this into a discussion and will respond to these questions in a point-wise manner

Point-1

I've followed the instructions and installed both magma-env-1 and magma-env-2

This sounds good 👍

The next step here, I'd recommend you to do is to make sure that the WHO resistance catalog for tb-profiler has been loaded (I realized we didn't document this on Readme, I'll fix this).

NOTE: Basically the instructions, which are used to build the necessary docker containers.

⚠️ We do highly recommend usage of the containers wherever possible 🐳

1. Download magma_resistance_db_who_v1.zip and unzip it

wget https://github.com/TORCH-Consortium/MAGMA/files/14559680/resistance_db_who_v1.zip

unzip resistance_db_who

2. Activate magma-env-1, which has tb-profiler

conda activate magma-env-1

3. Move inside that folder and use tb-profiler load_library functionality to load the database

cd resistance_db_who

tb-profiler load_library ./resistance_db_who

Success, would look like this

Point-2

... how to incorporate the sample sheet ...

For this, we provide a dummy samplesheet, which you'd have to customize as per your samples and other metadata from the sequencer and sample.

Reference: Mentioned on the readme here.

Below is a formatted version of that sample CSV.

Study	Sample	Library	Attempt	R1	R2	Flowcell	Lane	Index Sequence
Study_Name	S0001	1	1	full_path_to_directory_of_fastq_files/S0001_01_R1.fastq.gz	full_path_to_directory_of_fastq_files/S0001_01_R1.fastq.gz	1	1	1
Study_Name	S0002	1	1	full_path_to_directory_of_fastq_files/S0002_01_R1.fastq.gz	full_path_to_directory_of_fastq_files/S0002_01_R2.fastq.gz	1	1	1
Study_Name	S0003	1	1	full_path_to_directory_of_fastq_files/S0003_01_R1.fastq.gz	full_path_to_directory_of_fastq_files/S0003_01_R2.fastq.gz	1	1	1
Study_Name	S0004	1	1	full_path_to_directory_of_fastq_files/S0004_01_R1.fastq.gz	full_path_to_directory_of_fastq_files/S0004_01_R2.fastq.gz	1	1	1

Point-3

... execute the YAML file

Actually the YAML is a condensed form of all inputs to the pipeline, therefore, you can pass/give it to the pipeline rather than execute in this case.

This information is covered here on Readme.

[foamy1881]

Thank you for the feedback and is running at the moment with this command line:

nextflow run 'https://github.com/TORCH-Consortium/MAGMA' \
		 -profile conda_local, server \ 
		 -r v1.1.1 \
		 -params-file  my_parameters_1.yml

However when should we use my_parameters_2.yml? do we run them separately?

[foamy1881]

First of all please excuse myself for being a newbie given I'm new to the environment to analyze TB clinical samples from clinician side. I manage to run the above and the it successfully checked the samplesheet and fastq and below is the output via docker "nextflow" command.

nextflow run 'https://github.com/torch-consortium/magma' -params-file my_parameters_1.yml -profile docker -r v1.1.1

N E X T F L O W  ~  version 23.10.1
Launching `https://github.com/torch-consortium/magma` [high_brattain] DSL2 - revision: d417232998 [v1.1.1]
executor >  local (577)
[d2/62fe5c] process > VALIDATE_FASTQS_WF:SAMPLESHEET_VALIDATION                            [100%] 1 of 1 ✔
[4e/46fff3] process > VALIDATE_FASTQS_WF:FASTQ_VALIDATOR (Study_Name.M607.L1.A1.1.1.1)     [100%] 575 of 575 ✔
[59/bc3622] process > VALIDATE_FASTQS_WF:UTILS_FASTQ_COHORT_VALIDATION (joint_name: joint) [100%] 1 of 1 ✔
Completed at: 17-Mar-2024 17:00:38
Duration    : 3h 6m 43s
CPU hours   : 12.4
Succeeded   : 577

Here is the details of my_parameters_1.yml to run the above.

# Sample contents of my_parameters_1.yml file
input_samplesheet: /media/yang/PhD_NGS1/MTBC_Myanmar_Isolates_Illumina/MTBC_MAGMA_Analysis_120424/MAGMA_samplesheet.csv
only_validate_fastqs: true
conda_envs_location: /home/yang/miniconda3/envs/mtbc-magma-env-1, /home/yang/miniconda3/envs/mtbc-magma-env-2 
median_coverage_cutoff: 5
breadth_of_coverage_cutoff: 0.95
rel_abundance_cutoff: 0.65
ntm_fraction_cutoff: 0.40
site_representation_cutoff: 0.80

Here is my issue for a beginner,

1. I must be missing something as it does not carry forward any analysis further probably I need to set a config file for my_parameters_1.yml to run them.
2. The conda_envs I have input both environments and I'm not too sure whether this is correct?
3. I found it hard for beginner/intermediate that how are we suppose to run a full analysis? can someone please guide me step-by-step guidelines, lets say I have the raw fastq files for starting? Please forgive my lack of knowledge in this and I'm learning towards analysis myself for my future TB samples.
4. can we input under the config file to have our own output dir with direction of pathway so they can create all the result analysis in this output folder?

Many thanks to everyone. Every help will be appreciated.
MAGMA_samplesheet.csv

[abhi18av]

Hi @foamy1881 ,

Actually, I must congratulate you since you're almost there. Plus, you've been able to successfully run the FASTQ_VALIDATION part.

Now, the next steps are to

1. ... it does not carry forward any analysis further

Yes, for this you'd need to put only_validate_fastqs: false in YAML file.

2. The conda_envs_location

This value should only mention the parent directory of the conda environments. In this case it should be a single value, /home/yang/miniconda3/envs/

As mentioned in the warnings on the main README and the response above , conda setups are relatively fragile therefore we expect users to follow exactly the instructions. The environment names must be magma-env-1 and magma-env-2.

You can use the examples mentioned in conda documentation to change the names.

NOTE: It is possible to change the default conda env names, but it requires digging deeper into the Nextflow config overrides and not recommended for beginners.

3. Documentation

Agree 💯 on the documentation, and happy to note that we are addressing this through creation of videos and improving the documentation. CC @TimHHH @mdediegofuertes @vrennie

4. Custom output directory

Sure, this is absolutely possible and you just need to provide a parameter called outdir

CHANGES

Therefore, in the end your YML file should look like

# Sample contents of my_parameters_1.yml file
input_samplesheet: /media/yang/PhD_NGS1/MTBC_Myanmar_Isolates_Illumina/MTBC_MAGMA_Analysis_120424/MAGMA_samplesheet.csv

outdir: /media/yang/PhD_NGS1/MTBC_Myanmar_Isolates_Illumina/results_MTBC_MAGMA_Analysis_120424


only_validate_fastqs: false
conda_envs_location: /home/yang/miniconda3/envs 
median_coverage_cutoff: 5
breadth_of_coverage_cutoff: 0.95
rel_abundance_cutoff: 0.65
ntm_fraction_cutoff: 0.40
site_representation_cutoff: 0.80

[foamy1881]

The software is currently operational, but it appears to be running on a standard setup rather than a high-performance computing (HPC) environment, leading to an overload on my 16GB PC and self-restart. It was under VM environment created with Ubuntu 22.04 image. Is there a way to enable a low-memory mode? I recall seeing this option previously, but I'm unsure how to incorporate it into the YAML file. Additionally, despite specifying the output directory in the YAML file as directed, the software still generates an output folder within the conda environment, rather than following the path outlined in the YAML file. Your help is greatly appreciated.

[abhi18av]

Hi @foamy1881 ,

but it appears to be running on a standard setup rather than a high-performance computing (HPC) environment

Ah, this was the missing piece! I didn't understand before that you were running on HPC system. At this point, I believe its best to have a meeting and try to setup the pipeline as per your cluster.

We generally don't consider HPC cluster system users as beginners but given the variety of HPC systems available (IBM LSF, SLURM, PBS/PRO) with relevant queues and the way your shared file system is setup, this depends a lot on your system configuration.

Additionally, despite specifying the output directory in the YAML file as directed, the software still generates an output folder within the conda environment, rather than following the path outlined in the YAML file.

I have a feeling that there might further context I'm missing, because the parameter should output the results under any other folder, unless it is being overridden somehow.

---

MEETING SLOTS

I am currently in Brazil, GMT-3, please share 3 options of your available slots on my address abhinavsharma [at] sun [dot] ac [dot] za

---

NOTE: For future readers / users

We are working on a transition to the nf-core standard #188 which would facilitate anyone to basically rely upon the vast documentation of nf-core ecosystem and benefit from the standardized templates that the community maintains.

[foamy1881]

Thanks will email and guess a zoom meeting will help a lot for helping me to setup a proper workflow