Pipeline finishing with no results

[jackbrougher]

Like the title says - I ran the pipeline overnight and when I opened this morning it said it was done/didn't report any errors. However, there was nothing in my output folder except for the temp folder?

[FEI38750]

The pipeline usually runs longer than overnight >8 hours. Could you please show what's in your temp folder? Or do you have a job log file on HPC that records all the messages during the running, which benefits troubleshooting?

[FEI38750]

Like the title says - I ran the pipeline overnight and when I opened this morning it said it was done/didn't report any errors. However, there was nothing in my output folder except for the temp folder?

Has your issue been solved?

[jackbrougher]

Sorry to be a drag, I've been trying to fix this on my own. I believe I'm running into a memory issue, but I'm honestly unsure why.
I set the pipeline up on an r4.16xlarge EC2 instance on AWS which should more than meet the computational requirements. The pipeline itself + my data are on a mounted storage volume 3 TB in size. About 600 GB of that is the pipeline, and another 500 GB are my data. So even with a large output, there should still be around 2 TB left in free storage?

[FEI38750]

Empirically, during the MTD running for bulk RNA-seq data, it may take ~15x higher storage requirement than the compressed fastq (e.g., fq.gz) input files, or ~3x higher than the uncompressed fastq input files. For example, 500GB uncompressed fastq input files may need an additional ~1.5TB storage space for MTD running.
Most of this storage is occupied by the temp folder, which facilitates users to check or utilize the intermediate files such as unclassified reads. You can delete the temp file to release the storage space after running.

[jackbrougher]

I increased my storage with that information, but I've hit a new issue. My volume is mounted as mentioned as /newvolume/. I've placed my fastq's (they're all compressed as fastq.gz's), and in a sub folder under /newvolume/MTD/expdata/. The CSV sample sheet is also in that folder.

Currently, when I run I'm hitting this issue?

bash MTD/MTD.sh -i MTD/expdata/samplesheet.csv -o MTD/output/ -h 9606 -t 64
MTD directory is /newvolume/MTD
MTD/MTD.sh: line 78: cd: MTD/expdata: No such file or directory
cat: samplesheet.csv: No such file or directory
MTD/MTD.sh: line 92: cd: MTD/output//temp: No such file or directory
find: ‘MTD/expdata’: No such file or directory
find: ‘MTD/expdata’: No such file or directory
cat: MTD/expdata/samplesheet.csv: No such file or directory
The samples' fastq files in the input folder do not match with your samplesheet.csv
Please double-check with the samplesheet.csv and input files. Please ensure no other fastq files are under the input folder and its subfolders. You can refer to the user guide on https://github.com/FEI38750/MTD.

[FEI38750]

Please use the full path such as ~/MTD/MTD.sh

[jackbrougher]

Will report back once I fix that - I'm receiving an error: /home/ubuntu/MTD/MTD.sh: No such file or directory when I run ~/MTD/.

Which makes sense because my install is under newvolume/MTD/. I'm assuming something is messed up with either the permissions or the volume itself, though it does appear to be mounted correctly. I'll get this fixed an report back.