ISSUE-177 Parallelize SAM algorithm in mineraly.py using Joblib

[aheermann]

Adds joblib functionality to mineral.py as specified in #177. Will be one possible parallelization option that can be specified in config file

[lewismc]

@aheermann thanks for opening this PR. Please resolve above conflicts. Thank you.

[lewismc]

Hi @aheermann thanks for posting your PR. Please consider and address the comments in my review.

[lewismc]

@aheermann this is much better. Thanks for updating.
Please see above, specifically resolve conflict(s) in pycoal/mineral.py.
Additionally, now that #161 is addressed via #178 you should merge your branch with master. All of the CI check above need to pass before we merge this into master.
Please let me know when I can test this out... right now I can't.
Thank you

[lewismc]

Excellent @aheermann this looks much better.
Can you please scope out the test regressions.
We can discuss this on Friday if you want... I will investigate in the meantime as well.

[coveralls]

Coverage decreased (-1.5%) to 65.47% when pulling 3faa8a3 on ISSUE-177 into 1f0b03d on master.

[lewismc]

I'm going to pull the code locally @aheermann and test. Please standby before merging thank you. Excellent job.

[lewismc]

@capstone-coal/19-usc-capstone-team please be in a position to discuss and compare results of this review by tomorrows meeting. Thank you all in advance.

[lewismc]

@aheermann please update this PR to resolve conflict.

@capstone-coal/19-usc-capstone-team has anyone else tested this PR out?

[Luner]

During our meeting I believe @aheermann mentioned looking into the issues involving the memory leak with joblib. He mentioned that there is a known issue within the library.

[lewismc]

@aheermann is this ready too be tested again then? Thanks for pushing most recent commit.

[lewismc]

PING here... do you have a status update please?

[aheermann]

@lewismc, this should have fixed the memory leak, as long as you make sure you have psutil installed. Past that, I imagine this branch will not be merged to master, but instead be combined with the pytorch one, for the config file branch.

[lewismc]

@aheermann I'll test this right now and provide feedback.

Past that, I imagine this branch will not be merged to master, but instead be combined with the pytorch one, for the config file branch.

No let's merge this one first. We can then update #199 and finally #198

[lewismc]

BTW, installation on < python: stable 3.7.5 resulted in failure. Upgrading to Python 3.7.5 fixed that. Just in case anyone tries.

[lewismc]

Hi @aheermann this has fixed the issue. See below for log

2019-11-20 10:41:33,058 Starting mineral classification with input file 'avng.jpl.nasa.gov/AVNG_2015_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr' and spectral library '../pycoal/tests/s07_AV95_envi.hdr'.
2019-11-20 10:41:33,058 Instantiated Mineral Classifier with following specification: -classifier function 'SAM'
2019-11-20 10:41:33,058 Starting generation of three-band RGB image from input file: 'avng.jpl.nasa.gov/AVNG_2015_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr' with following RGB values R: '680.0', G: '532.5', B: '472.5'
2019-11-20 10:41:41,134 Saving RGB image as 'ang20150420t182050_corr_v1e_img_rgb.hdr'
2019-11-20 10:41:41,254 Completed RGB image generation. Time elapsed: '0:00:08'
2019-11-20 10:41:41,302 Starting Mineral Classification for image 'avng.jpl.nasa.gov/AVNG_2015_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr', saving classified image to 'ang20150420t182050_corr_v1e_img_class.hdr'
2019-11-20 10:41:41,384 Classifying a 14674X739 image
2019-11-20 18:26:23,455 Completed Mineral Classification. Time elapsed: '7:44:42'

Excellent work here. If we take into consideration that the initial processing time was well over a day... or maybe even longer, then this represents a huge speedup. I'm going to go ahead and merge this into master branch. Excellent work.