Readthedocs documentation #160

ddtm · 2017-03-02T19:34:44Z

Addresses #113

Updating fork master

MarcCote

This is awesome. Thank you for taking the time of doing this. I pointed out some typos but overall this is all good.

MarcCote · 2017-03-03T12:19:56Z

docs/source/autoresume.rst

+**must be resumable**, i.e. be capable of restoring their state after being
+killed and rerun.
+
+You can engage the autoresumption by passing ``-m`` or ``--autoresume`` during


-m -> -r

MarcCote · 2017-03-03T12:24:12Z

docs/source/usage.rst

+
+* Each invokation of ``smart-dispatch`` creates a so-called **batch** of **jobs**. Smart Dispatch will do its best to create as many simultaneous jobs so as to effecitvely utilze the allocated resources.
+* Each job is basically a single PBS file that is run by the queue management system on the cluster (either ``msub`` or ``qsub``).
+* A job spawns mulitple concurrent **workers** that all cooperate to execute the requested commands.


mulitple -> multiple

Note: You will need to adapt this line based on how you change line 14. (See comment)

MarcCote · 2017-03-03T12:25:54Z

docs/source/usage.rst

+This directory holds generated PBS files (``job_commands_{pbs_index}.sh``) as well as three command lists:
+
+``commands.txt``:
+    A list pending commands (this is where the workers are taking their next commands to execute from).


list pending -> list of pending

MarcCote · 2017-03-03T12:26:55Z

docs/source/usage.rst

+``logs/``
+^^^^^^^^^
+
+Output and error logs in are saved in this directory. The root level contains logs for actual commands. There are also two additional subfolder:


in are -> are

subfolder -> subfolders

MarcCote · 2017-03-03T12:33:06Z

smartdispatch.sublime-project

@@ -0,0 +1,8 @@
+{


Maybe it was not intended but I'd rather not add *.sublime-projet or any other file related to IDE/text editor. Each developer has probably its own customized version.

coveralls · 2017-04-12T21:13:32Z

Coverage remained the same at 94.532% when pulling e8a7d0a on ddtm:readthedocs into c94f08e on SMART-Lab:master.

bouthilx · 2017-07-20T15:21:27Z

docs/source/usage.rst

+
+In order to understand the contents of the generated folders/files, it's good to know how ``smart-dispatch`` deals with **commands** that a user requests to launch on the cluster:
+
+* Each invokation of ``smart-dispatch`` creates a so-called **batch** of **jobs**. Smart Dispatch will do its best to create as many simultaneous jobs so as to effecitvely utilze the allocated resources.


invokation -> invocation

I stumble each time I try to read the second sentence. 😟

What about something like this?

Smart Dispatch will distribute commands to jobs such that each of the latter uses an entire node. Jobs may run many commands concurrently if necessary to use a maximum number of cores and GPUs. The distribution is based on number of cores per node / per command and number of GPUs per node / per command.

bouthilx · 2017-07-20T17:29:22Z

docs/source/usage.rst

+* Each invokation of ``smart-dispatch`` creates a so-called **batch** of **jobs**. Smart Dispatch will do its best to create as many simultaneous jobs so as to effecitvely utilze the allocated resources.
+* Each job is basically a single PBS file that is run by the queue management system on the cluster (either ``msub`` or ``qsub``).
+* A job spawns mulitple concurrent **workers** that all cooperate to execute the requested commands.
+* Each worker (basically, a python script) is executing commands sequentially.


Is it the worker or the command that is basically a python script? If the prior, I don't understand why it needs to be specified.

bouthilx · 2017-07-21T15:15:12Z

@ddtm Did you have any problem installing sphinxcontrib-autoprogram? I installed it (version 0.1.3) and I got an error when compiling the doc because of a p2-p3 incompatibilty of textwrap.

Exception occurred:
  File "/home/bouthilx/.virtualenvs/smartdispatch/local/lib/python2.7/site-packages/sphinxcontrib/autoprogram.py", line 223, in make_rst
    yield textwrap.indent(usage, '    ')
AttributeError: 'module' object has no attribute 'indent'
The full traceback has been saved in /tmp/sphinx-err-lVwo2K.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Makefile:20: recipe for target 'html' failed
make: *** [html] Error 1

I had to pull a fresh version from bitbucket and reinstall it to be able to compile the doc.

bouthilx · 2017-07-21T15:17:13Z

@ddtm I know you currently are outside for an internship, but would it be possible for you the make the small fix for the PR?

bouthilx · 2017-08-29T18:02:57Z

@mgermain We could use sphinx-argparse instead of sphinxcontrib-autoprogram. See a comparison here.

mgermain · 2017-08-30T14:45:31Z

If it does everything that sphinxcontrib-autoprogram does and is better supported I have no objection. @ddtm <https://github.com/ddtm> might.

…

On Tue, Aug 29, 2017 at 2:02 PM, Xavier Bouthillier < ***@***.***> wrote: @mgermain <https://github.com/mgermain> We could use sphinx-argparse <http://sphinx-argparse.readthedocs.io/en/latest/sample.html> instead of sphinxcontrib-autoprogram. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBvm_hWHZzWwM_5RFvpKc_0gGpsqB4wks5sdFJSgaJpZM4MRbV0> .

Why: sphinxcontrib-autoprogram's pip version does not support python 3. There is a more recent version in bitbucket's repository but the installation is tedious for the few features it gives. (See comment SMART-Lab#160 (comment) for more details on the problem) How: sphinxcrontrib-autoprogram was only used to generate automatic documentation of the script smart-dispatch so the replacement was trivial. The output is slightly different, arguments being included in tables for sphinx-argparse while not for sphinxcrontrib-autoprogram. Overall sphinx-argparse seems good enough.

ddtm and others added 6 commits November 11, 2016 13:57

Sublime project

a08efa3

Merge https://github.com/SMART-Lab/smartdispatch

f276026

Updating fork master

Merge remote-tracking branch 'upstream/master'

11b1d14

Ported original docs to Sphinx

e86db9d

Rolled back to the default theme

2841454

Extended docs

bbaa074

MarcCote approved these changes Mar 3, 2017

View reviewed changes

ddtm added 5 commits April 12, 2017 14:51

Autoprogram dependency added

691ce5a

Readthedocs config added

248a936

Updated rtd config

881fd43

Config bugfix

3f13f38

Sphinx hacks to find the scripts path

e8a7d0a

bouthilx reviewed Jul 20, 2017

View reviewed changes

IshmaelBelghazi added a commit to IshmaelBelghazi/smartdispatch that referenced this pull request Aug 8, 2017

Adresses SMART-Lab#160

151020e

bouthilx self-assigned this Aug 29, 2017

bouthilx added CCW pending labels Aug 29, 2017

bouthilx pushed a commit to bouthilx/smartdispatch that referenced this pull request Sep 14, 2017

Adresses SMART-Lab#160

db9a31a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Readthedocs documentation #160

Readthedocs documentation #160

ddtm commented Mar 2, 2017

MarcCote left a comment

MarcCote Mar 3, 2017

MarcCote Mar 3, 2017

bouthilx Jul 20, 2017 •

edited

Loading

MarcCote Mar 3, 2017

MarcCote Mar 3, 2017 •

edited by bouthilx

Loading

MarcCote Mar 3, 2017

MarcCote Mar 3, 2017

coveralls commented Apr 12, 2017

bouthilx Jul 20, 2017

bouthilx Jul 20, 2017

bouthilx Jul 20, 2017

bouthilx commented Jul 21, 2017

bouthilx commented Jul 21, 2017

bouthilx commented Aug 29, 2017 •

edited

Loading

mgermain commented Aug 30, 2017 via email


		In order to understand the contents of the generated folders/files, it's good to know how ``smart-dispatch`` deals with commands that a user requests to launch on the cluster:

		* Each invokation of ``smart-dispatch`` creates a so-called batch of jobs. Smart Dispatch will do its best to create as many simultaneous jobs so as to effecitvely utilze the allocated resources.

Readthedocs documentation #160

Are you sure you want to change the base?

Readthedocs documentation #160

Conversation

ddtm commented Mar 2, 2017

MarcCote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bouthilx Jul 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcCote Mar 3, 2017 • edited by bouthilx Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Apr 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bouthilx commented Jul 21, 2017

bouthilx commented Jul 21, 2017

bouthilx commented Aug 29, 2017 • edited Loading

mgermain commented Aug 30, 2017 via email

bouthilx Jul 20, 2017 •

edited

Loading

MarcCote Mar 3, 2017 •

edited by bouthilx

Loading

bouthilx commented Aug 29, 2017 •

edited

Loading