self.running_procs becomes negative, leading to AssertionError

[BolunThompson]

Possibly low priority since I only found the problem while fixing another bug. Minimal test case:

echo "abc" > /tmp/data.txt
cat /tmp/data.txt
echo
echo
echo

Exception:

Traceback (most recent call last):
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 535, in <module>
    main()
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 531, in main
    scheduler.run()
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 510, in run
    self.parse_and_run_cmd(input_cmd)
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 444, in parse_and_run_cmd
    response = self.compile_and_add(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 289, in compile_and_add
    self.wait_unsafe()
  File "/home/bolun/pash/compiler/pash_compilation_server.py", line 433, in wait_unsafe
    assert self.running_procs == 1
           ^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Checking if self.running_procs already equals zero before decrementing it in remove_process does solve the immediate issue, but ideally the number should already match up.

[angelhof]

Interesting, it seems to have to do with the receiving of exit messages from the runtime. The daemon somehow does not handle some of them correctly so it might decrement running processes more than once? It is not priority bug, but should be good if someone wants to take a stab.

[BolunThompson]

This works on the ubuntu 20.04 container (vs the 24.04 system I develop on).

Looking at this further, I believe this is the same issue as #738, since, similarly, echo $(./test.sh) works while ./test.sh leads to the exception above.