Concern: code structure

[matejpavlovic]

Original concern

Mir's event-based architecture seems to induce
code fragmentation since handling of individual events is supposed
to be simple, fast, and non-blocking, i.e. expensive or complex
operations should be split or delegated to separate modules.
Admittedly, one can still maintain decent level of code readability
by placing logically connected event handlers next to each other in
source code. However, splitting otherwise logically continuous
operations into multiple event handlers makes them non-atomic. This
can become a source of concurrency bugs and complicate logical
reasoning about correctness of the implementation.

(copied @sergefdrv 's post in umbrella discussion #46 )

[matejpavlovic]

Yes, the event-based architecture does sometimes require the programmer to split a logically continuous operation in multiple event handlers.
This brings about some inconvenience, since context needs to be explicitly maintained between the handler invocations,
as compared to just using variables within the same handler function.

In particular, this is exclusively the case of cryptographic operations (signing / verifying, hashing).
I think that these are the only "artificlial" events that split up logically continuous computation.
Other separation into events seems rather natural to me, e.g. having separate handlers for processing incoming messages or timer events.
But admittedly, the use of cryptographic operations in a protocol implementation is not negligible and,
the way it is currently implemented, does induce some splitting of event handler code.

I can also imagine other operations that break the control flow (e.g. persisting to durable storage or other I/O like sending messages),
but those usually do not need to be executed in the middle of an event handler, as they usually do not produce a result
required for immediate continuation of the computation.

This leaves us with cryptographic operations.
Theoretically, the cryptographic operations could be performed in-line, without breaking the control flow
even without losing modularity (which could be maintained through dependency injection directly in the protocol module).
In the current design, splitting the control flow on cryptographic operations is done exclusively for performance reasons.
Since a single thread is executing the protocol logic,
we should avoid it performing cryptographic (or otherwise expensive) operations on the critical path of its execution.
Moreover, those expensive operations need to be parallelized to achieve good performance.

Fundamentally, this is the options that I see to address the concern of event handler fragmentation:

1. Perform computationally expensive operations directly as part of the protocol logic
   • a) Still handled by a single thread.
     • This results in the simplest and least error-prone code, easy to read and write.
     • Still allows for natural traceability and reproducibility of execution.
     • I am convinced, however that reasonable performance cannot be achieved in this case.
   • b) Parallelize the execution of the protocol logic
     • If done right, the implementation could perform well, exploiting the parallelization.
     • In this case we would need to introduce explicit synchronization, which is, in my view,
       a greater potential source of errors than explicit splitting of event handlers only where needed,
       since it introduces very fine-grained parallelism.
     • This approach introduces true non-determinism in the protocol execution, making it very hard to analyze and reason about.
     • Reproducibility and traceability of execution would be significantly harder to achieve.
2. Perform computationally expensive operations off the critical path of protocol execution.
   • a) Single-threaded protocol execution
     • This is the approach currently taken.
     • Offers a natural way of tracing and reproducing the execution.
     • Can be expected to offer decent performance (if it ends up being insufficient, still allows for more explicit parallelism).
     • Does not require error-prone synchronization.
     • Clutters the code to some extent.
   • b) Multi-threaded protocol execution
     • The only advantage is potentially slightly better performance compared to 2a),
       but for the price of basically all disadvantages mentioned above.

I think that the above options apply regardless of whether we adopt an event-based model or not
and the trade-offs are inherent to parallel vs. sequential execution.
When talking about the danger of concurrency bugs, I think that the current approach is in fact very favorable.
Some concurrency (a certain form of "non-atomicity" of the event handlers) is inevitable for performance reasons.
We reduce the concurrency that needs to be reasoned about to the bare necessary minimum,
so we actually reduce the danger of concurrency bugs compared to other approaches that involve finer-grained concurrency.
Based on the above reasoning, the current approach seems to me like a good compromise.

That said, I'm still very open to exploring other options for addressing the concern.
Maybe there are options that I have not thought of and that do not fall in any of the categories described above.
There also might be other advantages and disadvantages not mentioned above.
Or there might be ways to keep the same architecture, but design abstractions on top of it that make the programmer's life easier.
All ideas welcome :)

[sergefdrv]

This brings about some inconvenience, since context needs to be explicitly maintained between the handler invocations, as compared to just using variables within the same handler function.

More than that, it can break logical atomicity of the operation and introduce additional concurrency. Spreading otherwise logically atomic operation across multiple individual event handlers requires one to take extra care in order to ensure consistent execution of the operation despite non-atomicity, which may not be immediately obvious and require alternative forms of "synchronization" (e.g. flag variables). I expect we will see enough flaky concurrency bugs induced this way.

If we want to rely on the event-based architectures to enjoy execution traceability then we should handle each piece of interest as a separate event. However, this will require additional boilerplate code which no one likes to write. It would also introduce additional points of concurrency that no one likes to deal with. We'd more likely to end up with rather coarse-grained execution traces.

On the other hand, relying on the event-based architecture to control non-deterministic events and achieve reproducibility of test scenarios only requires the capability of simulation with controlled injection of otherwise non-deterministic events. It seems to boil down just to IO (e.g. communication) and local real-time clocks. One could achieve the same level of control by using dependency injection and utilizing simulation-specific implementation of abstract IO and timer components.

More importantly, the event-based architecture seems to induce coarse-grained modularization because of the additional efforts required to deal with defining, emitting, and dispatching events. Fine-grained modularity of any complex system practically demands multiple layers of abstraction which doesn't seem to scale well in the event-based architecture because it would require additional effort on each level of event dispatching.

One could imagine keeping the coarse-grained modularity of the event-based approach at high level, but achieve finer-grained levels modularization withing individual event processing modules by other means, e.g. dependency injection. However, the requirement to process events strictly sequentially would significantly restrict the lower-level modularity mechanisms, e.g. not allow injecting dependencies that may potentially block or require execution of concurrent operations. Thus I expect we will achieve only limited modularization with the event-centric approach.

[matejpavlovic]

I think the only way of, strictly speaking, not "introducing additional concurrency" is going for option
1a) where all computation, including the expensive / blocking one, is executed by a single thread.
Any departure from 1a) inevitably introduces concurrency (non-atomicity) in one form or another.

Ideally we'd like to have very "coarse-grained execution traces" with little concurrency, because they are easy to both write and reason about. Unfortunately, some degree of concurrency (cutting the execution in smaller pieces that can be performed in any order) is necessary for performance reasons. The only question is how much of it we need and how we handle it.
Ultimately, I think that the execution should be

• As coarse-grained as possible (for easy reasoning)
• As fine-grained as necessary (for concurrency and thus performance)

The current architecture starts from the most coarse-grained execution, only handling inherently non-deterministic input as external events: I/O and time. If we left it at just that, it would have 1a) and kill performance. Thus, we fundamentally need to add concurrency. The only deterministic part of the computation where we introduce "artificial" concurrency (splitting the computation) so far is crypto (signing, verifying, hashing). We do this in a very explicit way (by using two distinct events for "before" and "after" the crypto primitive). We do have to think about other events that might happen "in between", adding cognitive overhead and thus potential for bugs, but we would have to do that regardless of the form of concurrency we choose. I argue that the current "explicit" handling of concurrency is actually easy to reason about and debug compared to others (e.g. using locks and language-level synchronization mechanisms).

Regarding "coarse-grained modularity of the event based approach" an "finer-grained levels of modularization within individual event processing modules by other means, e.g. dependency injection", if I understand correctly what you mean by it, I think that the architecture is in favor of exactly that. It gives the programmer a big freedom to implement their module the way they want, using as many abstractions and dependency injections inside it as they want, even starting threads and implementing fine-grained concurrency (with all the consequences it implies). It's on the programmer implementing the module to make the choice. Personally, I chose a very event-centric approach even inside the ISS Protocol module because it is easy for me to reason about it that way, but that is not required by the architecture itself.

Bottom line: The main point of the concern seems to be explicitly splitting the protocol execution not just in places where inherent non-determinism of inputs requires it (I/O, time), but also where it is not fundamentally necessary (crypto). Is that right?
My point is that, implicitly or explicitly, there is no choice but to split the protocol execution for performance reasons and I prefer it to be explicit and event-based, due to (at least for me) less complicated reasoning about it, traceability and ultimately debugging.

[sergefdrv]

@matejpavlovic I thought that one of the advantages of the event-centric architecture was the ability to use event interceptor as a diagnostic tool. However, you seem to advocate here for a very "coarse-grained execution traces", which seems to be rather motivated by desired reproducibility of test scenarios.

[matejpavlovic]

I don't see the contradiction here. I think that using the event interceptor as a diagnostic tool and reproducibility of test scenarios go hand in hand.
Regarding the granularity of the traces produced by the system and captured, replayed, and analyzed by diagnostic tools: what I consider the most "coarse-grained" (and ideal for reasoning about) is when one event in the trace corresponds to one external event (I/O or timer) and that event deterministically defines the next state of the protocol.