Network Layer (next gen) for Complete Connectivity - this represents a revolution in network connectivity!

[CMCDragonkai]

This is a high-level issue, a large epic with multiple branches.

Requirements of this design

1. Complete connectivity is to ensure that all Polykey nodes regardless of their network environment can connect to each other if there is a sufficient link (potential multistep) between them.
2. The ability for 2 nodes in symmetric NAT to discover and connect to each other.
3. The ability for all nodes to act as relays - and thus decentralized relaying without rely on the seed nodes.
4. The ability to ensure network segregation, firstly through mainnet and testnet separation, and subsequently private networks in PKE.
5. The ability to simulate the network - create a "chaosnet" where one can deploy autonomous Polykey agents to perform random tasks in a loop, in order to test network stability empirically.
6. The ability to do decentralized signalling efficiently, through each hop, that is user acceptance testing and simulation testing must be done according to Decentralised NAT Signalling #365 that was closed.
7. The ability to attempt network passthrough routers and other equipment via UPnP Network Traversal (by asking the router nicely) - UPNP #536
8. Optimised mdns for local network discovery - to avoid conflict with existing mdns stacks, and integration into existing OS stacks.
9. The ability to signal AND relay over IPv4 and IPv6 and change between the 2.
10. Increased bandwidth with optimised QUIC implementation by parallelising the send and recv calls.
11. Get past corporate firewalls that block UDP, but enabling optional HTTP based relaying.
12. Increased debug transparency and audit transparency - local to the node and global to the dashboard whether public or private.

Additional context

• Derived from the older cancelled issue Decentralised NAT Relaying #182.
• UPnP exploration in Network Traversal (by asking the router nicely) - UPNP #536
• Trusted seed nodes certificates can provide identity and protocols to prove network membership CSR feature for Keys domain to allow KN root certificates to be signed by an external CA #154
• Public mainnet and testnet dashboard design Testnet & Mainnet Status Page testnet.polykey.com mainnet.polykey.com #599
• Initial algorithm for decentralized signalling is in Decentralised NAT Signalling #365, however performance of the algorithm doesn't have empirical tests yet - and I feel that it will require further improvement

Specification

Network Identity and Membership

This is needed to separate mainnet, testnet and private networks, possibly allowing PK to connect to multiple networks simultaneously but forming segregation on every command. It's probably easier to just make PK connect to only a single network.

We also need to figure out the exact role and relationship between the certificate identity chain, and the sigchain claim chain.

Decentralized Signalling and Relaying Algorithm Design

The initial design is in #365. However it has not been applied to relaying yet, nor has it been empirically tested.

Relaying ultimately has to be done through network encapsulation. We have to make use of our RPC to pass through binary blocks from one node to another. This will need to make use of our RAW mode in our RPC for efficiency reasons. We may need to test the performance of this though, because QUIC does actually have a datagram mode, I'm not sure if this is relevant here.

And DOS prevention in #556.

Network Simulation/Visualisation

Involves the usage of Orchestrator and Pulumi.

Chaosnet

It will be important to create autonomous looped Polykey agents to be able to simulate a dirty and chaotic network to see how robust our mechanisms are.

NodeGraph Garbage Collection

The NodeGraph since #537 is capable of multi-host addressing. However GC of the NodeGraph has been neglected, and I believe fast manipulation of this node graph will need to be dealt with.

Discovery Performance and Efficiency

MDNS and related algorithms suffer from some in-efficiencies.

QUIC Performance

MatrixAI/js-quic#54

IPv4 and IPv6 Conformance

#669

Sub-Issues & Sub-PRs created

1. ...
2. ...
3. ...

[linear]

ENG-305 Network Layer for Complete Connectivity

[CMCDragonkai]

I expect to promote this issue to a "Project" as it something that would fit onto a roadmap. As per the discussion on the Project Graph concept - pipeline parallelism would make sense down a particular Project Path, so we can incrementally delivery "Project" rather than spreading the team across multiple "Project" each with their own paths.