K8s node/enclave migration

[bsenthilr]

Trying to build helm charts to deploy single CCF node on K8s.

• When the POD migrates to a new node(enclave) eg., during security patching of the node on nodepool, will "recover" process restore the CCF node on a new patched node(different SGX enclave) using PVC(ledger-dir & snapshots) and recovery shares? This will restore the POD using DR process for single CCF node based setup. Is this understanding right?

• How to determine limits for EPC sizes & CPU for CCF node. Is there a recommended settings to plug into helm charts?

• Are there references available for a sample working helm charts?

Thanks
Senthil

[achamayou]

Hi @bsenthilr

1. The disaster recovery process is only applicable if more than f nodes are shut down/crashed at the same time. Otherwise, you can remove and add nodes to the network via reconfiguration. No DR is required in that case, starting from a snapshot is the most efficient way to bring the node up to date.
2. EPC on Ice Lake can be up to 50% of the physical memory on the machine. The tradeoff is that the larger the value, the slower the boot time for the enclave. The default for js_generic is 2Gb, which results in ~2s of enclave startup time if I remember correctly.
3. We do not provide sample helm charts.

[bsenthilr]

Thanks @achamayou .
For near term we plan to only host single governor node trading off for availability. This is related to discussion 6669. So losing it implies DR since F is 0 in this setup.

When a single governor node gets patched K8s would try to spin off a new SGX node and schedule the POD to this node. Was assuming this would be similar to single node restart and DR procedure will kick in and recovery share would recover the ledger. Currently we have this automated. Is this not a proper procedure?
Basically wondering if there are anything else to account for enclave migration since any sealed data by old enclave would be unsealable unless they are encrypted by recovery keys.

Thanks for info regarding EPC will size SKU accordingly.

[achamayou]

CCF does not currently seal any data with enclave-specific keys. The only thing that is sealed is the ledger secrets, and that's done with a wrapper key shared across members. More detail can be found on this page.

It is possible to automate disaster recovery, but note that on each DR restart, the service has a new service identity. Clients need to be able to cope with that, and should bootstrap trust in the service again.
Clients also need to be aware that the new service may have started from a truncated ledger and rolled back some of their transactions. They can use the receipts they have previously obtained to confirm whether that's the case.

[bsenthilr]

Thanks for clarification that matches my understanding.