Multithreaded key/exit message generation

[valefar-on-discord]

Forwarded from

• [feature request] Multithreaded key generation ethereum/staking-deposit-cli#364

As suggested by Preston, an investigation into building a multithreaded solution for generating keys along with exit messages could drastically improve performance when working with large quantities.

[remyroy]

The multiprocessing module should be the basis for CPU-bound multithreaded in Python: https://docs.python.org/3/library/multiprocessing.html

[remyroy]

Work started in https://github.com/remyroy/ethstaker-deposit-cli/tree/multiprocess

[remyroy]

Some initial results from this improvement. On my modest 8 cores/16 threads CPU, the full process (creation, deposit and verification) of creating 100 keys for mainnet requires 28 seconds with the multiprocess path instead of 4 minutes and 11 seconds without it.

[remyroy]

Another test with 1000 keys took 4 minutes and 29 seconds. It seems to scale linearly.

[valefar-on-discord]

I did a 1000 key generation and got 27.6 minutes with scrypt and 16 minutes with pbkdf2 for the full process.
So pbkdf2 + multithreading should be a beast 😁

[remyroy]

It seems like there is some kind of race issue after adding multiprocess support in our test_exit_transaction_menmonic_multiple test. This is what the CI outputs in a non-deterministic way (sometime it works, sometime it doesn't):

2024-05-09T16:08:06.3531967Z ##[group]Run pytest tests
2024-05-09T16:08:06.3532415Z �[36;1mpytest tests�[0m
2024-05-09T16:08:06.3586034Z shell: /usr/bin/bash -e {0}
2024-05-09T16:08:06.3586444Z env:
2024-05-09T16:08:06.3586924Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3587572Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib/pkgconfig
2024-05-09T16:08:06.3588252Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3588899Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3589489Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3590163Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib
2024-05-09T16:08:06.3590621Z ##[endgroup]
2024-05-09T16:08:08.2956600Z ============================= test session starts ==============================
2024-05-09T16:08:08.2958120Z platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0
2024-05-09T16:08:08.2959833Z rootdir: /home/runner/work/ethstaker-deposit-cli/ethstaker-deposit-cli
2024-05-09T16:08:08.2961158Z plugins: asyncio-0.23.6
2024-05-09T16:08:08.2962352Z asyncio: mode=strict
2024-05-09T16:08:08.2962952Z collected 711 items
2024-05-09T16:08:08.2963351Z 
2024-05-09T16:08:31.6307244Z tests/test_cli/test_existing_mnemonic.py .....                           [  0%]
2024-05-09T16:08:34.7508379Z tests/test_cli/test_exit_transaction_keystore.py ....                    [  1%]
2024-05-09T16:08:36.4061359Z tests/test_cli/test_exit_transaction_mnemonic.py .F.....                 [  2%]
2024-05-09T16:08:39.2593198Z tests/test_cli/test_generate_bls_to_execution_change.py ...              [  2%]
2024-05-09T16:09:03.8446919Z tests/test_cli/test_new_mnemonic.py .......                              [  3%]
2024-05-09T16:09:08.1184764Z tests/test_cli/test_regeneration.py .                                    [  3%]
2024-05-09T16:09:08.1195770Z tests/test_credentials.py .                                              [  3%]
2024-05-09T16:09:16.3293731Z tests/test_deposit.py ........                                           [  5%]
2024-05-09T16:09:16.4031321Z tests/test_intl/test_json_schema.py .................................... [ 10%]
2024-05-09T16:09:16.4752560Z ....................................                                     [ 15%]
2024-05-09T16:09:16.5074869Z tests/test_key_handling/test_key_derivation/test_mnemonic.py ........... [ 16%]
2024-05-09T16:09:16.7078626Z ........................................................................ [ 26%]
2024-05-09T16:09:16.9109373Z ........................................................................ [ 36%]
2024-05-09T16:09:19.6588158Z ........................................................................ [ 47%]
2024-05-09T16:09:23.0039761Z ........................................................................ [ 57%]
2024-05-09T16:09:26.3645323Z ........................................................................ [ 67%]
2024-05-09T16:09:29.7298447Z ........................................................................ [ 77%]
2024-05-09T16:09:32.7004665Z ................................................................         [ 86%]
2024-05-09T16:09:32.7586341Z tests/test_key_handling/test_key_derivation/test_path.py ........        [ 87%]
2024-05-09T16:09:32.7728990Z tests/test_key_handling/test_key_derivation/test_tree.py ............... [ 89%]
2024-05-09T16:09:32.8515167Z ........                                                                 [ 90%]
2024-05-09T16:09:37.3657005Z tests/test_key_handling/test_keystore.py ............                    [ 92%]
2024-05-09T16:09:37.3675366Z tests/test_utils/test_constants.py ..                                    [ 92%]
2024-05-09T16:09:37.9183379Z tests/test_utils/test_crypto.py ..........                               [ 94%]
2024-05-09T16:09:37.9292972Z tests/test_utils/test_intl.py ..........                                 [ 95%]
2024-05-09T16:09:37.9383465Z tests/test_utils/test_ssz.py .........                                   [ 96%]
2024-05-09T16:09:40.6441651Z tests/test_utils/test_validation.py ......................               [100%]
2024-05-09T16:09:40.6443173Z 
2024-05-09T16:09:40.6443987Z =================================== FAILURES ===================================
2024-05-09T16:09:40.6445580Z ___________________ test_exit_transaction_menmonic_multiple ____________________
2024-05-09T16:09:40.6446349Z 
2024-05-09T16:09:40.6449610Z     def test_exit_transaction_menmonic_multiple() -> None:
2024-05-09T16:09:40.6450486Z         # Prepare folder
2024-05-09T16:09:40.6451457Z         my_folder_path = os.path.join(os.getcwd(), 'TESTING_TEMP_FOLDER')
2024-05-09T16:09:40.6452678Z         clean_exit_transaction_folder(my_folder_path)
2024-05-09T16:09:40.6453539Z         if not os.path.exists(my_folder_path):
2024-05-09T16:09:40.6454459Z             os.mkdir(my_folder_path)
2024-05-09T16:09:40.6455167Z     
2024-05-09T16:09:40.6455778Z         runner = CliRunner()
2024-05-09T16:09:40.6456389Z         inputs = []
2024-05-09T16:09:40.6456918Z         data = '\n'.join(inputs)
2024-05-09T16:09:40.6457677Z         arguments = [
2024-05-09T16:09:40.6458358Z             '--language', 'english',
2024-05-09T16:09:40.6459065Z             '--non_interactive',
2024-05-09T16:09:40.6459976Z             'exit-transaction-mnemonic',
2024-05-09T16:09:40.6460858Z             '--output_folder', my_folder_path,
2024-05-09T16:09:40.6461742Z             '--chain', 'mainnet',
2024-05-09T16:09:40.6463056Z             '--mnemonic', 'aban aban aban aban aban aban aban aban aban aban aban abou',
2024-05-09T16:09:40.6464318Z             '--validator_start_index', '0',
2024-05-09T16:09:40.6465315Z             '--validator_indices', '0 1 2 3',
2024-05-09T16:09:40.6466049Z             '--epoch', '1234',
2024-05-09T16:09:40.6466468Z         ]
2024-05-09T16:09:40.6466982Z         result = runner.invoke(cli, arguments, input=data)
2024-05-09T16:09:40.6467458Z     
2024-05-09T16:09:40.6467871Z >       assert result.exit_code == 0
2024-05-09T16:09:40.6468302Z E       AssertionError: assert 1 == 0
2024-05-09T16:09:40.6469046Z E        +  where 1 = <Result FileExistsError(17, 'File exists')>.exit_code
2024-05-09T16:09:40.6469661Z 
2024-05-09T16:09:40.6470047Z tests/test_cli/test_exit_transaction_mnemonic.py:80: AssertionError
2024-05-09T16:09:40.6470698Z =========================== short test summary info ============================
2024-05-09T16:09:40.6471710Z FAILED tests/test_cli/test_exit_transaction_mnemonic.py::test_exit_transaction_menmonic_multiple - AssertionError: assert 1 == 0
2024-05-09T16:09:40.6472741Z  +  where 1 = <Result FileExistsError(17, 'File exists')>.exit_code
2024-05-09T16:09:40.6473382Z =================== 1 failed, 710 passed in 94.01s (0:01:34) ===================
2024-05-09T16:09:40.7114730Z ##[error]Process completed with exit code 1.

This needs further test and improvements.