Reth has trouble sending Payload Status to lighthouse

[ChosunOne]

Describe the bug

This is a companion issue to the one reported here: sigp/lighthouse#6734

Either Lighthouse or Reth are struggling to communicate payload status with one another fairly regularly. I'm not certain if it is every block, but it happens frequently enough so that I can see it every epoch or so. I'm running everything on the same machine, so connectivity isn't the problem.

Steps to reproduce

Rust:

% rustc --version
rustc 1.83.0 (90b35a623 2024-11-26)

Lighthouse:

% lighthouse --version
Lighthouse v6.0.1-0d90135
BLS library: blst
BLS hardware acceleration: true
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)

Reth:

% reth --version
reth Version: 1.1.4-dev
Commit SHA: 058cfe2f98952633da53f877b538a1ea3f9d657f
Build Timestamp: 2024-12-19T15:21:06.089859378Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

Node logs

pre-database compaction

Dec 20 02:29:02 viserion reth[69389]: 2024-12-19T17:29:02.827294Z  INFO Block added to canonical chain number=21438062 hash=0x91a41a83e762ffe5b19a44082b2f05d3afd6ec6abebff1c7efac83e1263dd62d peers=87 txs=159 gas=13.97 Mgas gas_throughput=9.80 Mgas/second full=46.6% base_fee=21.55gwei blobs=0 excess_blobs=631 elapsed=1.424893523s
Dec 20 02:29:03 viserion reth[69389]: 2024-12-19T17:29:03.011123Z  INFO Canonical chain committed number=21438062 hash=0x91a41a83e762ffe5b19a44082b2f05d3afd6ec6abebff1c7efac83e1263dd62d elapsed=1.128555ms
Dec 20 02:29:14 viserion reth[69389]: 2024-12-19T17:29:14.967405Z  INFO Block added to canonical chain number=21438063 hash=0xf978ecc3ab8118ebf12bd9d1da31093b81970174ee2a944b81afb128d220986d peers=86 txs=201 gas=15.95 Mgas gas_throughput=9.21 Mgas/second full=53.2% base_fee=21.37gwei blobs=6 excess_blobs=628 elapsed=1.730926651s
Dec 20 02:29:15 viserion reth[69389]: 2024-12-19T17:29:15.147110Z  INFO Canonical chain committed number=21438063 hash=0xf978ecc3ab8118ebf12bd9d1da31093b81970174ee2a944b81afb128d220986d elapsed=570.289µs
Dec 20 02:29:30 viserion reth[69389]: 2024-12-19T17:29:30.126923Z  INFO Block added to canonical chain number=21438064 hash=0x7b36feddbffc5a17bfa2ff13d062ce0da6d57c5caf82f12132d20ec7a4aa2591 peers=84 txs=146 gas=15.15 Mgas gas_throughput=3.14 Mgas/second full=50.5% base_fee=21.54gwei blobs=0 excess_blobs=631 elapsed=4.829080736s
Dec 20 02:29:30 viserion reth[69389]: 2024-12-19T17:29:30.321637Z  INFO Canonical chain committed number=21438064 hash=0x7b36feddbffc5a17bfa2ff13d062ce0da6d57c5caf82f12132d20ec7a4aa2591 elapsed=633.349µs
Dec 20 02:29:40 viserion reth[69389]: 2024-12-19T17:29:40.027004Z  INFO Block added to canonical chain number=21438065 hash=0xffa7839205d3442b05a8d48ebee0b4ba2d3bf2a43b91f3f371b6499b87d9857d peers=83 txs=233 gas=20.10 Mgas gas_throughput=9.59 Mgas/second full=67.0% base_fee=21.56gwei blobs=5 excess_blobs=628 elapsed=2.096544851s
Dec 20 02:29:40 viserion reth[69389]: 2024-12-19T17:29:40.246623Z  INFO Canonical chain committed number=21438065 hash=0xffa7839205d3442b05a8d48ebee0b4ba2d3bf2a43b91f3f371b6499b87d9857d elapsed=743.437µs
Dec 20 02:29:52 viserion reth[69389]: 2024-12-19T17:29:52.345813Z  INFO Block added to canonical chain number=21438066 hash=0xc47471c4a66b7e5f540d13a462e041dc6370ade96087bfd9a3043efab41392a0 peers=81 txs=158 gas=12.96 Mgas gas_throughput=6.98 Mgas/second full=43.2% base_fee=22.48gwei blobs=6 excess_blobs=630 elapsed=1.856591584s
Dec 20 02:29:52 viserion reth[69389]: 2024-12-19T17:29:52.556883Z  INFO Canonical chain committed number=21438066 hash=0xc47471c4a66b7e5f540d13a462e041dc6370ade96087bfd9a3043efab41392a0 elapsed=2.108119ms
Dec 20 02:30:02 viserion reth[69389]: 2024-12-19T17:30:02.380929Z  INFO Block added to canonical chain number=21438067 hash=0x60ed62f2b5cc97a7352141f2967d25aa9fab91b444e02353f9bcb965c0d975f2 peers=82 txs=125 gas=10.07 Mgas gas_throughput=8.66 Mgas/second full=33.6% base_fee=22.09gwei blobs=0 excess_blobs=633 elapsed=1.162100465s
Dec 20 02:30:02 viserion reth[69389]: 2024-12-19T17:30:02.583355Z  INFO Canonical chain committed number=21438067 hash=0x60ed62f2b5cc97a7352141f2967d25aa9fab91b444e02353f9bcb965c0d975f2 elapsed=447.547µs
Dec 20 02:30:15 viserion reth[69389]: 2024-12-19T17:30:15.482908Z  INFO Block added to canonical chain number=21438068 hash=0x7ac353a26b609b69c9c2aa832db4d64b96914eecf306318d6b07a6e48b3231d2 peers=83 txs=238 gas=17.60 Mgas gas_throughput=6.84 Mgas/second full=58.7% base_fee=21.19gwei blobs=3 excess_blobs=630 elapsed=2.572723124s
Dec 20 02:30:15 viserion reth[69389]: 2024-12-19T17:30:15.697971Z  INFO Canonical chain committed number=21438068 hash=0x7ac353a26b609b69c9c2aa832db4d64b96914eecf306318d6b07a6e48b3231d2 elapsed=862.393µs
Dec 20 02:30:33 viserion reth[69389]: 2024-12-19T17:30:33.725088Z  INFO Block added to canonical chain number=21438069 hash=0xe59075be1edf60376438c2352e5b68ba19cd78ae5e5636539b77d21dabdcef14 peers=83 txs=187 gas=14.24 Mgas gas_throughput=1.61 Mgas/second full=47.5% base_fee=21.65gwei blobs=2 excess_blobs=630 elapsed=8.86600547s
Dec 20 02:30:33 viserion reth[69389]: 2024-12-19T17:30:33.727088Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xe59075be1edf60376438c2352e5b68ba19cd78ae5e5636539b77d21dabdcef14) })

post-database compaction normal logs

Dec 23 18:06:00 viserion reth[5331]: 2024-12-23T09:06:00.548167Z  INFO Block added to canonical chain number=21464179 hash=0x3821e9d93310cccd37b41424e00d700fbca88abdd588ec6933b15c62e06b7556 peers=130 txs=65 gas=5.23 Mgas g>
Dec 23 18:06:00 viserion reth[5331]: 2024-12-23T09:06:00.752360Z  INFO Canonical chain committed number=21464179 hash=0x3821e9d93310cccd37b41424e00d700fbca88abdd588ec6933b15c62e06b7556 elapsed=331.62µs
Dec 23 18:06:15 viserion reth[5331]: 2024-12-23T09:06:15.054354Z  INFO Block added to canonical chain number=21464180 hash=0x5580d38e5592a021f9dd9ba1e361fcf2f541abef0840197952af62d8090aadc9 peers=130 txs=214 gas=27.26 Mgas>
Dec 23 18:06:15 viserion reth[5331]: 2024-12-23T09:06:15.271282Z  INFO Canonical chain committed number=21464180 hash=0x5580d38e5592a021f9dd9ba1e361fcf2f541abef0840197952af62d8090aadc9 elapsed=1.413995ms
Dec 23 18:06:28 viserion reth[5331]: 2024-12-23T09:06:28.029365Z  INFO Block added to canonical chain number=21464181 hash=0xf5e90203a0ffcebb52b497672b9b9ad2bbe829a5ab4f58c3c33856aacb8f5a4f peers=130 txs=165 gas=13.01 Mgas>
Dec 23 18:06:28 viserion reth[5331]: 2024-12-23T09:06:28.272978Z  INFO Canonical chain committed number=21464181 hash=0xf5e90203a0ffcebb52b497672b9b9ad2bbe829a5ab4f58c3c33856aacb8f5a4f elapsed=857.258µs
Dec 23 18:06:39 viserion reth[5331]: 2024-12-23T09:06:39.242652Z  INFO Block added to canonical chain number=21464182 hash=0xbb0b220d006eaa61fe9409e0d0a8f648ffb8210754b40c28f866d7473c4bec67 peers=130 txs=184 gas=16.05 Mgas>
Dec 23 18:06:39 viserion reth[5331]: 2024-12-23T09:06:39.464026Z  INFO Canonical chain committed number=21464182 hash=0xbb0b220d006eaa61fe9409e0d0a8f648ffb8210754b40c28f866d7473c4bec67 elapsed=968.399µs
Dec 23 18:06:55 viserion reth[5331]: 2024-12-23T09:06:55.803468Z  INFO Block added to canonical chain number=21464183 hash=0xa0329fa8d2b1fd2e0f4aa37abb1edac31e87e2d18272c3806d3f94cadc7fb4f5 peers=129 txs=196 gas=16.26 Mgas>
Dec 23 18:06:55 viserion reth[5331]: 2024-12-23T09:06:55.978680Z  INFO Canonical chain committed number=21464183 hash=0xa0329fa8d2b1fd2e0f4aa37abb1edac31e87e2d18272c3806d3f94cadc7fb4f5 elapsed=1.89629ms
Dec 23 18:07:01 viserion reth[5331]: 2024-12-23T09:07:01.238437Z  INFO Block added to canonical chain number=21464184 hash=0xe53cb2e379f8a3dca798d4dc58b01714e7ca74b3b7f2fa76489b4ad47986b9c4 peers=130 txs=106 gas=7.00 Mgas >
Dec 23 18:07:01 viserion reth[5331]: 2024-12-23T09:07:01.454022Z  INFO Canonical chain committed number=21464184 hash=0xe53cb2e379f8a3dca798d4dc58b01714e7ca74b3b7f2fa76489b4ad47986b9c4 elapsed=426.289µs
Dec 23 18:07:14 viserion reth[5331]: 2024-12-23T09:07:14.520544Z  INFO Block added to canonical chain number=21464185 hash=0x2fffd0eebb61c13132c19cb6f402da32eadb48cd59c588579cf397392ac02734 peers=130 txs=165 gas=14.76 Mgas>
Dec 23 18:07:14 viserion reth[5331]: 2024-12-23T09:07:14.723313Z  INFO Canonical chain committed number=21464185 hash=0x2fffd0eebb61c13132c19cb6f402da32eadb48cd59c588579cf397392ac02734 elapsed=626.059µs
Dec 23 18:07:30 viserion reth[5331]: 2024-12-23T09:07:30.616756Z  INFO Block added to canonical chain number=21464186 hash=0xcb2d68df44ab08386e58e75cd03c51d1eb08fad59123c33d1154250f4f00d0ea peers=130 txs=230 gas=24.45 Mgas>
Dec 23 18:07:30 viserion reth[5331]: 2024-12-23T09:07:30.850304Z  INFO Canonical chain committed number=21464186 hash=0xcb2d68df44ab08386e58e75cd03c51d1eb08fad59123c33d1154250f4f00d0ea elapsed=1.240105ms
Dec 23 18:07:38 viserion reth[5331]: 2024-12-23T09:07:38.151014Z  INFO Block added to canonical chain number=21464187 hash=0x27c792e312c93a24ca43c5dcb9f083a985256a6f2c24dde3832dd47f0c9f5e85 peers=130 txs=119 gas=9.78 Mgas >
Dec 23 18:07:42 viserion reth[5331]: 2024-12-23T09:07:42.218679Z  INFO Canonical chain committed number=21464187 hash=0x27c792e312c93a24ca43c5dcb9f083a985256a6f2c24dde3832dd47f0c9f5e85 elapsed=1.329053ms
Dec 23 18:07:51 viserion reth[5331]: 2024-12-23T09:07:51.322436Z  INFO Block added to canonical chain number=21464188 hash=0xd7ce0a9acbb37f98f833480156d3603d996c79c958b6589d43c0daf6bdd25b39 peers=130 txs=206 gas=22.58 Mgas>
Dec 23 18:07:51 viserion reth[5331]: 2024-12-23T09:07:51.531872Z  INFO Canonical chain committed number=21464188 hash=0xd7ce0a9acbb37f98f833480156d3603d996c79c958b6589d43c0daf6bdd25b39 elapsed=1.867255ms
Dec 23 18:08:03 viserion reth[5331]: 2024-12-23T09:08:03.820018Z  INFO Block added to canonical chain number=21464189 hash=0xaf5dfd476ca9ed8e10ce580918e14db43c4a576fba7c5a639d1c662831b4a493 peers=130 txs=132 gas=10.34 Mgas>
Dec 23 18:08:04 viserion reth[5331]: 2024-12-23T09:08:04.038360Z  INFO Canonical chain committed number=21464189 hash=0xaf5dfd476ca9ed8e10ce580918e14db43c4a576fba7c5a639d1c662831b4a493 elapsed=470.994µs
Dec 23 18:08:16 viserion reth[5331]: 2024-12-23T09:08:16.432439Z  INFO Block added to canonical chain number=21464190 hash=0xb088c7c8bfef8f0433dc859ba160da697d6ba25fb3efd6f9a0eec7d15ac427d7 peers=130 txs=210 gas=19.47 Mgas>
Dec 23 18:08:16 viserion reth[5331]: 2024-12-23T09:08:16.639875Z  INFO Canonical chain committed number=21464190 hash=0xb088c7c8bfef8f0433dc859ba160da697d6ba25fb3efd6f9a0eec7d15ac427d7 elapsed=1.404838ms
Dec 23 18:08:29 viserion reth[5331]: 2024-12-23T09:08:29.532769Z  INFO Block added to canonical chain number=21464191 hash=0x82277652d4662cb4f36bdb1db097221c44cf4c278addd5b31fbc683c29c2465c peers=130 txs=121 gas=13.62 Mgas>
Dec 23 18:08:29 viserion reth[5331]: 2024-12-23T09:08:29.737336Z  INFO Canonical chain committed number=21464191 hash=0x82277652d4662cb4f36bdb1db097221c44cf4c278addd5b31fbc683c29c2465c elapsed=1.555403ms
Dec 23 18:08:38 viserion reth[5331]: 2024-12-23T09:08:38.633320Z  INFO Block added to canonical chain number=21464192 hash=0x1d6170e3a76dcfe04bff4e5c5224761940a5fcd7bacc729ad3ff0eb7b5eeff07 peers=130 txs=182 gas=19.05 Mgas>
Dec 23 18:08:38 viserion reth[5331]: 2024-12-23T09:08:38.820776Z  INFO Canonical chain committed number=21464192 hash=0x1d6170e3a76dcfe04bff4e5c5224761940a5fcd7bacc729ad3ff0eb7b5eeff07 elapsed=3.268716ms
Dec 23 18:08:50 viserion reth[5331]: 2024-12-23T09:08:50.548433Z  INFO Block added to canonical chain number=21464193 hash=0x64061ab25054b48d6d70ac141a8d9ede3a62aed4fa30cd916a2d9e16e152dd81 peers=130 txs=99 gas=7.37 Mgas g>
Dec 23 18:08:50 viserion reth[5331]: 2024-12-23T09:08:50.751690Z  INFO Canonical chain committed number=21464193 hash=0x64061ab25054b48d6d70ac141a8d9ede3a62aed4fa30cd916a2d9e16e152dd81 elapsed=670.373µs
Dec 23 18:09:01 viserion reth[5331]: 2024-12-23T09:09:01.048209Z  INFO Block added to canonical chain number=21464194 hash=0xa4bc88c0c2b8c0894e71e6f366abce5108bb5c15c6201d8b3e794adcfda86272 peers=130 txs=80 gas=7.67 Mgas g>
Dec 23 18:09:01 viserion reth[5331]: 2024-12-23T09:09:01.248355Z  INFO Canonical chain committed number=21464194 hash=0xa4bc88c0c2b8c0894e71e6f366abce5108bb5c15c6201d8b3e794adcfda86272 elapsed=1.255814ms
Dec 23 18:09:14 viserion reth[5331]: 2024-12-23T09:09:14.528419Z  INFO Block added to canonical chain number=21464195 hash=0xbcdf940b6d8e60ad2c1e19ced22adef45559d14ffc6d70182e7592b0bc7f4c1d peers=130 txs=200 gas=15.71 Mgas>
Dec 23 18:09:14 viserion reth[5331]: 2024-12-23T09:09:14.728069Z  INFO Canonical chain committed number=21464195 hash=0xbcdf940b6d8e60ad2c1e19ced22adef45559d14ffc6d70182e7592b0bc7f4c1d elapsed=674.4µs
Dec 23 18:09:25 viserion reth[5331]: 2024-12-23T09:09:25.915533Z  INFO Block added to canonical chain number=21464196 hash=0x29db77c44961cb8aa2a876f3b96aa38cd80bdb3dd36b5bc4d8ed7ae74d56f614 peers=130 txs=168 gas=13.83 Mgas>
Dec 23 18:09:26 viserion reth[5331]: 2024-12-23T09:09:26.098748Z  INFO Canonical chain committed number=21464196 hash=0x29db77c44961cb8aa2a876f3b96aa38cd80bdb3dd36b5bc4d8ed7ae74d56f614 elapsed=1.01641ms
Dec 23 18:09:38 viserion reth[5331]: 2024-12-23T09:09:38.337866Z  INFO Block added to canonical chain number=21464197 hash=0x71d585868f2dc682c45c8e5be96db175163c23fe98674718a296db2f1791c8b0 peers=130 txs=198 gas=15.24 Mgas>
Dec 23 18:09:38 viserion reth[5331]: 2024-12-23T09:09:38.544023Z  INFO Canonical chain committed number=21464197 hash=0x71d585868f2dc682c45c8e5be96db175163c23fe98674718a296db2f1791c8b0 elapsed=935.867µs
Dec 23 18:09:49 viserion reth[5331]: 2024-12-23T09:09:49.974867Z  INFO Block added to canonical chain number=21464198 hash=0x7b93a72c32829a57db8af6a30ce84b14087dd5ea21998a06101cde3e9b3c8c1d peers=130 txs=150 gas=11.36 Mgas>
Dec 23 18:09:50 viserion reth[5331]: 2024-12-23T09:09:50.202259Z  INFO Canonical chain committed number=21464198 hash=0x7b93a72c32829a57db8af6a30ce84b14087dd5ea21998a06101cde3e9b3c8c1d elapsed=5.333146ms
Dec 23 18:10:02 viserion reth[5331]: 2024-12-23T09:10:02.606044Z  INFO Block added to canonical chain number=21464199 hash=0x1d947a43c8f45e8766c51bd375bbfd6af91e9b74070c278085ddc888f3481543 peers=130 txs=137 gas=10.47 Mgas>
Dec 23 18:10:02 viserion reth[5331]: 2024-12-23T09:10:02.798481Z  INFO Canonical chain committed number=21464199 hash=0x1d947a43c8f45e8766c51bd375bbfd6af91e9b74070c278085ddc888f3481543 elapsed=2.93354ms
Dec 23 18:10:18 viserion reth[5331]: 2024-12-23T09:10:18.922238Z  INFO Block added to canonical chain number=21464200 hash=0xb33448280c79ad54a746cae53f23e18a6ba93281eab666a536cd98cbe8f309a0 peers=130 txs=295 gas=24.17 Mgas>
Dec 23 18:10:19 viserion reth[5331]: 2024-12-23T09:10:19.102114Z  INFO Canonical chain committed number=21464200 hash=0xb33448280c79ad54a746cae53f23e18a6ba93281eab666a536cd98cbe8f309a0 elapsed=1.733251ms
Dec 23 18:10:28 viserion reth[5331]: 2024-12-23T09:10:28.826877Z  INFO Block added to canonical chain number=21464201 hash=0xfb03d3d8195f44bf725e9b392701bba6a7e8e0440d73106e36469b44fa16c44f peers=130 txs=376 gas=23.41 Mgas>
Dec 23 18:10:29 viserion reth[5331]: 2024-12-23T09:10:29.052035Z  INFO Canonical chain committed number=21464201 hash=0xfb03d3d8195f44bf725e9b392701bba6a7e8e0440d73106e36469b44fa16c44f elapsed=1.488756ms
Dec 23 18:10:38 viserion reth[5331]: 2024-12-23T09:10:38.236809Z  INFO Block added to canonical chain number=21464202 hash=0x10ef638469d614763a82194884844e255adfb79aa0020065fe3760bb8363e3e8 peers=130 txs=150 gas=13.67 Mgas>
Dec 23 18:10:38 viserion reth[5331]: 2024-12-23T09:10:38.453148Z  INFO Canonical chain committed number=21464202 hash=0x10ef638469d614763a82194884844e255adfb79aa0020065fe3760bb8363e3e8 elapsed=878.278µs
Dec 23 18:10:54 viserion reth[5331]: 2024-12-23T09:10:54.260015Z  INFO Block added to canonical chain number=21464203 hash=0x0a2f65abb6daa602775f18599665292b2e9ac2ac0df6251d1668ce067dfecf46 peers=130 txs=163 gas=12.56 Mgas>
Dec 23 18:10:54 viserion reth[5331]: 2024-12-23T09:10:54.458768Z  INFO Canonical chain committed number=21464203 hash=0x0a2f65abb6daa602775f18599665292b2e9ac2ac0df6251d1668ce067dfecf46 elapsed=595.28µs
Dec 23 18:11:05 viserion reth[5331]: 2024-12-23T09:11:05.761512Z  INFO Block added to canonical chain number=21464204 hash=0xd8d41f5356cb52f8ae0db7fc9ea686ef2960bb31b7ab4d84c8c4050a11aaecfc peers=130 txs=135 gas=13.10 Mgas>
Dec 23 18:11:05 viserion reth[5331]: 2024-12-23T09:11:05.979811Z  INFO Canonical chain committed number=21464204 hash=0xd8d41f5356cb52f8ae0db7fc9ea686ef2960bb31b7ab4d84c8c4050a11aaecfc elapsed=1.242609ms
Dec 23 18:11:14 viserion reth[5331]: 2024-12-23T09:11:14.341186Z  INFO Block added to canonical chain number=21464205 hash=0xbafa24523b33c7998f8574293174f74803ed66ffe22d0af645cf5c62b0d6a8e7 peers=130 txs=198 gas=14.00 Mgas>
Dec 23 18:11:14 viserion reth[5331]: 2024-12-23T09:11:14.553842Z  INFO Canonical chain committed number=21464205 hash=0xbafa24523b33c7998f8574293174f74803ed66ffe22d0af645cf5c62b0d6a8e7 elapsed=1.345745ms
Dec 23 18:11:25 viserion reth[5331]: 2024-12-23T09:11:25.625331Z  INFO Block added to canonical chain number=21464206 hash=0x3994c66cec8e73997811dc143cfba2e934302cec35d20ab9a6d8ef5438885cae peers=130 txs=122 gas=6.93 Mgas >
Dec 23 18:11:25 viserion reth[5331]: 2024-12-23T09:11:25.816548Z  INFO Canonical chain committed number=21464206 hash=0x3994c66cec8e73997811dc143cfba2e934302cec35d20ab9a6d8ef5438885cae elapsed=336.359µs
Dec 23 18:11:41 viserion reth[5331]: 2024-12-23T09:11:41.649113Z  INFO Block added to canonical chain number=21464207 hash=0x56151cecc689a48aa255094f4b281b80ac0c58fb30cbd7a4f8f4a79ae1796147 peers=130 txs=290 gas=25.94 Mgas>
Dec 23 18:11:41 viserion reth[5331]: 2024-12-23T09:11:41.855999Z  INFO Canonical chain committed number=21464207 hash=0x56151cecc689a48aa255094f4b281b80ac0c58fb30cbd7a4f8f4a79ae1796147 elapsed=3.04463ms
Dec 23 18:11:49 viserion reth[5331]: 2024-12-23T09:11:49.932625Z  INFO Block added to canonical chain number=21464208 hash=0x1f3ede27795e5e0512129ed7ecc6254558620a18d9a5945f79ee2b97864f543d peers=130 txs=129 gas=11.37 Mgas>
Dec 23 18:11:50 viserion reth[5331]: 2024-12-23T09:11:50.111918Z  INFO Canonical chain committed number=21464208 hash=0x1f3ede27795e5e0512129ed7ecc6254558620a18d9a5945f79ee2b97864f543d elapsed=401.582µs
Dec 23 18:12:03 viserion reth[5331]: 2024-12-23T09:12:03.259128Z  INFO Block added to canonical chain number=21464209 hash=0xcb1be662622a23afe7901c70260fba2b4b725e2ad040df8285b9fe8b38035c87 peers=130 txs=231 gas=20.64 Mgas>
Dec 23 18:12:03 viserion reth[5331]: 2024-12-23T09:12:03.464627Z  INFO Canonical chain committed number=21464209 hash=0xcb1be662622a23afe7901c70260fba2b4b725e2ad040df8285b9fe8b38035c87 elapsed=2.221889ms
Dec 23 18:12:18 viserion reth[5331]: 2024-12-23T09:12:18.387366Z  INFO Block added to canonical chain number=21464210 hash=0x5677d05cb80cdb128dc46919e5bbf1a045fe9894be4c8a582905a7d40aafa981 peers=130 txs=168 gas=14.70 Mgas>
Dec 23 18:12:18 viserion reth[5331]: 2024-12-23T09:12:18.560796Z  INFO Canonical chain committed number=21464210 hash=0x5677d05cb80cdb128dc46919e5bbf1a045fe9894be4c8a582905a7d40aafa981 elapsed=982.505µs
Dec 23 18:12:26 viserion reth[5331]: 2024-12-23T09:12:26.370593Z  INFO Block added to canonical chain number=21464211 hash=0x6448d716a4d243572bc0c3c7fe07ce43b44579d8b4ed7621b1da36b6ca6bdba1 peers=130 txs=161 gas=14.10 Mgas>
Dec 23 18:12:26 viserion reth[5331]: 2024-12-23T09:12:26.559905Z  INFO Canonical chain committed number=21464211 hash=0x6448d716a4d243572bc0c3c7fe07ce43b44579d8b4ed7621b1da36b6ca6bdba1 elapsed=590.491µs
Dec 23 18:12:39 viserion reth[5331]: 2024-12-23T09:12:39.188177Z  INFO Block added to canonical chain number=21464212 hash=0x9f907a83657574e09d1ddc6bd8b5fba6f0c8719653c87576251d00ebad924a3f peers=130 txs=157 gas=10.79 Mgas>
Dec 23 18:12:39 viserion reth[5331]: 2024-12-23T09:12:39.388666Z  INFO Canonical chain committed number=21464212 hash=0x9f907a83657574e09d1ddc6bd8b5fba6f0c8719653c87576251d00ebad924a3f elapsed=446.548µs
Dec 23 18:12:54 viserion reth[5331]: 2024-12-23T09:12:54.276830Z  INFO Block added to canonical chain number=21464213 hash=0x79ba95b17b2900233f95b09352ed0e360e9c2286c0e38ddbeb5baf31cdbd0ecb peers=130 txs=132 gas=19.58 Mgas>
Dec 23 18:12:54 viserion reth[5331]: 2024-12-23T09:12:54.492513Z  INFO Canonical chain committed number=21464213 hash=0x79ba95b17b2900233f95b09352ed0e360e9c2286c0e38ddbeb5baf31cdbd0ecb elapsed=620.558µs
Dec 23 18:13:04 viserion reth[5331]: 2024-12-23T09:13:04.155804Z  INFO Block added to canonical chain number=21464214 hash=0xe2cf58adc09450f76105cfe9cc2f71d5a70555b7d5918104ae8378bded045134 peers=130 txs=194 gas=14.45 Mgas>
Dec 23 18:13:04 viserion reth[5331]: 2024-12-23T09:13:04.362147Z  INFO Canonical chain committed number=21464214 hash=0xe2cf58adc09450f76105cfe9cc2f71d5a70555b7d5918104ae8378bded045134 elapsed=769.911µs
Dec 23 18:13:14 viserion reth[5331]: 2024-12-23T09:13:14.138041Z  INFO Block added to canonical chain number=21464215 hash=0xe62ad3cc37f06bdf78d6a3e0c28989166538ccb41cb218a8d0a4051758f60877 peers=130 txs=170 gas=13.87 Mgas>
Dec 23 18:13:14 viserion reth[5331]: 2024-12-23T09:13:14.338549Z  INFO Canonical chain committed number=21464215 hash=0xe62ad3cc37f06bdf78d6a3e0c28989166538ccb41cb218a8d0a4051758f60877 elapsed=584.761µs
Dec 23 18:13:29 viserion reth[5331]: 2024-12-23T09:13:29.181877Z  INFO Block added to canonical chain number=21464216 hash=0xce1294ef4e2d942bcedc9b7c6d086b6c07d039966382b34f847577e80b996d83 peers=130 txs=163 gas=12.43 Mgas>
Dec 23 18:13:29 viserion reth[5331]: 2024-12-23T09:13:29.402128Z  INFO Canonical chain committed number=21464216 hash=0xce1294ef4e2d942bcedc9b7c6d086b6c07d039966382b34f847577e80b996d83 elapsed=1.331999ms
Dec 23 18:13:38 viserion reth[5331]: 2024-12-23T09:13:38.607395Z  INFO Block added to canonical chain number=21464217 hash=0xac1b4edc83a573470467d79251c9ac4ee1d237b1ada67dc1f34b38045807378c peers=130 txs=174 gas=14.24 Mgas>
Dec 23 18:13:38 viserion reth[5331]: 2024-12-23T09:13:38.796731Z  INFO Canonical chain committed number=21464217 hash=0xac1b4edc83a573470467d79251c9ac4ee1d237b1ada67dc1f34b38045807378c elapsed=501.712µs
Dec 23 18:13:51 viserion reth[5331]: 2024-12-23T09:13:51.287914Z  INFO Block added to canonical chain number=21464218 hash=0x403a261747bf29be6da3830cfd051119767773ea389ee135be9e185c59ba66b8 peers=130 txs=175 gas=13.74 Mgas>
Dec 23 18:13:51 viserion reth[5331]: 2024-12-23T09:13:51.496155Z  INFO Canonical chain committed number=21464218 hash=0x403a261747bf29be6da3830cfd051119767773ea389ee135be9e185c59ba66b8 elapsed=564.251µs
Dec 23 18:14:04 viserion reth[5331]: 2024-12-23T09:14:04.550140Z  INFO Block added to canonical chain number=21464219 hash=0x578f3fbd866b72d816adc3bca4ef5680e1c86ec1d710fe94b6d7f3b5c8a0376e peers=131 txs=106 gas=8.44 Mgas >
Dec 23 18:14:08 viserion reth[5331]: 2024-12-23T09:14:08.975215Z  INFO Canonical chain committed number=21464219 hash=0x578f3fbd866b72d816adc3bca4ef5680e1c86ec1d710fe94b6d7f3b5c8a0376e elapsed=1.007453ms
Dec 23 18:14:16 viserion reth[5331]: 2024-12-23T09:14:16.625056Z  INFO Block added to canonical chain number=21464220 hash=0x4fe9e541947f0b2f217fbf1ec64bf10db17e0b94f160e97cbc43e7c22cd85ef9 peers=130 txs=195 gas=15.20 Mgas>
Dec 23 18:14:16 viserion reth[5331]: 2024-12-23T09:14:16.817940Z  INFO Canonical chain committed number=21464220 hash=0x4fe9e541947f0b2f217fbf1ec64bf10db17e0b94f160e97cbc43e7c22cd85ef9 elapsed=1.247458ms
Dec 23 18:14:24 viserion reth[5331]: 2024-12-23T09:14:24.935939Z  INFO Block added to canonical chain number=21464221 hash=0xfcd6ee33f10dfe60b73498e8c08d034d9bd3379dd813bd1e6fc303c06a92461c peers=130 txs=95 gas=5.74 Mgas g>
Dec 23 18:14:25 viserion reth[5331]: 2024-12-23T09:14:25.132213Z  INFO Canonical chain committed number=21464221 hash=0xfcd6ee33f10dfe60b73498e8c08d034d9bd3379dd813bd1e6fc303c06a92461c elapsed=286.514µs
Dec 23 18:14:40 viserion reth[5331]: 2024-12-23T09:14:40.726030Z  INFO Block added to canonical chain number=21464222 hash=0x2fe77fde0eb6c46b8a0eeadf4eb8bf0dada5f80a3e110258fc41a275ea443556 peers=130 txs=273 gas=20.07 Mgas>
Dec 23 18:14:40 viserion reth[5331]: 2024-12-23T09:14:40.929037Z  INFO Canonical chain committed number=21464222 hash=0x2fe77fde0eb6c46b8a0eeadf4eb8bf0dada5f80a3e110258fc41a275ea443556 elapsed=1.178578ms
Dec 23 18:14:51 viserion reth[5331]: 2024-12-23T09:14:51.378812Z  INFO Block added to canonical chain number=21464223 hash=0xf90fee70f5d84cc4d9944a8115d373e4cdaaa330ba1bc83c7817f5595c959ee8 peers=130 txs=178 gas=13.20 Mgas>
Dec 23 18:14:51 viserion reth[5331]: 2024-12-23T09:14:51.580424Z  INFO Canonical chain committed number=21464223 hash=0xf90fee70f5d84cc4d9944a8115d373e4cdaaa330ba1bc83c7817f5595c959ee8 elapsed=877.756µs
Dec 23 18:15:04 viserion reth[5331]: 2024-12-23T09:15:04.165206Z  INFO Block added to canonical chain number=21464224 hash=0x14d952ae8a0e9e091c4cbbe9624af64b11343aa353f766a9ab017d2d8f497f4e peers=130 txs=137 gas=12.40 Mgas>
Dec 23 18:15:04 viserion reth[5331]: 2024-12-23T09:15:04.364271Z  INFO Canonical chain committed number=21464224 hash=0x14d952ae8a0e9e091c4cbbe9624af64b11343aa353f766a9ab017d2d8f497f4e elapsed=2.073717ms
Dec 23 18:15:20 viserion reth[5331]: 2024-12-23T09:15:20.536309Z  INFO Block added to canonical chain number=21464225 hash=0xdf370467d22da0e53c4c76f3c208e2f63683186194f74d8942009244352ee679 peers=130 txs=283 gas=18.20 Mgas>
Dec 23 17:59:18 viserion reth[5331]: 2024-12-23T08:59:18.376988Z  INFO Canonical chain committed number=21464145 hash=0xd7725a4b4c54506bda60e795957221a3e90618c2ee56216348cbe02c5a87fd05 elapsed=939.294µs
Dec 23 17:59:26 viserion reth[5331]: 2024-12-23T08:59:26.029059Z  INFO Block added to canonical chain number=21464146 hash=0xf1fafa3c78edeccbc6286447ef1f978f079f369838376c2696ce38e10bc277a2 peers=130 txs=152 gas=10.46 Mgas>
Dec 23 17:59:26 viserion reth[5331]: 2024-12-23T08:59:26.229690Z  INFO Canonical chain committed number=21464146 hash=0xf1fafa3c78edeccbc6286447ef1f978f079f369838376c2696ce38e10bc277a2 elapsed=1.181603ms
Dec 23 17:59:38 viserion reth[5331]: 2024-12-23T08:59:38.653846Z  INFO Block added to canonical chain number=21464147 hash=0x1e6ddcfa99a5c8bebcad4533451e5fbf93a1183bb12940fc60e607792513f638 peers=130 txs=178 gas=14.35 Mgas>
Dec 23 17:59:38 viserion reth[5331]: 2024-12-23T08:59:38.847007Z  INFO Canonical chain committed number=21464147 hash=0x1e6ddcfa99a5c8bebcad4533451e5fbf93a1183bb12940fc60e607792513f638 elapsed=636.569µs
Dec 23 17:59:51 viserion reth[5331]: 2024-12-23T08:59:51.763547Z  INFO Block added to canonical chain number=21464148 hash=0xad65201dcf3b4ed511977e240201851abe6c4dd9fa93c430033a718dd4e59025 peers=130 txs=158 gas=16.22 Mgas>
Dec 23 17:59:51 viserion reth[5331]: 2024-12-23T08:59:51.968599Z  INFO Canonical chain committed number=21464148 hash=0xad65201dcf3b4ed511977e240201851abe6c4dd9fa93c430033a718dd4e59025 elapsed=2.051434ms
Dec 23 18:00:02 viserion reth[5331]: 2024-12-23T09:00:02.443181Z  INFO Block added to canonical chain number=21464149 hash=0xe9774ecf17110447b8e8d18b5ed8ad7da73138580b3d8ef04b0927f0c2a63b5c peers=130 txs=138 gas=23.69 Mgas>
Dec 23 18:00:02 viserion reth[5331]: 2024-12-23T09:00:02.658560Z  INFO Canonical chain committed number=21464149 hash=0xe9774ecf17110447b8e8d18b5ed8ad7da73138580b3d8ef04b0927f0c2a63b5c elapsed=526.629µs
Dec 23 18:00:15 viserion reth[5331]: 2024-12-23T09:00:15.190748Z  INFO Block added to canonical chain number=21464150 hash=0x5c441cdae76c6b01b42fef0a7dd13eacfab43f8d1f32b576d5593e53ad6a822e peers=130 txs=287 gas=14.95 Mgas>
Dec 23 18:00:15 viserion reth[5331]: 2024-12-23T09:00:15.384945Z  INFO Canonical chain committed number=21464150 hash=0x5c441cdae76c6b01b42fef0a7dd13eacfab43f8d1f32b576d5593e53ad6a822e elapsed=2.542386ms
Dec 23 18:00:27 viserion reth[5331]: 2024-12-23T09:00:27.166298Z  INFO Block added to canonical chain number=21464151 hash=0x826098094fcc91e429a9844d297a39f4ca1f827b8391fed3c1a79ce84b8e0e6a peers=130 txs=325 gas=21.54 Mgas>
Dec 23 18:00:27 viserion reth[5331]: 2024-12-23T09:00:27.429695Z  INFO Canonical chain committed number=21464151 hash=0x826098094fcc91e429a9844d297a39f4ca1f827b8391fed3c1a79ce84b8e0e6a elapsed=3.089636ms
Dec 23 18:00:38 viserion reth[5331]: 2024-12-23T09:00:38.651643Z  INFO Block added to canonical chain number=21464152 hash=0xcdd70d5aa5f6f01ea7c3f9defd372bc084fcc41ce1981eb5433a1a2fb2ef106e peers=130 txs=182 gas=15.05 Mgas>
Dec 23 18:00:38 viserion reth[5331]: 2024-12-23T09:00:38.861515Z  INFO Canonical chain committed number=21464152 hash=0xcdd70d5aa5f6f01ea7c3f9defd372bc084fcc41ce1981eb5433a1a2fb2ef106e elapsed=1.872034ms
Dec 23 18:00:49 viserion reth[5331]: 2024-12-23T09:00:49.918163Z  INFO Block added to canonical chain number=21464153 hash=0xcbed4e591f5c965352dddfd3fe190a612a14e7fa8de29fb05941781861756a5c peers=129 txs=125 gas=8.64 Mgas >
Dec 23 18:00:50 viserion reth[5331]: 2024-12-23T09:00:50.125970Z  INFO Canonical chain committed number=21464153 hash=0xcbed4e591f5c965352dddfd3fe190a612a14e7fa8de29fb05941781861756a5c elapsed=673.449µs
Dec 23 18:01:04 viserion reth[5331]: 2024-12-23T09:01:04.324579Z  INFO Block added to canonical chain number=21464154 hash=0x6e11ec33cb8ad25e1f9ed70b44a2d4290fd56b744e6bf9629ff97c380831128a peers=131 txs=308 gas=24.43 Mgas>
Dec 23 18:01:04 viserion reth[5331]: 2024-12-23T09:01:04.522779Z  INFO Canonical chain committed number=21464154 hash=0x6e11ec33cb8ad25e1f9ed70b44a2d4290fd56b744e6bf9629ff97c380831128a elapsed=1.131779ms
Dec 23 18:01:14 viserion reth[5331]: 2024-12-23T09:01:14.482953Z  INFO Block added to canonical chain number=21464155 hash=0xe294f145bee862eefc553613d22e17257049856a220d20406930d2e83becfe12 peers=130 txs=165 gas=17.79 Mgas>
Dec 23 18:01:18 viserion reth[5331]: 2024-12-23T09:01:18.521122Z  INFO Canonical chain committed number=21464155 hash=0xe294f145bee862eefc553613d22e17257049856a220d20406930d2e83becfe12 elapsed=819.355µs
Dec 23 18:01:28 viserion reth[5331]: 2024-12-23T09:01:28.028196Z  INFO Block added to canonical chain number=21464156 hash=0x2d49ec66432de4e1b2dccdc277d4b3fe90a43197a5873ad8b5fd2c8587907db0 peers=130 txs=182 gas=16.71 Mgas>
Dec 23 18:01:28 viserion reth[5331]: 2024-12-23T09:01:28.238953Z  INFO Canonical chain committed number=21464156 hash=0x2d49ec66432de4e1b2dccdc277d4b3fe90a43197a5873ad8b5fd2c8587907db0 elapsed=579.741µs
Dec 23 18:01:45 viserion reth[5331]: 2024-12-23T09:01:45.962684Z  INFO Block added to canonical chain number=21464157 hash=0x027f3c0cce5838d4a1964b57aa2c0daef176db1d216455be26ab7459b0b78fe3 peers=130 txs=134 gas=11.68 Mgas>
Dec 23 18:01:47 viserion reth[5331]: 2024-12-23T09:01:47.148647Z  INFO Canonical chain committed number=21464157 hash=0x027f3c0cce5838d4a1964b57aa2c0daef176db1d216455be26ab7459b0b78fe3 elapsed=800.46µs
Dec 23 18:01:50 viserion reth[5331]: 2024-12-23T09:01:50.478594Z  INFO Block added to canonical chain number=21464158 hash=0xf6a9cabfb06fe629e28f3abe58c0ae728851cb81d0d180d0537debc88c516e50 peers=130 txs=236 gas=15.54 Mgas>
Dec 23 18:01:50 viserion reth[5331]: 2024-12-23T09:01:50.666706Z  INFO Canonical chain committed number=21464158 hash=0xf6a9cabfb06fe629e28f3abe58c0ae728851cb81d0d180d0537debc88c516e50 elapsed=2.063307ms
Dec 23 18:02:02 viserion reth[5331]: 2024-12-23T09:02:02.299519Z  INFO Block added to canonical chain number=21464159 hash=0x4ff597b1f1d31f1e2852abbb642e63931a7067df900c556ac586e63244d6e564 peers=130 txs=202 gas=14.17 Mgas>
Dec 23 18:02:02 viserion reth[5331]: 2024-12-23T09:02:02.517695Z  INFO Canonical chain committed number=21464159 hash=0x4ff597b1f1d31f1e2852abbb642e63931a7067df900c556ac586e63244d6e564 elapsed=1.076023ms

post-database compaction error logs

Dec 23 12:57:11 viserion reth[5331]: 2024-12-23T03:57:11.836050Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x7c931078005ee63fe658126e10e70a72cce9113036f9e79d5c6>
Dec 23 12:57:34 viserion reth[5331]: 2024-12-23T03:57:34.714858Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x92f162efd9e5edfdfa144855c2339dacddd999a5d1399cb17c5>
Dec 23 12:57:57 viserion reth[5331]: 2024-12-23T03:57:57.858123Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x9b66a85f09ee78a6bf3d10e3797b43c2ead5a704d684f583f5a>
Dec 23 14:11:58 viserion reth[5331]: 2024-12-23T05:11:58.432739Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xa45158d1ceef54581c05d5b0e435e29f7dfb83fc33dc00f442f>
Dec 23 14:30:30 viserion reth[5331]: 2024-12-23T05:30:30.494810Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x35834241988a8efc84fe761511cbecff2fefd3888b4e9f14d33>
Dec 23 14:30:30 viserion reth[5331]: 2024-12-23T05:30:30.523652Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x35834241988a8efc84fe761511cbecff2fefd3888b4e9f14d33>
Dec 23 14:43:26 viserion reth[5331]: 2024-12-23T05:43:26.477800Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x9286f63bdaf95adbc2a4f9fe63440dd386ac03a651fa72c26dc>
Dec 23 14:50:00 viserion reth[5331]: 2024-12-23T05:50:00.076807Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4e1fb4a34d16c76582d405190a9ee73f5afb0d642ad2674ac27>
Dec 23 14:50:00 viserion reth[5331]: 2024-12-23T05:50:00.084805Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4e1fb4a34d16c76582d405190a9ee73f5afb0d642ad2674ac27>
Dec 23 15:02:21 viserion reth[5331]: 2024-12-23T06:02:21.843992Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x118d0ff1ca208c03204613beceb1c959fd3559d8fa5b199a41a>
Dec 23 15:02:34 viserion reth[5331]: 2024-12-23T06:02:34.312066Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x394d7d41a95f6ea1f6457923d5d4fb5f34916fa9163fe485d45>
Dec 23 15:07:23 viserion reth[5331]: 2024-12-23T06:07:23.766671Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x8f9a3e6498c60f2525942b34b9805a1f3ff673bcbc41c0da1f3>
Dec 23 15:35:22 viserion reth[5331]: 2024-12-23T06:35:22.447870Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x20e7f2b4292adacb174babcc79c31ef3af098f793c7fe05d2d7>
Dec 23 16:27:37 viserion reth[5331]: 2024-12-23T07:27:37.038356Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xdc015210af8718e2d6305affa13143c660bacc956f77b478a73>
Dec 23 16:50:58 viserion reth[5331]: 2024-12-23T07:50:58.044333Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x796842212aebefad78a4ee1225124f1b14ed85d506385d7465d>
Dec 23 17:10:32 viserion reth[5331]: 2024-12-23T08:10:32.970674Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x80d1f003922601552569d3bc4a7eaf3203de6ec5f9bf5f53a48>
Dec 23 17:11:22 viserion reth[5331]: 2024-12-23T08:11:22.186140Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4f383ed7377a29327cbd02393d937cfaf7959f51bc60ee1f192>
Dec 23 18:01:45 viserion reth[5331]: 2024-12-23T09:01:45.964238Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x027f3c0cce5838d4a1964b57aa2c0daef176db1d216455be26a>

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

reth Version: 1.1.4-dev
Commit SHA: 058cfe2
Build Timestamp: 2024-12-19T15:21:06.089859378Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

block_interval = 5

If you've built Reth from source, provide the full command you used

RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf --features jemalloc,asm-keccak

Code of Conduct

• I agree to follow the Code of Conduct

[mattsse]

does this happen consistently or only close after restart?

the block execution times are very high, exceeding 2s at some point, which seems odd.

what's your CPU and disk?

[ChosunOne]

It happens consistently, even after performing database compaction as well, though notably less frequently.
I have 32 GB of 2666MHz memory RAM with an AMD Ryzen 5 3400G CPU and a Kingston SKC300D4096G nvme ssd.

[ChosunOne]

Here's a good sample of the interval I'm seeing it today:

Dec 23 12:57:11 viserion reth[5331]: 2024-12-23T03:57:11.836050Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x7c931078005ee63fe658126e10e70a72cce9113036f9e79d5c6>
Dec 23 12:57:34 viserion reth[5331]: 2024-12-23T03:57:34.714858Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x92f162efd9e5edfdfa144855c2339dacddd999a5d1399cb17c5>
Dec 23 12:57:57 viserion reth[5331]: 2024-12-23T03:57:57.858123Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x9b66a85f09ee78a6bf3d10e3797b43c2ead5a704d684f583f5a>
Dec 23 14:11:58 viserion reth[5331]: 2024-12-23T05:11:58.432739Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xa45158d1ceef54581c05d5b0e435e29f7dfb83fc33dc00f442f>
Dec 23 14:30:30 viserion reth[5331]: 2024-12-23T05:30:30.494810Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x35834241988a8efc84fe761511cbecff2fefd3888b4e9f14d33>
Dec 23 14:30:30 viserion reth[5331]: 2024-12-23T05:30:30.523652Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x35834241988a8efc84fe761511cbecff2fefd3888b4e9f14d33>
Dec 23 14:43:26 viserion reth[5331]: 2024-12-23T05:43:26.477800Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x9286f63bdaf95adbc2a4f9fe63440dd386ac03a651fa72c26dc>
Dec 23 14:50:00 viserion reth[5331]: 2024-12-23T05:50:00.076807Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4e1fb4a34d16c76582d405190a9ee73f5afb0d642ad2674ac27>
Dec 23 14:50:00 viserion reth[5331]: 2024-12-23T05:50:00.084805Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4e1fb4a34d16c76582d405190a9ee73f5afb0d642ad2674ac27>
Dec 23 15:02:21 viserion reth[5331]: 2024-12-23T06:02:21.843992Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x118d0ff1ca208c03204613beceb1c959fd3559d8fa5b199a41a>
Dec 23 15:02:34 viserion reth[5331]: 2024-12-23T06:02:34.312066Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x394d7d41a95f6ea1f6457923d5d4fb5f34916fa9163fe485d45>
Dec 23 15:07:23 viserion reth[5331]: 2024-12-23T06:07:23.766671Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x8f9a3e6498c60f2525942b34b9805a1f3ff673bcbc41c0da1f3>
Dec 23 15:35:22 viserion reth[5331]: 2024-12-23T06:35:22.447870Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x20e7f2b4292adacb174babcc79c31ef3af098f793c7fe05d2d7>
Dec 23 16:27:37 viserion reth[5331]: 2024-12-23T07:27:37.038356Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xdc015210af8718e2d6305affa13143c660bacc956f77b478a73>
Dec 23 16:50:58 viserion reth[5331]: 2024-12-23T07:50:58.044333Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x796842212aebefad78a4ee1225124f1b14ed85d506385d7465d>
Dec 23 17:10:32 viserion reth[5331]: 2024-12-23T08:10:32.970674Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x80d1f003922601552569d3bc4a7eaf3203de6ec5f9bf5f53a48>
Dec 23 17:11:22 viserion reth[5331]: 2024-12-23T08:11:22.186140Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x4f383ed7377a29327cbd02393d937cfaf7959f51bc60ee1f192>
Dec 23 18:01:45 viserion reth[5331]: 2024-12-23T09:01:45.964238Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0x027f3c0cce5838d4a1964b57aa2c0daef176db1d216455be26a>

[ChosunOne]

And here is a graph of block processing latency over 12 hours (execution time in yellow, state root in green, stacked):