Race condition in compiler download when more than one Hardhat process is compiling

[nezouse]

Version of Hardhat

2.12.7

What happened?

We are using hardhat to compile and test our contracts. Everything was working fine, but recently sometimes when running build we got
Error: spawn /home/github/.cache/hardhat-nodejs/compilers-v2/linux-amd64/solc-linux-amd64-v0.8.16+commit.07a7930e EACCES

and right now it happens on nearly every run.

We are running it on self hosted runner with turborepo, so a few packages can be compiled at the same time (which i guess may be the cause of error, but everything was fine before)

Minimal reproduction steps

contracts-common:setup-build: > hardhat compile
contracts-common:setup-build:
contracts-common:setup-build: An unexpected error occurred:
contracts-common:setup-build:
contracts-common:setup-build: Error: spawn /home/github/.cache/hardhat-nodejs/compilers-v2/linux-amd64/solc-linux-amd64-v0.8.16+commit.07a7930e EACCES
contracts-common:setup-build: at Process.ChildProcess._handle.onexit (node:internal/child_process:283:[19]
contracts-common:setup-build: at onErrorNT (node:internal/child_process:478:16)
contracts-common:setup-build: at processTicksAndRejections (node:internal/process/task_queues:83:[21] {
contracts-common:setup-build: errno: -13,
contracts-common:setup-build: code: 'EACCES',
contracts-common:setup-build: syscall: 'spawn /home/github/.cache/hardhat-nodejs/compilers-v2/linux-amd64/solc-linux-amd64-v0.8.16+commit.07a7930e',
contracts-common:setup-build: path: '/home/github/.cache/hardhat-nodejs/compilers-v2/linux-amd64/solc-linux-amd64-v0.8.16+commit.07a7930e',
contracts-common:setup-build: spawnargs: [ '--standard-json' ],
contracts-common:setup-build: cmd: '/home/github/.cache/hardhat-nodejs/compilers-v2/linux-amd64/solc-linux-amd64-v0.8.16+commit.07a7930e --standard-json'
contracts-common:setup-build: }
contracts-common:setup-build:  ELIFECYCLE  Command failed with exit code 1.

Search terms

solc, compile

[fvictorio]

@nezouse do you mean that you have multiple Hardhat projects and they are being compiled at the same time?

If that's the case, maybe we have a race condition in our compiler downloader.

[HenryNguyen5]

We are having the same issue, our project is setup so that we're installing multiple versions of compilers across our project. https://github.com/smartcontractkit/chainlink/actions/runs/4620609361/jobs/8170964978

[fvictorio]

Thank you @HenryNguyen5, I'll see if I can create a minimal reproduction for this.

[HenryNguyen5]

Hi @fvictorio, I opened a PR here that resolves it.

I was unable to create a minimal repro on my side, I was just using the chainlink repo to repro. LMK if you want a bash script to repro it on smartcontractkit/chainlink, the repo is pretty big though.

[fvictorio]

I created a reproduction example here. It doesn't really produces the same error as the one reported by OP, but assuming the underlying cause is the same, it should be enough.

@HenryNguyen5 I tried your PR in that repository and it didn't really fix the problem. This makes sense because, while you added a mutex, that is local to the process. The problem here is that multiple hardhat compile processes are executed at the same time.

Edit: better reproduction repository here -> https://github.com/robercano/hardhat-concurrency-issue

[robercano]

Let me know if you need any help or extra tests with the reproduction repo

[HenryNguyen5]

I created a reproduction example here. It doesn't really produces the same error as the one reported by OP, but assuming the underlying cause is the same, it should be enough.

@HenryNguyen5 I tried your PR in that repository and it didn't really fix the problem. This makes sense because, while you added a mutex, that is local to the process. The problem here is that multiple hardhat compile processes are executed at the same time.

Edit: better reproduction repository here -> robercano/hardhat-concurrency-issue

Could you explain to me why this portion of the code had a getConcurrencySafeDownloader call in the first place then? It led me to believe this is where concurrent operations happen regarding multiple hardhat compilers being downloaded. Running the PR branch against our repository smartcontractkit/chainlink fixes the issue of multiple compiler downloads, leading to the PR checks passing.

I think your reproduction repository is exposing a different concurrency bug. The one I was running into was that a single invocation of hardhat compile with multiple supported versions would cause a race cond. The reproduction repo assumes that hardhat compile is called multiple times, so that each process is unaware of each other.

[fvictorio]

I think your reproduction repository is exposing a different concurrency bug. The one I was running into was that a single invocation of hardhat compile with multiple supported versions would cause a race cond. The reproduction repo assumes that hardhat compile is called multiple times, so that each process is unaware of each other.

Ah, my bad. We might need to re-visit this then to see if there are actually two problems here.

[HenryNguyen5]

Ah, my bad. We might need to re-visit this then to see if there are actually two problems here.

I tried creating a minimal repro for you for multi-version-single-compile, but it doesnt show up unless there is a large codebase along with multiple compilation versions. It seems like there needs to be multiple compiler downloads interspersed with compiilations that are undergoing / overlapping to surface this separate bug.

I'll end up creating a test branch you can checkout from https://github.com/smartcontractkit/chainlink and run locally so that you can triage effectively.

The codebase is large enough that after a few iterations you'll see the unpatched HH error out on a compiler violation but the patch version will continue over many iterations (until you get bored :)).

I hope that'll be enough to get this fix merged in, the changes are pretty minimal and its breaking our CI badly

[HenryNguyen5]

Hi @fvictorio, please see the following test branch with its associated script: https://github.com/smartcontractkit/chainlink/blob/fix/RE_1365_hardhat_compiler_failure/contracts/test_compilation.sh

[fvictorio]

@HenryNguyen5 approximately how many times did the compilation run until you got the error? I've been running it for a while and I'm still not observing it.

[fvictorio]

Ah, my bad, I was running hh clean --global to remove the downloaded compilers, but the repo is using an older version of TypeChain which has a bug that prevents --global from working. I'm trying again now.

[fvictorio]

Ok, I can confirm it now, thanks for the clear reproduction!

[fvictorio]

I'm going to close this issue in favor of two new issues I opened: #4490 for the problem when a single compilation has an issue, and #4491 for the problem when multiple concurrent compilations are executed by different Hardhat processes. Thanks everyone for the info!