Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Replay - Failed Capturing - `GLIBC_2.29' not found (better-sqlite3) #27902

Closed
aNyMoRe0505 opened this issue Sep 26, 2023 · 35 comments · Fixed by #27968
Closed

Test Replay - Failed Capturing - `GLIBC_2.29' not found (better-sqlite3) #27902

aNyMoRe0505 opened this issue Sep 26, 2023 · 35 comments · Fixed by #27968

Comments

@aNyMoRe0505
Copy link

Current behavior

Hi, recently I've been testing v13's Test Replay.
After the CI run, I saw the report on the dashboard, but the Tests for review section in the overview is empty.
Checking the pipeline logs, I found the following error:

Test Replay - Failed Capturing - /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /root/.cache/Cypress/13.2.0/Cypress/resources/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node)

Is it because the image I'm using is not supported ? Thank you

Desired behavior

No response

Test code to reproduce

N/A

Cypress Version

13.2.0

Node version

v18.4.0

Operating System

N/A

Debug Logs

No response

Other

No response

@MikeMcC399
Copy link
Contributor

@aNyMoRe0505

Could you say exactly which operating system name and operating system version your CI is running under? It looks like it must be a Linux variant. In the issue form I read "N/A" under Operating System.

@aNyMoRe0505
Copy link
Author

@MikeMcC399 thank you for your quick response.
I'm not sure if I correctly understand your question.
Our image is running on "slim", and I might need a bit more time to provide you with the version.
If I misunderstood, please let me know. Thank you.

@MikeMcC399
Copy link
Contributor

MikeMcC399 commented Sep 26, 2023

@aNyMoRe0505

Our image is running on "slim"

"slim" is not enough to identify the operating system. It usually means that it is a minimum configuration of an operating system, however it doesn't say which one.

An example of an operating system name is "Ubuntu" with version number "22.04.3 LTS"

Please check that you are running one of the operating systems supported by Cypress according to the documentation listed under System requirements.

Which CI system are you running on? Typical CI systems which Cypress Cloud provides templates for are shown below. CI Provider Examples gives some other CI systems which are also used with Cypress Cloud.

image

@jennifer-shehane
Copy link
Member

There's some guidance for CentOS Linux 7 users in our docs, we've typically found that is the case when this error is encountered. https://docs.cypress.io/guides/cloud/test-replay#Troubleshooting

@MikeMcC399
Copy link
Contributor

@aNyMoRe0505

Were you able to find out your exact operating system? You may be able to see it listed in the logs of a test run or by executing

npx cypress info

You can find an example on cypress info where it lists the System Platform and version darwin (19.2.0):

Cypress Version: 4.1.0
System Platform: darwin (19.2.0)
System Memory: 17.2 GB free 670 MB

@aNyMoRe0505
Copy link
Author

@jennifer-shehane @MikeMcC399
Thank you for your response.
I'm sorry, I'm a bit short on time today.
I will reply tomorrow.

@drochgenius
Copy link

Facing the same issue on Ubuntu 18, Cypress 13.3.0.

It looks like this is your first time using Cypress: 13.3.0

[STARTED] Task without title.
[TITLE]  Verified Cypress!       /root/.cache/Cypress/13.3.0/Cypress
[SUCCESS]  Verified Cypress!       /root/.cache/Cypress/13.3.0/Cypress

Opening Cypress...

DevTools listening on ws://127.0.0.1:39581/devtools/browser/24b031d5-d560-46f1-a5ab-92ec828ca269
/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /root/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
Error: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /root/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
    at process.func [as dlopen] (node:electron/js2c/asar_bundle:2:1869)
    at Module._extensions..node (node:internal/modules/cjs/loader:1354:18)
    at Object.func [as .node] (node:electron/js2c/asar_bundle:2:1869)
    at Module.load (node:internal/modules/cjs/loader:1124:32)
    at Module._load (node:internal/modules/cjs/loader:965:12)
    at f._load [as origLoad] (node:electron/js2c/asar_bundle:2:13377)
    at PackherdModuleLoader.tryLoad (evalmachine.<anonymous>:1:741495)
    at Function.<anonymous> (evalmachine.<anonymous>:1:751709)
    at d._load (<embedded>:4510:29351)
    at Module.require (node:internal/modules/cjs/loader:1148:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.H (<embedded>:4562:137346)
    at s.exports (<embedded>:4562:143111)
    at <embedded>:4569:3280
    at tryCatcher (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/util.js:16:23)
    at Function.<anonymous> (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/method.js:39:29)
    at Object.runElectron (<embedded>:4569:3050)
    at Object.startInMode (<embedded>:4569:5738)
    at <embedded>:4569:4826
    at tryCatcher (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._settlePromise (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:582:21)
    at Promise._settlePromise0 (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at Promise._settlePromise (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:582:21)
    at Promise._settlePromise0 (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:694:18)
    at Promise._fulfill (/root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/promise.js:638:18)
    at /root/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/bluebird/js/release/nodeback.js:42:21
    at <embedded>:2086:115512
Missing baseUrl in compilerOptions. tsconfig-paths will be skipped

@Narretz
Copy link
Contributor

Narretz commented Sep 27, 2023

I had this after updating Cypress to 13.3.0 from 12.5.1, but with a quite old version of cypress-browsers (with Chrome 96) in CI. the glibc version is probably updated when ubuntu gets updated. I updated to a recent cypress-browsers version and it works. This is probably because recent Cypress versions use recent version of Electron, which has a requirement for glibc.

This is actually about opening Cypress #27920

@aNyMoRe0505
Copy link
Author

aNyMoRe0505 commented Sep 28, 2023

@MikeMcC399

Thank you for your response!
Below is the information I obtained after executing cypress info

Cypress Version: 13.2.0 (stable)
System Platform: linux (Debian - 10.13)
System Memory: 67.4 GB free 57.9 GB

I also want to ask a beginner question make sure I understand correctly. Does the choice of Docker image determine the system operation?

It's a bit strange.
I was able to run the tests yesterday and there were no replay records in Cypress Cloud.
However, today, I can't even run the tests, experiencing the same situation as @drochgenius

@MikeMcC399
Copy link
Contributor

@aNyMoRe0505

If you are using a Docker image then you could update to a more recent one from https://hub.docker.com/r/cypress/browsers, such as

docker pull cypress/browsers:node-18.15.0-chrome-106.0.5249.61-1-ff-99.0.1-edge-114.0.1823.51-1

depending on your needs.

@tkharuk
Copy link

tkharuk commented Sep 28, 2023

Hi,
last week we've upgraded a couple of our project to Cypress v13,
and it worked great for a couple of days!

Error suddenly appeared on "nighlty" run on CircleCI

  • yesterday it was green, now it is read - no changes in code or infra
config.yml:
    docker:
      - image: 'cypress/base:16.13.2'
    steps:
      - jq/install
    ...

Attempt to upgrade to latest cypress/base

  • tried to update docker to cypress/base:20.6.1 and cypress/base:18 and faced new issue
curl: command not found

coming from jq orb

Immediate actions?

As test checks are mandatory to merge PR, this now blocks multiple teams.

@MikeMcC399 @jennifer-shehane what would you suggest to do? rollback to Cypress v12 as an immediate action?

Is there anything on your side that can be done? (pipeline stopped working without our changes in codebase/infra)


P.S. Maybe documentation should be more demanding and show the significance of upgrading the infrastructure, which can be breaking:

We've followed

unlike in migration to v12 it doesn't mention any node or any other infrastructural changes required.

In a more detailed instruction it did mention:

Node 14 support has been removed and Node 16 support has been deprecated. Node 16 may continue to work with Cypress v13, but will not be supported moving forward to closer coincide with Node 16's end-of-life schedule. It is recommended that users update to at least Node 18.

Maybe we misunderstood the "may" part and that it will stop working after 1 week

@MikeMcC399
Copy link
Contributor

MikeMcC399 commented Sep 28, 2023

@tkharuk

  • It sounds like you are encountering issues which are unrelated to `GLIBC_2.29' which is the topic of this thread.

  • The issue with curl is discussed in curl: command not found cypress-docker-images#649 and you will need to install curl yourself.

  • jq orb is not part of Cypress.

In practice, if you use Node.js 16 it will not suddenly stop working on its own. It is just in end-of-life and unsupported. GitHub for instance will continue running Node.js until early next year (2024).
Edit: I didn't know that binaries (cypress.zip on https://download.cypress.io) for previous Cypress 13.x releases had been replaced when I wrote the previous comment.

@BernhardPosselt
Copy link

Running into a similar issue on the latest node image for Debian Buster (node:20.7.0-buster)

Error: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /root/.cache/Cypress/13.2.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)

I suppose that's because Buster ships glibc 2.28. Has Cypress dropped support for that Debian version?

@tkharuk
Copy link

tkharuk commented Sep 28, 2023

@MikeMcC399 I'm facing the same issue:

/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /root/.cache/Cypress/13.0.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
Error: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /root/.cache/Cypress/13.0.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node)
    at process.func [as dlopen] (node:electron/js2c/asar_bundle:5:1810)
    at Module._extensions..node (node:internal/modules/cjs/loader:1205:18)
    at Object.func [as .node] (node:electron/js2c/asar_bundle:5:1810)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:829:12)
    at c._load [as origLoad] (node:electron/js2c/asar_bundle:5:13339)
    at PackherdModuleLoader.tryLoad (evalmachine.<anonymous>:1:741495)
    at Function.<anonymous> (evalmachine.<anonymous>:1:751709)

And it started happening after a couple of days after an upgrade on a nightly run.
@MikeMcC399 any ideas how is this possible? Why it was green yesterday and red today without any code changes?

the curl issue is another step issue after I did what you proposed - upgrading to "cypress/browsers:node-18.15.0-chrome-106.0.5249.61-1-ff-99.0.1-edge-114.0.1823.51-1".
So cypress/base:16 had necessary tools but cypress/base:18+ does not.

@BernhardPosselt
Copy link

BernhardPosselt commented Sep 28, 2023

@tkharuk same here, it built fine yesterday but CI broke today (with locked dependencies in yarn.lock so deps shouldn't have changed). Sounds like they are pulling down binaries dynamically?

@josephearl
Copy link

@BernhardPosselt same here, was working fine yesterday morning, now failing

Please revert whatever change was made!

@MikeMcC399
Copy link
Contributor

@tkharuk

I have to pass on answering your question as I am just a community contributor and I'm not part of the Cypress engineering team.

@MikeMcC399
Copy link
Contributor

MikeMcC399 commented Sep 28, 2023

@drochgenius

I have just been looking at Ubuntu 18.04 at it seems that there is another related issue.

With Node.js 16
node -v shows
v16.20.2

With Node.js 18
node -v shows
(node: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.28' not found (required by node))

On Ubuntu 18.04:

ldd --version
(ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27), 

so that is a problem of its own right going forward, as Node.js 16 has reached end-of-life.

GitHub deprecated Ubuntu 18.04 and removed it as a runner operating system on April 4, 2023. According to https://ubuntu.com/18-04 it is now on extended support.

Edit: See also

Ubuntu 20.04 is the earliest version supported by Node.js 18.
See Node.js 18 announcement -> Toolchain and Compiler Upgrades and Node.js 18 > BUILDING.md.

@MatiDiez
Copy link

MatiDiez commented Sep 28, 2023

Same thing happening here (Just to add votes)

Running cypress on github actions is failing with version GLIBC_2.29 not found error.
It was working perfectly yesterday with Cypress 13.2.0

@grollat
Copy link

grollat commented Sep 28, 2023

Same here on Amazon Linux

@drochgenius
Copy link

drochgenius commented Sep 28, 2023

I found that if you are running Ubuntu 18, it ships with an earlier version of GLIBC.
I suspect something changed in the last release that now requires GLIBC > 2.29.

If you are using Chrome browser for running your test, then
another hypothesis is that the requirement could come from the latest stable version of Chrome, rather than Cypress.
The stable version was updated on Sept 27. https://www.ubuntuupdates.org/package/google_chrome/stable/main/base/google-chrome-stable

By installing version 114 of Chrome, I saw the issue fixed.

ENV CHROME_VERSION=114.0.5735.198-1
RUN wget -q https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb && \
    dpkg -i google-chrome-stable_${CHROME_VERSION}_amd64.deb

Also, it seems that if you're able to upgrade Ubuntu to version 19, or 20, you should not have the issue because they updated glibc in those distibutions.

@BernhardPosselt
Copy link

Does Cypress download a separate Chromium instance? Because I'm installing Chromium in Debian Buster; would be kinda weird to ship a non working Chromium version through the official package manager.

@jennifer-shehane jennifer-shehane changed the title Test Replay - Failed Capturing - /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found Test Replay - Failed Capturing - `GLIBC_2.29' not found (better-sqlite3) Sep 28, 2023
@RomekRJM
Copy link

@tkharuk same here, it built fine yesterday but CI broke today (with locked dependencies in yarn.lock so deps shouldn't have changed). Sounds like they are pulling down binaries dynamically?

Does Cypress actually pull external dependencies? I'm asking, as similarly to ☝️ we are using docker image with cypress + chrome, pinned to a specific sha256 tag and it also stopped working recently. Last successful run was 2 days ago and we had no changes since then.

@samtsai
Copy link
Contributor

samtsai commented Sep 28, 2023

To add, I downgraded Cypress to 12.8.0 and this issue is resolved. Going to troubleshoot a little more to see what else changed.

@jason-stoudt
Copy link

jason-stoudt commented Sep 28, 2023

Another data point:

Application Data: /home/githubrunner/.config/cypress/cy/development
Browser Profiles: /home/githubrunner/.config/cypress/cy/development/browsers
Binary Caches: /home/githubrunner/.cache/Cypress

Cypress Version: 13.1.0 (stable)
System Platform: linux (Amazon Linux - 2 (Karoo))
System Memory: 8.16 GB free 6.77 GB

@jennifer-shehane
Copy link
Member

jennifer-shehane commented Sep 28, 2023

There are 2 separate issues that are happening In this thread. We’d like to make sure each of you is following the proper thread to get a resolution. They are both showing VERY similar errors which makes this confusing.

  • This original issue is tracking an error that displays when recording Test Replay, where it fails to capture. This does NOT prevent Cypress from opening. You’ll see the following error referencing better-sqlite3.node at the end of the error.
    Test Replay - Failed Capturing - /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /root/.cache/Cypress/13.2.0/Cypress/resources/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node)
  • This issue covers an error some are seeing OPENING cypress. Cypress 13 - GLIBC_2.29 not found (addon-native) - preventing opening cypress #27920 The error message references addon-native.node at the end of the error message.
    Opening Cypress... DevTools listening on ws://127.0.0.1:42471/devtools/browser/31a66146-18bd-478a-b7cc-acbae92826e7 /lib64/libm.so.6: version 'GLIBC_2.29' not found (required by /home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node) Error: /lib64/libm.so.6: version 'GLIBC_2.29' not found (required by /home/jenkins/.cache/Cypress/13.3.0/Cypress/resources/app/packages/server/lib/modes/addon/addon-native.node).

Please comment and follow the correct issue based on what error is occurring.

For this issue, where Test Replay is not captured, we are investigating better documentation to provide and seeing if we can provide a better error message. This is not something we can fix since it’s system dependent as noted above.

We’re investigating the new issue with opening cypress. Follow in that issue for updates. #27920

@MikeMcC399
Copy link
Contributor

@drochgenius

If you are using Chrome browser for running your test, then another hypothesis is that the requirement could come from the latest stable version of Chrome, rather than Cypress. The stable version was updated on Sept 27. https://www.ubuntuupdates.org/package/google_chrome/stable/main/base/google-chrome-stable

By installing version 114 of Chrome, I saw the issue fixed.

Also, it seems that if you're able to upgrade Ubuntu to version 19, or 20, you should not have the issue because they updated glibc in those distibutions.

@jennifer-shehane
Copy link
Member

jennifer-shehane commented Sep 28, 2023

The Cypress provided docker-images with Node.js 18+ are on GLIBC 2.31. You should be able to use our docker-images and not encounter this error. cypress-io/cypress-docker-images

As mentioned before we're investigating possible fixes/catching/documentation for people not using our docker-images this so it doesn't require changes to get around the error.

@MikeMcC399
Copy link
Contributor

On Ubuntu 18.04 / Node.js 16.16.0 / [email protected] I can now successfully execute

npx cypress run --record --key <my-key>

I get the warning

  • Test Replay - Failed Capturing - /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/mike/.cache/Cypress/13.3.0/Cypress/resources/app/node_modules/better-sqlite3/build/Release/better_sqlite3.node)

so Test Replay has failed, but Cypress has recorded everything else to the Cypress Cloud.

Previously on Ubuntu 18.04 it was not possible to even start Cypress 13.x without getting a GLIBC error, so that is a definite improvement for this environment.

@MikeMcC399
Copy link
Contributor

@aNyMoRe0505

At lot of other users have joined your issue! But just to get back to you as the original poster of the issue, is it now resolved for you or are you still having problems?

If yes, then perhaps you could reply with your current error message and let us know which Docker image you are running in?

@drochgenius
Copy link

In our case, we were able to upgrade to Ubuntu 20, and it resolved the issue.

@aNyMoRe0505
Copy link
Author

@MikeMcC399
I'm sorry, I've been a bit busy lately.
I'll try to use the image provided by Cypress and get back to here as soon as possible after testing.

@jennifer-shehane
Copy link
Member

The Test Replay error is still an issue, but we have a fix already that we’re working on for this issue. Will update the issue when it’s out.

@jennifer-shehane
Copy link
Member

This will be fixed in the release going out today 13.3.1

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Oct 11, 2023

Released in 13.3.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.3.1, please open a new issue.

@cypress-bot cypress-bot bot locked as resolved and limited conversation to collaborators Oct 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.