Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredumps on Ubuntu pipeline runs #760

Open
ksaur opened this issue Feb 14, 2024 · 2 comments
Open

coredumps on Ubuntu pipeline runs #760

ksaur opened this issue Feb 14, 2024 · 2 comments

Comments

@ksaur
Copy link
Contributor

ksaur commented Feb 14, 2024

We thought that TVM was causing Ubuntu out-of-memory errors, so we skipped all the TVM tests in Ubuntu (#709).

But now, I am seeing additional coredumps in Ubuntu runs that aren't related to TVM.
Example 1
Example 2

It reaches the end of the Test with Pytest stage successfully, and then dies:

 ========== 598 passed, 66 skipped, 1410 warnings in 214.69s (0:03:34) ========== 
 /home/runner/work/_temp/c325d19a-2f4b-46db-bd6e-51d55c415279.sh: line 1:  2350 Aborted  (core dumped) pytest 
 Error: Process completed with exit code 134.

It also appears to be transient. Is this at all related to what you were seeing @mshr-h ? Maybe the problem is bigger than TVM?

@mshr-h
Copy link
Collaborator

mshr-h commented Feb 14, 2024

I was seeing a similar error, like all the tests are passed but the stage fails.
Can you try to skip the TVM installation in the pipeline? I'm guessing when the Python interpreter imports TVM, something happens.

@ksaur
Copy link
Contributor Author

ksaur commented Feb 21, 2024

It's been a full week (with many runs) of this and no error. Maaaaaybe the memory issue was transient on the github-actions side? Let's hope 🤞...I will leave this open a month or so to see if happens again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants