Garbage collection too passive on worker processes #237
Labels
feature (importance: high)
High-importance feature
Comments
MilesCranmer
changed the title
|
This actually might be the reason behind the various segfaults seen during the Windows CI tests... which I could just not seem to identify a root cause of in the source. Maybe |
|
cross-ref JuliaLang/julia#50673 |
|
Fix with #270 |
MilesCranmer
added a commit
that referenced
this issue
Feb 19, 2024
[Diff since v0.23.1](v0.23.1...v0.23.2) **Merged pull requests:** - Formatting overhaul (#278) (@MilesCranmer) - Avoid julia-formatter on pre-commit.ci (#279) (@MilesCranmer) - Make it easier to select expression from Pareto front for evaluation (#289) (@MilesCranmer) **Closed issues:** - Garbage collection too passive on worker processes (#237) - How can I set the maximum number of nests? (#285)
MilesCranmer
added a commit
that referenced
this issue
Feb 19, 2024
[Diff since v0.23.1](v0.23.1...v0.23.2) **Merged pull requests:** - Formatting overhaul (#278) (@MilesCranmer) - Avoid julia-formatter on pre-commit.ci (#279) (@MilesCranmer) - Make it easier to select expression from Pareto front for evaluation (#289) (@MilesCranmer) **Closed issues:** - Garbage collection too passive on worker processes (#237) - How can I set the maximum number of nests? (#285)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This might be a Julia bug. Well, not really a bug, but just something that needs better default behavior.
Right now it seems that Julia workers don't know about the memory allocation of other Julia workers, and so garbage collection does not happen as aggressively as it should.
The solution we can implement in SymbolicRegression.jl is for worker processes internall created by
addprocsto have a much smallerheap-size-limit(see https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint).The best resource allocation seems to be letting the head worker have as many threads and a large portion of memory available. But the worker processes should have a single thread and small memory available, so they garbage collect frequently. I think this could be done automatically.
The text was updated successfully, but these errors were encountered: