[BUG]: Spontaneous TaskFailedException

[gm89uk]

What happened?

After running a search with a custom loss function overnight, I opened an excel sheet and the TaskFailedException happened. I'm running on 16GB ram, I'm not sure if that has anything to do with it, but I've been running the same code for months, with many more things open and this only happened with beta2.

Thank you again for your work with this.

Version

v1.0.0beta2

Operating System

Windows

Interface

Julia REPL

Relevant log output

ERROR: TaskFailedExceptionre      Equation
Stacktrace: 7.582e-01  1.594e+01  y = 2.3557
  [1] wait  7.182e-01  2.713e-02  y = -0.201 * a
    @ .\task.jl:352 [inlined]-02  y = 0.23635 * (r - a)
  [2] fetch 6.126e-01  6.060e-02  y = (0.61059 * (r - p)) - a
    @ .\task.jl:372 [inlined]-02  y = r - ((p * 1.8532) + (a / 0.44432))
  [3] _main_search_loop!(state::SymbolicRegression.SearchUtilsModule.SearchState{…}, datasets::Vector{…}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{…}, options::Options{…})
    @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:829
  [4] _equation_search(datasets::Vector{Dataset{…}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multithreading, 1, true}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{…}, DynamicExpressions.OperatorEnumModule.OperatorEnum, DynamicExpressions.NodeModule.Node, Expression, @NamedTuple{}, MutationWeights, true, true, nothing, Nothing, 5}, saved_state::Nothing)
    @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:533
  [5] equation_search(datasets::Vector{…}; options::Options{…}, saved_state::Nothing, runtime_options::Nothing, runtime_options_kws::@Kwargs{…})
    @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:52338)))) * 1.2304
  [6] equation_search  1.932e-03  y = ((0.15476 * ((l + (d * 1.2641)) + g)) + (r - ((a - -13.138) * ((p * 0.13204) - -2.3591)))) * 1.2316
    @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:504 [inlined]707) * a)) - p) * 1.2366) - p)
  [7] equation_search(X::Matrix{…}, y::Matrix{…}; niterations::Int64, weights::Vector{…}, options::Options{…}, variable_names::Vector{…}, display_variable_names::Vector{…}, y_variable_names::Nothing, parallelism::Symbol, numprocs::Nothing, procs::Nothing, addprocs_function::Nothing, heap_size_hint_in_bytes::Nothing, runtests::Bool, saved_state::Nothing, return_state::Bool, run_id::Nothing, loss_type::Type{…}, verbosity::Int64, progress::Nothing, X_units::Nothing, y_units::Nothing, extra::@NamedTuple{}, v_dim_out::Val{…}, multithreaded::Nothing, varMap::Nothing)- p) / 0.78218
    @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:475- (((p * 0.12872) - -2.3832) * (a - -6.1259)))) / 0.76823
  [8] #equation_search#21065e-04  y = (r + (((0.13712 * ((((g * 0.76823) - (p * (r / 1.5913))) + l) + (d * 1.2775))) - p) - (((p * 0.12872) - -2.3832) * (a - -6.1228)))) / 0.76823
    @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:497 [inlined]p * 0.50005)))) * 0.13297)) - ((a - -6.1258) * ((p * 0.13199) - -2.4217))) / 0.77853
  [9] _update(m::SRRegressor{…}, verbosity::Int64, old_fitresult::Nothing, old_cache::Nothing, X::@NamedTuple{…}, y::Vector{…}, w::Vector{…}, options::Options{…}, classes::Nothing)) / 0.77853
    @ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:191 0.13464) + (r - (((0.13255 * p) - -2.4211) * (a - -6.1259)))) - p) / 0.77253
 [10] update(m::SRRegressor{DynamicQuantities.SymbolicDimensions{…}, DataType}, verbosity::Int64, old_fitresult::Nothing, old_cache::Nothing, X::@NamedTuple{a::SubArray{…}, r::SubArray{…}, p::SubArray{…}, d::SubArray{…}, l::SubArray{…}, g::SubArray{…}}, y::Vector{Float32}, w::Vector{Float32})
    @ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:142(r + -2.3217) * (r - d)) * -0.063971) * g) + p) * 0.64795)))) + l) * -0.13559))) - p) / 0.79963
 [11] fit(m::SRRegressor{DynamicQuantities.SymbolicDimensions{…}, DataType}, verbosity::Int64, X::@NamedTuple{a::SubArray{…}, r::SubArray{…}, p::SubArray{…}, d::SubArray{…}, l::SubArray{…}, g::SubArray{…}}, y::Vector{Float32}, w::Vector{Float32})
    @ SymbolicRegression.MLJInterfaceModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MLJInterface.jl:136(p + 0.65992)) + -2.3217) * g) * (-0.063971 * (r - d))) + (p * 0.63568)) * r))) + l) * -0.13559))) - p) / 0.79963
 [12] fit_only!(mach::Machine{SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, true}; rows::Nothing, verbosity::Int64, force::Bool, composite::Nothing)
    @ MLJBase C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:69280831 * r) * (-1.6799 + (r - (d * d)))) * g) + ((-0.29098 * d) + p)) * (r * 0.63569))))) * 0.13196) - ((a - -6.1258) * ((0.13254 * p) - -2.4217))) + r) - p) / 0.79963) / 0.9875
 [13] fit_only!84e-01  4.239e-04  y = ((r - p) + ((((l + ((1.3129 * d) + g)) - (((((r * ((d * d) - (-1.9856 + r))) * 0.091608) * g) + ((d * (-0.094242 * (d * d))) + p)) * (r * 0.63569))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
    @ C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:617 [inlined](((r * ((d * d) - (-1.9856 + r))) * 0.091608) * g) + ((d * (-0.094242 * (a + (d * d)))) + p)) * (r * 0.63569)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
 [14] #fit!#63679e-01  2.768e-04  y = ((r - p) + (((l + (((1.3129 * d) + g) - ((((((r + 0.63569) * ((d * d) - (-1.9856 + r))) * (0.091608 * g)) + ((d * -0.094242) * (a + (d * d)))) + p) * (0.63569 * r)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
    @ C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:789 [inlined]((((r + 0.63569) * ((d * d) - (-1.9856 + r))) * (0.091608 * g)) + ((d * -0.094242) * (a + (d * d)))) + (p / 1.0506)) * (0.63569 * r)))) * 0.13196) - ((a - -6.1258) * ((0.13196 * p) - -2.4217)))) / 0.79963
 [15] fit!(mach::Machine{SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, SRRegressor{DynamicQuantities.SymbolicDimensions{DynamicQuantities.FixedRational{Int32, 25200}}, DataType}, true})───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    @ MLJBase C:\Users\georg\.julia\packages\MLJBase\7nGJF\src\machines.jl:786
 [16] top-level scope
    @ REPL[55]:1

    nested task error: TaskFailedException
    Stacktrace:
     [1] wait
       @ .\task.jl:352 [inlined]
     [2] fetch
       @ .\task.jl:372 [inlined]
     [3] (::SymbolicRegression.var"#60#65"{SymbolicRegression.SearchUtilsModule.SearchState{…}, Int64, Int64})()
       @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:932

        nested task error: InterruptException:
        Stacktrace:
          [1] Array
            @ .\boot.jl:477 [inlined]
          [2] vect
            @ .\array.jl:163 [inlined]
          [3] convert
            @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MutationWeights.jl:125 [inlined]
          [4] sample_mutation(w::MutationWeights)
            @ SymbolicRegression.CoreModule.MutationWeightsModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\MutationWeights.jl:135
          [5] next_generation(dataset::Dataset{…}, member::PopMember{…}, temperature::Float64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics, options::Options{…}; tmp_recorder::Dict{…})
            @ SymbolicRegression.MutateModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\Mutate.jl:182
          [6] next_generation
            @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\Mutate.jl:154 [inlined]
          [7] reg_evol_cycle(dataset::Dataset{…}, pop::Population{…}, temperature::Float64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics, options::Options{…}, record::Dict{…})
            @ SymbolicRegression.RegularizedEvolutionModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\RegularizedEvolution.jl:36
          [8] s_r_cycle(dataset::Dataset{…}, pop::Population{…}, ncycles::Int64, curmaxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics; verbosity::Int64, options::Options{…}, record::Dict{…})
            @ SymbolicRegression.SingleIterationModule C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SingleIteration.jl:47
          [9] s_r_cycle
            @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SingleIteration.jl:19 [inlined]
         [10] _dispatch_s_r_cycle(in_pop::Population{…}, dataset::Dataset{…}, options::Options{…}; pop::Int64, out::Int64, iteration::Int64, verbosity::Int64, cur_maxsize::Int64, running_search_statistics::SymbolicRegression.AdaptiveParsimonyModule.RunningSearchStatistics)
            @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:1066
         [11] _dispatch_s_r_cycle
            @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:1049 [inlined]
         [12] macro expansion
            @ C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SymbolicRegression.jl:916 [inlined]
         [13] (::SymbolicRegression.var"#59#64"{…})()
            @ SymbolicRegression C:\Users\georg\.julia\packages\SymbolicRegression\qz9xV\src\SearchUtils.jl:258
Some type information was truncated. Use `show(err)` to see complete types.

Extra Info

No response

[MilesCranmer]

Do you mean that the search was still running, and opening the .csv file, which was being written to by PySR simultaneously, crashed things?

I think Excel has some kind of autosave feature so perhaps a concurrent write messed with Julia? Not sure. The easiest solution is to copy the file, and then open the copied version in excel.

But the error message you have seems unrelated - that’s the (harmless) message you see if you just Ctrl-C while it is running the search.

[gm89uk]

Hi Miles,
Sorry I should have clarified, the Excel worksheet opened was not the csv search, just a very large one that takes a lot of memory; I'm aware opening the HoF CSV during the search will crash it.

Sorry I cannot provide more insight to it! I will let you know if I can repro it tomorrow.

[MilesCranmer]

I'm aware opening the HoF CSV during the search will crash it.

I think it's actually generally safe to do this! Was just thinking about possible explanations and that was a possibility. But in general usage I don't think this would cause issues.

The error itself seems to be fairly benign though, this is what happens if the process is told to exit by hitting <ctrl-c>. But perhaps it was told to exit by some other mechanism, such as the computer going to sleep or something? I'm not sure. The InterruptException you see (I think) is just Julia exiting because something told it to stop

[gm89uk]

It's quite plausible I pressed ctrl + c while on the REPL by mistake. Thank you again for looking into it

[MilesCranmer]

That must be it! Let me know if this sort of thing comes up again though without input