Add experimental support for customisable benchmarking

[Zentrik]

Replaces #325 (closes #325)

This pr adds the ability to run a custom benchmarking function which has hooks to inject in custom functions. The current design supports running perf on a benchmark (see https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl) and profiling benchmarks (excluding the setup, teardown and gcscrub which the current bprofile includes).

[DilumAluthge]

I haven't added tests, but given the CI doesn't have perf available, not sure how useful it would be.

We might need to set up Buildkite CI on this repo. @vchuravy @staticfloat

[Zentrik]

I'd like to mark this feature as experimental or something to that effect so that we can make breaking changes to it without making a breaking change to BenchmarkTools.
The changes to samplefunc are largely just so that it matches customisable_func (theoretically it reduces overhead by a couple of instructions but that's not noticeable) and also so the hooks for samplefunc are exposed.

I'll make a separate pr removing Buildkite.

I've moved the LinuxPerf stuff to https://github.com/Zentrik/BenchmarkToolsPlusLinuxPerf.jl.

I do have a version of bprofile that uses this functionality to only profile the relevant stuff, but because samples are generally quite short (on the order of microseconds) not much gets profiled making it pretty useless.

bprofile_setup_prehook(_) = Profile.check_init()
function bprofile_prehook()
    results = samplefunc_prehook()
    status = ccall(:jl_profile_start_timer, Cint, ())
    if status < 0
        error(Profile.error_codes[status])
    end
    return results
end
function bprofile_posthook()
    Profile.stop_timer()
    return samplefunc_posthook()
end

# """
#     @bprofile expression [other parameters...]

# Run `@benchmark` while profiling. This is similar to

#     @profile @benchmark expression [other parameters...]

# but the profiling is applied only to the main
# execution (after compilation and tuning).
# The profile buffer is cleared prior to execution.

# View the profile results with `Profile.print(...)`.
# See the profiling section of the Julia manual for more
# information.
# """
macro bprofile(args...)
    _, params = prunekwargs(args...)
    if !haskw(args, :gctrial)
        args = (args..., Expr(:kw, :gctrial, false))
    end
    if !haskw(args, :gcsample)
        args = (args..., Expr(:kw, :gcsample, false))
    end
    tmp = gensym()
    return esc(
        quote
            local $tmp = $BenchmarkTools.@benchmarkable $(args...)
            $(
                if hasevals(params)
                    :(run(
                        $tmp, $BenchmarkTools.Parameters($tmp.params; evals=1); warmup=false
                    ))
                else
                    :($BenchmarkTools.tune!($tmp))
                end
            )
            $BenchmarkTools.Profile.stop_timer()
            $BenchmarkTools.Profile.clear()
            $BenchmarkTools.run(
                $tmp;
                setup_prehook=$BenchmarkTools.bprofile_setup_prehook,
                prehook=$BenchmarkTools.bprofile_prehook,
                posthook=$BenchmarkTools.bprofile_posthook,
                sample_result=$BenchmarkTools.samplefunc_sample_result,
                enable_customisable_func=:ALL,
                run_customisable_func_only=true,
            )
        end,
    )
end

[vchuravy]

Fairly late to the game here, but I would find something like this useful.
I do wonder if we need the code-duplication and if we maybe could use dispatch to configure the behaviour here.

[willow-ahrens]

@vchuravy I agree, it would be nice to avoid code duplication, and I wonder if this PR could be achieved using functions as arguments.

[Zentrik]

Is the code duplication concern referring to the changes in parameters.jl?
What do you mean by using functions as arguments?

[willow-ahrens]

apologies for my delayed review as I finished up my graduate studies. I would also like to see linuxperf working here.

When I discuss duplicated code, I'm referring to how this PR uses several kwargs with similar names, such as

run_customizable_func_only: Bool false
    enable_customizable_func: Symbol FALSE
    customizable_gcsample: Bool false
    setup_prehook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func))
    teardown_posthook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func))
    sample_result: _nothing_func (function of type typeof(BenchmarkTools._nothing_func))
    prehook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func))
    posthook: _nothing_func (function of type typeof(BenchmarkTools._nothing_func))

as well as several repetitions of things like

if params.enable_customizable_func == :LAST
        params.customizable_gcsample && gcscrub()
        s = b.customizable_func(b.quote_vals, params)
        trial.customizable_result = s[1]

        if params.run_customizable_func_only
            return_val = s[end]
        end
    end

I don't know whether it is possible to make a simpler and hopefully more modular approach just yet, but I'd like to try!

By "using functions as arguments", I was insufficiently gesturing at something like the following approach (let me know whether you think this would work):

To benchmark the function ex, we start by constructing the samplefunc f:

function f(evals)
   res = nothing
   for i = 1:evals
      res = ex
   end
   return (result=res;)
end

Then we would wrap this in the standard timing infrastructure g:

function g(f)
   function inner(evals)
      tic = time()
      data = f(evals)
      toc = time()
      return (data..., time=toc-tic)
   end
end

Then, if a user wishes to produce a custom benchmarking function, they might do

function profile(f)
   function inner(evals)
      traces = @profile begin
         data = f(evals)
      end
      return (data..., profile=traces)
   end
end

And at the end of the day, we would have a BenchmarkResult with a namedtuple of attributes, which might include things like cpu counts and sampling profilers. Perhaps we could use a more involved approach using wrapper objects instead of wrapper functions, and call methods like setup(::LinuxProfSampler) or sample(::LinuxProfSampler, evals), or possibly finalize(::LinuxProfSampler, data), I'm not quite sure.

Would this approach be something that would support both profiling and LinuxPerf-ing?

[Zentrik]

No worries for the delay, thanks for the review, you do raise some good points. But anyways I'm fairly busy now so if we're going to move forward with #375 then I'm not going to bother working on this.

Thanks again for all the time you've put into this.