Refactor (GLMakie) render order (experimental)

[ffreyer]

Description

Goals

The goal of this pr is to render scenes one by one in a depth-first order (render scene, then go deeper). This aims to:

1. make scene.clear = true work again (clear a section of the screen for the clearing scene)
2. fix/remove execution order dependence of rendering
3. provide a standard rendering order
4. allow layering based on scene order, e.g. allow scenes to act as backgrounds/overlays to another independent of plot/render object depth values (probably also separate scene.children into normal scenes and overlay scenes to make this usable)
5. maybe provide a transparent intermediate render buffer for saving transparent images, Reviving CairoMakie/GLMakie integration #3652
6. prepare for per-scene post-processing pipelines, which I hope will improve quality and perhaps also performance, and enable for more post-processors to exist in the future.

Related Issues

Related issues:

• fixes issue Fix clearing in GLMakie #2650 (Axis3 ticks in front of menu)
• https://discourse.julialang.org/t/glmakie-overlay-plots-in-animated-3d-scene/116750
• [Roadmap] GLMakie #2995
• Make rendering pipeline more flexible #604
• fixes The (background)color of Box always covers the 3D LScene.scene.backgroundcolor, and the latter cannot be changed #4153 (Box only if Box is used before LScene)

Fixed unreported (?) issues:

Empty scenes do not render if added after display

scene = Scene(backgroundcolor = :white, clear = true)
screen = display(scene)
s1 = Scene(scene, Rect2f(100, 100, 400, 300), backgroundcolor = :cyan, clear = true);
s2 = Scene(scene, Rect2f(50, 150, 100, 100), backgroundcolor = :purple, clear = true);

master	pr

Scene order depends on (plot) insertion order

scene = Scene(backgroundcolor = :white, clear = true)
screen = display(scene)
s1 = Scene(scene, Rect2f(100, 100, 400, 300), backgroundcolor = :cyan, clear = true);
s2 = Scene(scene, Rect2f(50, 150, 100, 100), backgroundcolor = :purple, clear = true);
scatter!(s2, Point2f(0)) # trigger scene 2 insert
scatter!(s1, Point2f(0)) # trigger scene 1 insert

master	pr (skipped plot)

Changes

This pr mainly changes the scene/plot/renderobject storage infrastructure to more closely represent the scene tree from Makie and allow for rendering to follow a depth-first order.

Changes in more detail

On master scenes, plots and renderobjects are managed by the following fields in Screen.

mutable struct Screen
    ...
    screen2scene::Dict{WeakRef, ScreenID}                     # map Scene -> scene ID
    screens::Vector{ScreenArea}                               # (scene ID, scene)
    renderlist::Vector{Tuple{ZIndex, ScreenID, RenderObject}} # (unused, scene ID, robj)
    cache::Dict{UInt64, RenderObject}                         # objectid(plot) -> robj
    cache2plot::Dict{UInt32, AbstractPlot}                    # robj.id -> plot
    ...
end

These are replaced by a scene_tree::GLSceneTree

struct GLSceneTree
    scene2index::Dict{WeakRef, Int}       # scene -> index into scenes
    robj2plot::Dict{UInt32, AbstractPlot} # robj.id -> plot (all robjs)
    scenes::Vector{GLScene}               # scenes in depth first order
    depth::Vector{Int}                    # depth level in tree, for insert!
end

which contains scene representatives holding on to rendering relevant information and the render objects corresponding to the scenes plots.

struct GLScene
    # Rendering Information
    viewport::Observable{Rect2f}  
    clear::Observable{Bool}
    backgroundcolor::Observable{RGBAf}
    visible::Observable{Bool}

    ssao::Makie.SSAO                      # Postprocessor information
    
    renderobjects::Vector{RenderObject}   # all robjs in scene

    plot2robj::Dict{UInt64, RenderObject} # ojectid(plot) -> robj
    robj2plot::Dict{UInt32, Plot}         # robj.id -> plot (local to scene)
end

To keep the GLSceneTree in sync with Makie scenes, they now notify the backend via push!(::Scene, ::Scene) -> insert!(screen, scene, parent, index). The index is the index into scene.children where the scene is inserted. Most of the insertion and deletion logic is in GLScene.jl.

In terms of rendering things look more different than they actually are. Previously we effectively had

# sort 2D robjs by z value
# resize buffers
# setup draw region, clear render buffers (outputs from plot shaders)
# draw scene backgrounds
# render plots/robjs for SSAO, run SSAO post processor
# render remaining non-OIT plots/robjs
# render OIT plots/robjs to different buffer, run OIT post-processor, merge into main buffer
# run FXAA
# copy to screen

And now we have:

# resize all buffers
# clear composition buffer
for glscene in tree.scenes
    # check for early exits/skips
    # setup draw region
    # clear color buffer and objectid buffer as necessary
    if !isempty(glscene.renderobjects)
        # prepare remaining render buffers
        # sort 2D robjs by z value
        # render plots/robjs for SSAO, run SSAO post processor
        # render remaining non-OIT plots/robjs
        # render OIT plots/robjs to different buffer, run OIT post-processor, merge into main buffer
        # run FXAA
    end
    # copy to composition buffer
end
# clear screen, then copy composition buffer to screen

Plots/Renderobjects are still rendered in the same order, except that they are split up by scene. The post processors and buffers have changed a little bit to accommodate a transparent color buffer. Specifically the default blending operation of

dst.rgb = src.a * src.rgb + (1 - src.a) * dst.rgb

has been replaced as it assumes the dst to be opaque. To figure out what it should be we can consider two transparent blends (c1, c2) onto an opaque color (c0) via the above. This gives us:

   c2.a * c2.rgb + (1 - c2.a) * (c1.a * c1.rgb + (1 - c1.a) * c0.rgb)
= c2.a * c2.rgb + (1 - c2.a) * c1.a * c1.rgb  + (1 - c2.a) * (1 - c1.a) * c0.rgb

This mean we want to have the destination color in a dst.a * dst.rgb, (1 - dst.a) format. The blending rules then become:

# normal RGBA source + formatted destination
dst.rgb = src.a * src.rgb + (1 - src.a) * dst.rgb
dst.a = (1 - src.a) * dst.a

# formatted source + formatted destimation
dst.rgb = src.rgb + src.a * dst.rgb
dst.a = src.a * dst.a

# formatted source + opaque destination:
dst.rgb = src.rgb + src.a * dst.rgb
dst.a = 1

TODO

• figure out what to do with FXAA and transparent buffers (merge with background beforehand? edge alpha blur?)
• figure out correct behavior with pre-multiplied alphas in FXAA
• re-evaluate robj2plot handling (global + per-scene vs just global)
• fix GLMakie based test failures (i.e. implementation errors)
• fix conceptual test failures (e.g. render order breaking assumptions in Makie, issues with depth being used across scenes)
• re-evaluate new infrastructure
• add useful utility functions to new infrastructure (e.g. haskey, get, isempty, etc)
• probably add overlay separation for scene.children
• document render order, new functionality & update old documentation
• evaluate & improve performance
• test picking
• consider reordering post processors for better visuals
• consider adding colorbuffer(screen, scene)
• consider merging OIT buffer with temp color buffer (i.e. commit to composition buffer before OIT, clear, reuse)
• preserve color buffer with clear = false and consider also preserving depth

For the future

Some things I want to change but probably not in this pr so that it does not become overwhelmingly big

• update Makie internals to rely on scene order and clearing over translate! when possible but not required
• analyze scene visibility based on clear and use that to skip or stencil-out hidden areas
• try to make post-processing pipeline controllable from Makie Scenes and optimize based on context

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

• Added an entry in CHANGELOG.md (for new features and breaking changes)
• Added or changed relevant sections in the documentation
• Added unit tests for new algorithms, conversion methods, etc.
• Added reference image tests for new plotting functions, recipes, visual options, etc.

[MakieBot]

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(fig)

	using	create	display	create	display
GLMakie	4.47s (4.45, 4.49) 0.02+-	108.01ms (107.06, 110.07) 1.02+-	606.31ms (554.84, 625.56) 24.23+-	9.10ms (8.14, 9.58) 0.49+-	72.34ms (72.07, 72.75) 0.23+-
master	4.44s (4.43, 4.46) 0.01+-	107.26ms (105.91, 108.90) 0.97+-	584.44ms (548.88, 782.58) 87.39+-	9.56ms (9.48, 9.62) 0.05+-	26.02ms (25.92, 26.19) 0.11+-
evaluation	0.99x slower X, 0.03s (2.00d, 0.00p, 0.01std)	0.99x invariant, 0.76ms (0.76d, 0.18p, 0.99std)	0.96x invariant, 21.87ms (0.34d, 0.54p, 55.81std)	1.05x faster✅, -0.46ms (-1.32d, 0.05p, 0.27std)	0.36x slower❌, 46.32ms (253.58d, 0.00p, 0.17std)
CairoMakie	3.71s (3.68, 3.75) 0.02+-	105.44ms (104.38, 106.19) 0.63+-	165.55ms (163.87, 168.08) 1.28+-	8.93ms (8.44, 9.29) 0.36+-	1.13ms (1.12, 1.14) 0.01+-
master	3.92s (3.88, 4.00) 0.04+-	105.60ms (104.61, 106.88) 0.73+-	164.72ms (163.52, 166.70) 1.04+-	9.20ms (8.82, 9.36) 0.18+-	1.13ms (1.12, 1.14) 0.01+-
evaluation	1.06x faster✅, -0.21s (-6.33d, 0.00p, 0.03std)	1.00x invariant, -0.16ms (-0.24d, 0.67p, 0.68std)	0.99x invariant, 0.83ms (0.71d, 0.21p, 1.16std)	1.03x invariant, -0.28ms (-0.99d, 0.10p, 0.27std)	1.00x invariant, 0.0ms (0.15d, 0.78p, 0.01std)
WGLMakie	4.65s (4.62, 4.68) 0.02+-	107.40ms (106.81, 108.44) 0.63+-	9.25s (9.18, 9.38) 0.07+-	11.09ms (10.99, 11.24) 0.10+-	118.87ms (116.69, 122.01) 1.76+-
master	4.55s (4.52, 4.57) 0.02+-	107.08ms (105.20, 110.99) 1.89+-	9.11s (8.91, 9.22) 0.11+-	11.13ms (10.97, 11.37) 0.13+-	118.49ms (114.00, 122.07) 2.47+-
evaluation	0.98x slower X, 0.11s (5.18d, 0.00p, 0.02std)	1.00x invariant, 0.32ms (0.23d, 0.69p, 1.26std)	0.98x slower X, 0.14s (1.55d, 0.02p, 0.09std)	1.00x invariant, -0.04ms (-0.34d, 0.54p, 0.11std)	1.00x invariant, 0.38ms (0.18d, 0.75p, 2.12std)

[asinghvi17]

Does this mean that GLMakie can support transparent backgrounds? Or would that need improvements to the shader / OIT?

[ffreyer]

In saved images with some more fixes yes, more or less

begin
    scene = Scene(backgroundcolor = (:orange, 0.3), clear = true)
    s1 = Scene(scene, Rect2f(100, 100, 100, 100), backgroundcolor = (:cyan, 0.3), clear = true);
    s2 = Scene(scene, Rect2f(300, 100, 100, 100), backgroundcolor = (:purple, 0.3), clear = true);
    scatter!(scene, rand(10), rand(10))
    screen = display(scene)
end

The main thing that's clashing with transparent backgrounds here is FXAA. It requires opaque pixels to do it's thing which you don't have in a transparent render. And also not in a transparent subscene.
Another annoyance is that you can't work with the usual (c.a * c.rgb + (1-c.a) * background.rgb), 1) blending because it assumes the background is opaque. I think with two transparent colors you have to do (c1.a * c1.rgb + (1-c1.a) * c2.a * c2.rgb, (1-c1.a) * (1-c2.a)) and do the final background blend as (composed.rgb + composed.a * background.rgb, 1).
But OIT doesn't care. Plot rendering doesn't care. SSAO I haven't tested but it also shouldn't care.
(Transparent windows I assume aren't possible from a GLFW/OS side but I haven't checked.)

[asinghvi17]

I see, I guess we could always turn FXAA off in the short term though.

What I meant was having a completely transparent background - at which point I guess you don't need a final blend. (Motivation was finally finishing that CairoMakie PR that allows rendering from other backends natively)

glfw/glfw#197 seems to indicate that GLFW supports transparent framebuffers and windows via a window hint.

[asinghvi17]

Can this be returned by colorbuffer as well somehow? Might be useful for people who want more precise renders / colors...

[ffreyer]

Not right now. I'm also not sure what you mean with precise renders...
As it is now, every scene effectively clears a section of this buffer, then does all the rendering on it and runs all the post processors with it. So it's the same as colorbuffer(detached_sub_scene) after that scene is done rendering, except with a bunch of old junk around it. At the end of a frame it maybe overwritten by another scene.
I could add a colorbuffer(screen, scene) that only renders a selected scene if that's what you want.

[asinghvi17]

The colorbuffer(screen, scene) thing is exactly what I need for CairoMakie :D since cropping the scene by its bbox also includes axis artifacts, etc.

[asinghvi17]

I see, I guess we could always turn FXAA off in the short term though.

What I meant was having a completely transparent background - at which point I guess you don't need a final blend. (Motivation was finally finishing that CairoMakie PR that allows rendering from other backends natively)

glfw/glfw#197 seems to indicate that GLFW supports transparent framebuffers and windows via a window hint.

[MakieBot]

Benchmark Results

SHA: 99f7f36c1aba9c45648e66df3c67d7adb7ad6bc9

Warning

These results are subject to substantial noise because GitHub's CI runs on shared machines that are not ideally suited for benchmarking.