Per Pixel Transparency / DisplayServer.window_set_mode sets the background to transparent.

[Alex2782]

Godot version

v4.2.dev6.official [57a6813]

System information

Godot v4.2.dev6 - macOS 13.6.0 - Vulkan (Forward+) - integrated Apple M1 - Apple M1 (8 Threads)

Issue description

Status: the demo is fully ported and working.
godotengine/godot-demo-projects#697

Renderer: Forward+ and Mobile. With 'Compatibility' everything seems to be ok.
I have already tried other 2D and 3D demos in the last weeks, such graphical problems did not occur there.

Note: when I use the system buttons to maximize the window (fullscreen), the background does not become transparent.

Screen-2023-10-11-011133.mp4

Debug: get_viewport().transparent_bg = false

func _on_Button_Fullscreen_pressed():

	print(get_viewport().transparent_bg)
	get_viewport().transparent_bg = false

	if DisplayServer.window_get_mode() == DisplayServer.WINDOW_MODE_FULLSCREEN:
		DisplayServer.window_set_mode(DisplayServer.WINDOW_MODE_WINDOWED)
	else:
		DisplayServer.window_set_mode(DisplayServer.WINDOW_MODE_FULLSCREEN)

	print(get_viewport().transparent_bg)

output:

false
false

Steps to reproduce

1. open and start: window_management demo
2. push the Fullscreen button
3. push the Fullscreen button again

Minimal reproduction project

window_management demo

[Calinou]

I can reproduce this in 4.2.dev4 and later (but not in 4.2.dev2 and earlier) on Linux.

I haven't managed to bisect this as the project fails to render in 4.2.dev3 (black screen for 3D rendering). A minimal project without 3D rendering doesn't exhibit the issue.

My bisect left me on 715ebcc which I'm not sure is the cause.

[Alex2782]

Per Pixel Transparency

if disabled, there are no more problems with 'Fullscreen', but then the background is black when 'Transparent' get_viewport().transparent_bg is enabled.

[akien-mga]

CC @bruvzg

[akien-mga]

I can't reproduce the issue on my Linux system, though Per Pixel Transparency doesn't seem to work at all for me, neither in 4.1.2 nor 4.2-beta4.

I do see a visual difference in this demo between 4.2-beta4 (top left) and 4.1.2 (bottom right):

$ inxi -CSG
System:
  Host: cauldron Kernel: 6.4.16-desktop-3.mga9 arch: x86_64 bits: 64
    Desktop: KDE Plasma v: 5.27.5 Distro: Mageia 9
CPU:
  Info: quad core model: Intel Core i7-8705G bits: 64 type: MT MCP cache:
    L2: 1024 KiB
  Speed (MHz): avg: 900 min/max: 800/4100 cores: 1: 900 2: 900 3: 900 4: 900
    5: 900 6: 900 7: 900 8: 900
Graphics:
  Device-1: Intel HD Graphics 630 driver: i915 v: kernel
  Device-2: AMD Polaris 22 XL [Radeon RX Vega M GL] driver: amdgpu v: kernel
  Device-3: Cheng Uei Precision Industry (Foxlink) HP Wide Vision FHD Camera
    type: USB driver: uvcvideo
  Display: x11 server: X.org v: 1.21.1.8 with: Xwayland v: 22.1.9 driver: X:
    loaded: intel,v4l dri: i965 gpu: i915 resolution: 2048x1152~60Hz
  API: OpenGL v: 4.6 Mesa 23.1.7 renderer: Mesa Intel HD Graphics 630 (KBL
    GT2)

[Alex2782]

Per Pixel Transparency active or deactivated? @akien-mga
if I disable it, then it looks like "top left", dark gray background and no graphic errors. (v4.2.beta2)

if "Per Pixel Transparency" is disabled and also "Transparent", then the background is black.

[akien-mga]

It was activated, I tested both the window management demo, and a new project with this setting + Transparent enabled.
For me the background is always black when transparent is enabled. This is likely a separate issue, I'll open a dedicated one to track this.

[Calinou]

I can't reproduce the issue on my Linux system, though Per Pixel Transparency doesn't seem to work at all for me, neither in 4.1.2 nor 4.2-beta4.

Is compositing enabled in your window manager? Window transparency requires it to be enabled (try Alt + Shift + F12 in KWin).

[akien-mga]

Is compositing enabled in your window manager? Window transparency requires it to be enabled (try Alt + Shift + F12 in KWin).

Good call, that was the issue. After enabling compositing with Alt+Shift+F12, I can reproduce the issue.

For me the bug doesn't even require enabling fullscreen, the demo is wrongly transparent out of the box, in a semi-transparent manner.

For each version tested, I show screenshots of the windowed mode with transparency off, then on.

In 4.1.3-stable:

In 4.2-dev1 and dev2:

In 4.2-dev3, rendering seems broken:

4.2-dev4 and later:

[akien-mga]

I can reproduce this in 4.2.dev4 and later (but not in 4.2.dev2 and earlier) on Linux.

I haven't managed to bisect this as the project fails to render in 4.2.dev3 (black screen for 3D rendering). A minimal project without 3D rendering doesn't exhibit the issue.

My bisect left me on 715ebcc which I'm not sure is the cause.

So I had a look too, 715ebcc is the first commit that exhibits the broken transparency because it's the first commit that fixes rendering. Before it, the project fails to render as you point out.

So it hides the real culprit. #80502 was a fixup to #80311, so I tried a local revert of both PRs, and that solves the black screen rendering, but still exhibits the broken transparency:

That's with current master plus the above two PRs reverted in reverse chronological order.

So that means they're not related to the bug, the real culprit must be another commit between 4.2.dev2 and 715ebcc.

To bisect further, you'd have to apply 715ebcc manually whenever building a commit that includes #80311 but not #80502.

[bruvzg]

For me the bug doesn't even require enabling fullscreen, the demo is wrongly transparent out of the box, in a semi-transparent manner.

I see what's wrong with macOS code, but not sure about Linux. I think there's no way to switch transparency for the window except at window creation time on X11. If transparent_bg is not set, it's something wrong with the renderer.

[akien-mga]

Found the culprit in the interactive changelog for dev3, reverting #79876 fixes the issue @Calinou and I described for Linux. It might be a different problem on macOS.

[clayjohn]

#79876 just exposes the problem. Before it transparent_bg was totally broken and did nothing at all.

It seems like the problem is further up the pipeline. I.e. for some reason per-pixel transparency is force enabling transparent_bg in the viewport.

What is the intended user experience for per-pixel window transparency?

[akien-mga]

Worth noting that #80933 is also involved, to revert #79876 on top of current master you also need to revert #80933 first.

If I revert only #79876 and keep the inverse luminance multiplier from #80933, I still get a bug:

i.e. with this diff:

commit a72ca646a918951bdb81e4c4325e7b9f8caf60b8
Author: Rémi Verschelde <[email protected]>
Date:   Thu Nov 9 22:19:00 2023 +0100

    Revert "Fix transparent viewport backgrounds with custom clear color"
    
    This reverts commit 6effd3cde7a481b57226cf5d03c97aa5728ff7e7.

diff --git a/servers/rendering/renderer_rd/forward_mobile/render_forward_mobile.cpp b/servers/rendering/renderer_rd/forward_mobile/render_forward_mobile.cpp
index 462fc4b524..71410bd752 100644
--- a/servers/rendering/renderer_rd/forward_mobile/render_forward_mobile.cpp
+++ b/servers/rendering/renderer_rd/forward_mobile/render_forward_mobile.cpp
@@ -931,19 +931,13 @@ void RenderForwardMobile::_render_scene(RenderDataRD *p_render_data, const Color
 		{
 			// regular forward for now
 			Vector<Color> c;
-			{
-				Color cc = clear_color.srgb_to_linear() * inverse_luminance_multiplier;
-				if (rb_data.is_valid()) {
-					cc.a = 0; // For transparent viewport backgrounds.
+			c.push_back(clear_color.srgb_to_linear() * inverse_luminance_multiplier); // our render buffer
+			if (rb_data.is_valid()) {
+				if (p_render_data->render_buffers->get_msaa_3d() != RS::VIEWPORT_MSAA_DISABLED) {
+					c.push_back(clear_color.srgb_to_linear() * inverse_luminance_multiplier); // our resolve buffer
 				}
-				c.push_back(cc); // Our render buffer.
-				if (rb_data.is_valid()) {
-					if (p_render_data->render_buffers->get_msaa_3d() != RS::VIEWPORT_MSAA_DISABLED) {
-						c.push_back(clear_color.srgb_to_linear() * inverse_luminance_multiplier); // Our resolve buffer.
-					}
-					if (using_subpass_post_process) {
-						c.push_back(Color()); // Our 2D buffer we're copying into.
-					}
+				if (using_subpass_post_process) {
+					c.push_back(Color()); // our 2D buffer we're copying into
 				}
 			}
 

[clayjohn]

I think I know what is going on on the rendering side now. It seems to be a duplicate of #61969 Which is caused by #61109. We write to alpha even when using opaque shaders, but alpha should be ignored when we copy the 3D buffer to the 2D buffer unless using transparent background.

In 3.x we solved this with an IGNORE_ALPHA flag in the post process shader #61382

[clayjohn]

Submitted a fix for the rendering side. Indeed the RD renderers were totally broken in multiple ways. #79876 made transparent background work, but it wasn't a full solution and wasn't fine grained enough.

The fix needs some testing in more complex projects, but works in the MRP here and in the MRP from #61969

[akien-mga]

The macOS part (original bug report) is fixed by #84683, so we can close this.

The Linux regression (not sure why it wouldn't affect other platforms, maybe it does) will be fixed by #84684, but it's a bit late to finalize it for 4.2 so that will likely be merged for 4.3 and cherry-picked for a 4.2.x release.