x11: work around mutter fullscreen issue

If the video has the same size as the screen, starting with --fs and
then leaving fullscreen doesn't actually leave fullscreen.

The reason is that mpv tries to restore the previous window size if
necessary (otherwise, you'd end up with a Window of nearly the same size
as the screen with some WMs). It will typically restore with the
rectangle set exactly to the screen if no other position or size is
forced. This triggers pre-EWMH fullscreen mode, which WMs detect using
various heuristics.

Apparently we triggered this with mutter (but strangely no other WMs).
It's possible that pre-EWMH fullscreen mode actually requires removing
decorations, and mutter either ignores this. But this is speculation and
I haven't checked.

Work this around by reducing the requested size by 1 pixel if it
happens.

This was observed with mutter 3.18.2.

Fixes #2072.

1 files changed, 11 insertions, 3 deletions

diff --git a/video/out/x11_common.c b/video/out/x11_common.c
index a9d6a2a632..1b780598af 100644
--- a/video/out/x11_common.c
+++ b/video/out/x11_common.c
@@ -1748,17 +1748,25 @@ static void vo_x11_fullscreen(struct vo *vo)
         x11->nofsrc = x11->winrc;
     }
 
+    struct mp_rect rc = x11->nofsrc;
+
     if (x11->wm_type & vo_wm_FULLSCREEN) {
         x11_set_ewmh_state(x11, "_NET_WM_STATE_FULLSCREEN", x11->fs);
         if (!x11->fs && (x11->pos_changed_during_fs ||
                          x11->size_changed_during_fs))
         {
+            if (x11->screenrc.x0 == rc.x0 && x11->screenrc.x1 == rc.x1 &&
+                x11->screenrc.y0 == rc.y0 && x11->screenrc.y1 == rc.y1)
+            {
+                // Workaround for some WMs switching back to FS in this case.
+                MP_VERBOSE(x11, "avoiding triggering old-style fullscreen\n");
+                rc.x1 -= 1;
+                rc.y1 -= 1;
+            }
             vo_x11_move_resize(vo, x11->pos_changed_during_fs,
-                                   x11->size_changed_during_fs,
-                                   x11->nofsrc);
+                                   x11->size_changed_during_fs, rc);
         }
     } else {
-        struct mp_rect rc = x11->nofsrc;
         if (x11->fs) {
             vo_x11_update_screeninfo(vo);
             rc = x11->screenrc;