x11_common: fix window mapping, refactor window creation/resize handling

vo_opengl creates an invisible window with VOFLAG_HIDDEN in order to
test whether the OpenGL context is useable. The visible window is
created at a later point. This has been broken forever (in vo_gl,
now called vo_opengl_old, it could be avoided by disabling auto-
detection explicitly using the "yuv" sub-option). Avoiding
VOFLAG_HIDDEN only mitigates the issue, and a related bug can still
happen with some window managers (see below).

As a hack, code was added to vo_gl to destroy the (hidden) window so
that all state was destroyed. Later, more hacks were added to deal with
some caveats that came with recreating the window, such as probing for
the context up to 4 times.

Attempt to fix the problem properly. There were two problems: first,
the window was not resized to video size before mapping. This was the
main cause for the placement issue, e.g. mapping the window as 320x200,
and then resizing it. Second, mpv tried to force the window position
with XSetWMNormalHints and PPosition with values it just read with
XGetGeometry. This messes up the window manager's default placement.
It seems to be a race condition, and behavior is different across
various WMs too: with IceWM, the window manager's placement is usually
preferred, and with Fluxbox, mpv's position is preferred. mpv's default
position is centering the window on the screen, which is much nicer for
video in general than typical WM default placement, so it's possible
that this bug was perceived as a feature. (Users who want this have to
use --geometry="50%:50%", doing this by default is probably not safe
with all WMs.)

Since the old code was hard to follow and full of issues, it's easier
to redo it. Move general window creation stuff out of the
vo_x11_config_vo_window function, and move the resize logic into it.

This has been tested on IcwWM, Fluxbox, awesome, Unity/Compiz.

1 files changed, 12 insertions, 9 deletions

diff --git a/video/out/x11_common.h b/video/out/x11_common.h
index 630d576b77..c54404fb7f 100644
--- a/video/out/x11_common.h
+++ b/video/out/x11_common.h
@@ -44,25 +44,21 @@ struct vo_x11_state {
     XIM xim;
     XIC xic;
 
-    GC vo_gc;
+    GC f_gc;    // used to paint background
+    GC vo_gc;   // used to paint video
     Colormap colormap;
 
     int wm_type;
     int fs_type;
-    int window_state;
+    bool window_hidden;
     int fs_flip;
     int fs_layer;
 
-    GC f_gc;
     XSizeHints vo_hint;
     unsigned int mouse_timer;
     int mouse_waiting_hide;
     int orig_layer;
     int old_gravity;
-    int vo_old_x;
-    int vo_old_y;
-    int vo_old_width;
-    int vo_old_height;
 
     // Current actual window position (updated on window move/resize events).
     int win_x;
@@ -72,17 +68,24 @@ struct vo_x11_state {
 
     int pending_vo_events;
 
+    // last non-fullscreen extends (updated on fullscreen or reinitialization)
+    int nofs_width;
+    int nofs_height;
+    int nofs_x;
+    int nofs_y;
+
     /* Keep track of original video width/height to determine when to
      * resize window when reconfiguring. Resize window when video size
      * changes, but don't force window size changes as long as video size
      * stays the same (even if that size is different from the current
      * window size after the user modified the latter). */
-    int last_video_width;
-    int last_video_height;
+    int old_dwidth;
+    int old_dheight;
     /* Video size changed during fullscreen when we couldn't tell the new
      * size to the window manager. Must set window size when turning
      * fullscreen off. */
     bool size_changed_during_fs;
+    bool pos_changed_during_fs;
 
     unsigned int olddecor;
     unsigned int oldfuncs;