vo_opengl: Simplify and clarify color correction code

This commit:

- Changes some of the #define and variable names for clarification and
  adds comments where appropriate.
- Unifies :srgb and :icc-profile, making them fit into the same step of
  the decoding process and removing the weird interactions between both
  of them.
- Makes :icc-profile take precedence over :srgb (to significantly reduce
  the number of confusing and useless special cases)
- Moves BT709 decompanding (approximate or actual) to the shader in all
  cases, making it happen before upscaling (instead of the old 0.45
  gamma function). This is the simpler and more proper way to do it.
- Enables the approx gamma function to work with :srgb as well due to
  this (since they now share the gamma expansion code).
- Renames :icc-approx-gamma to :approx-gamma since it is no longer tied
  to the ICC options or LittleCMS.
- Uses gamma 2.4 as input space for the actual 3DLUT, this is now a
  pretty arbitrary factor but I picked 2.4 mainly because a higher pure
  power value here seems to produce visually better results with wide
  gamut profiles, rather then the previous 1.95 or BT.709.
- Adds the input gamma space to the 3dlut cache header in case we change
  it more in the future, or even make it user customizable (though I
  don't see why the latter would really be necessary).
- Fixes the OSD's gamma when using :srgb, which was previously still
  using the old (0.45) approximation in all cases.
- Updates documentation on :srgb, it was still mentioning the old
  behavior from circa a year ago.

This commit should serve to both open up and make the CMS/shader code much
more accessible and less confusing/error-prone and simultaneously also
improve the performance of 3DLUTs with wide gamut color spaces.

I would liked to have made it more modular but almost all of these
changes are interdependent, save for the documentation updates.

Note: Right now, the "3DLUT takes precedence over SRGB" logic is just
coded into gl_lcms.c's compile_shaders function. Ideally, this should be
done earlier, when parsing the options (by overriding the actual
opts.srgb flag) and output a warning to the user.

Note: I'm not sure how well this works together with real-world
subtitles that may need to be color corrected as well. I'm not sure
whether :approx-gamma needs to apply to subtitles as well. I'll need to
test this on proper files later.

Note: As of now, linear light scaling is still intrinsically tied to
either :srgb or :icc-profile. It would be thinkable to have this as an
extra option, :linear-scaling or similar, that could be used with or
without the two color management options.

1 files changed, 27 insertions, 14 deletions

diff --git a/video/out/gl_video_shaders.glsl b/video/out/gl_video_shaders.glsl
index cd7e051dcd..15bb600e61 100644
--- a/video/out/gl_video_shaders.glsl
+++ b/video/out/gl_video_shaders.glsl
@@ -82,11 +82,15 @@ void main() {
     color = vertex_color;
 
 #ifdef USE_OSD_LINEAR_CONV
-    // If no 3dlut is being used, we need to pull up to linear light for
-    // the sRGB function. *IF* 3dlut is used, we do not.
+    // Although we are not scaling in linear light, both 3DLUT and SRGB still
+    // operate on linear light inputs so we have to convert to it before
+    // either step can be applied.
     color.rgb = bt709_expand(color.rgb);
+    // NOTE: This always applies the true BT709, maybe we need to use
+    // approx-gamma here too?
 #endif
 #ifdef USE_OSD_3DLUT
+    color.rgb = pow(color.rgb, vec3(1/2.4)); // linear -> 2.4 3DLUT space
     color = vec4(texture3D(lut_3d, color.rgb).rgb, color.a);
 #endif
 #ifdef USE_OSD_SRGB
@@ -369,35 +373,44 @@ void main() {
     color.gb = vec2(128.0/255.0);
 #endif
 #ifdef USE_INPUT_GAMMA
+    // Pre-colormatrix input gamma correction (eg. for MP_IMGFLAG_XYZ)
     color = pow(color, vec3(input_gamma));
 #endif
 #ifdef USE_COLORMATRIX
+    // Conversion from Y'CbCr or other spaces to RGB
     color = mat3(colormatrix) * color + colormatrix[3];
     color = clamp(color, 0, 1);
 #endif
 #ifdef USE_CONV_GAMMA
+    // Post-colormatrix converted gamma correction (eg. for MP_IMGFLAG_XYZ)
     color = pow(color, vec3(conv_gamma));
 #endif
-#ifdef USE_LINEAR_CONV_INV
-    // Convert from linear RGB to gamma RGB before putting it through the 3D-LUT
-    // in the final stage.
-    color = pow(color, vec3(0.45));
+#ifdef USE_LINEAR_LIGHT
+    // If we are scaling in linear light (SRGB or 3DLUT option enabled), we
+    // expand our source colors before scaling
+#ifdef USE_APPROX_GAMMA
+    // We differentiate between approximate BT.709 (gamma 1.95) ...
+    color = pow(color, vec3(1.95));
+#else
+    // ... and actual BT709 (two-part function)
+    color = bt709_expand(color);
+#endif
 #endif
+    // Image upscaling happens roughly here
 #ifdef USE_GAMMA_POW
+    // User-defined gamma correction factor (via the gamma sub-option)
     color = pow(color, inv_gamma);
 #endif
 #ifdef USE_3DLUT
+    // For the 3DLUT we are arbitrarily using 2.4 as input gamma to reduce
+    // the amount of rounding errors, so we pull up to that space first and
+    // then pass it through the 3D texture.
+    color = pow(color, vec3(1/2.4));
     color = texture3D(lut_3d, color).rgb;
 #endif
 #ifdef USE_SRGB
-    // Go from "linear" (0.45) to BT.709 to true linear to sRGB
-#ifndef USE_3DLUT
-    // Unless we are using a 3DLUT, in which case USE_LINEAR_CONV_INV
-    // already triggered earlier.
-    color = pow(color, vec3(0.45));
-#endif
-    color = bt709_expand(color.rgb);
-    color.rgb = srgb_compand(color.rgb);
+    // Compand from the linear scaling gamma to the sRGB output gamma
+    color = srgb_compand(color.rgb);
 #endif
 #ifdef USE_DITHER
     vec2 dither_pos = gl_FragCoord.xy / dither_size;