lua: makenode: prevent lua stack corruption

Normally there was no issue, but when the code converted a deeply
nested table into an mpv node - it didn't ensure the stack has room.

Lua doesn't check stack overflow when invoking lua_push* functions,
and leaves this responsibility to the (c) user via lua_checkstack.

Normally that's not an issue because when a lua (or autofree) function
is called, it's guaranteed at least LUA_MINSTACK (20) pushes.

However, pushnode and makenode are recursive, and each iteration can
add few values at the stack (which are popped when the recursion
unwinds), so checkstack must be used on (recursive) entry.

pushnode already checked the stack, makenode did not.

This commit checks the stack at makenode as well. The value of 6
(stack places to reserve) is with some room to spare, and in pratice
each iteration needs 2-3 at most (pushnode also leaves room).

Example which could previously corrupt the stack:
  utils.format_json({d1={d2={<8 more times>}}}

This uses makenode to convert the lua table into an mpv node which
the json writer uses as input, and if the depth is 10 or more then
corruption could occur. mp.command_native is also affected, as well as
any other mp/utils command which takes a lua table as input.

While at it, fix the error string which pushnode used (luaL_checkstack
uses the provided string with "Stack overflow (%s)", so the user
message only needs to be additional info).

1 files changed, 3 insertions, 1 deletions

diff --git a/player/lua.c b/player/lua.c
index 83f4df3ec1..bdb08e304a 100644
--- a/player/lua.c
+++ b/player/lua.c
@@ -670,6 +670,8 @@ static int script_set_property_number(lua_State *L)
 
 static void makenode(void *tmp, mpv_node *dst, lua_State *L, int t)
 {
+    luaL_checkstack(L, 6, "makenode");
+
     if (t < 0)
         t = lua_gettop(L) + (t + 1);
     switch (lua_type(L, t)) {
@@ -869,7 +871,7 @@ static int script_get_property_number(lua_State *L)
 
 static void pushnode(lua_State *L, mpv_node *node)
 {
-    luaL_checkstack(L, 6, "stack overflow");
+    luaL_checkstack(L, 6, "pushnode");
 
     switch (node->format) {
     case MPV_FORMAT_STRING: