Everything You Didn’t Want to Know About Lua’s Multi-Values

I’ve been spending a lot of time lately writing Fennel and contributing to its compiler. Since it’s a language that compiles to Lua, that also means spending a lot of time getting to know Lua. This article will be focused on Lua (though I’ll address Fennel from time to time in the footnotes).

Lua is a neat, elegant, and relatively simple language. I find it particularly notable for its embeddability into other programs; capability of hosting many different styles of programming; and the excellent performance of one of its implementations, LuaJIT. In many ways, it feels like a more elegant and restrained JavaScript - it supports tail call elimination^[1] and coroutines, it doesn’t have the confusing var/let distinction (though it lacked a feature like const until Lua 5.4), and it doesn’t privilege class syntax over other styles of programming (object oriented programming can be done in Lua with metatables, but it’s not given special syntax like it is in JS).

That said, it has aspects that are undoubtedly quirky - arrays and dictionaries are unified into "tables", whose integer indices start at 1; the standard library is unusually minimal, a nod to its embeddability; it, like JavaScript, makes the immense error of making variables global by default and local only when you ask. Curiously, it also supports a quite uncommon language feature: multiple return values (which we’ll call “multivals” for brevity’s sake). It is this last subject that we’ll be unpacking more about today.

Since Lua is a dynamic language (as opposed to, say, Go) multiple returns are implemented in a rather odd way. This isn’t explored in as much detail as I’d like in Lua’s documentation, so I wanted to provide a one-stop resource that explains everything I can think to write down about this idiosyncratic part of the language.

A spoiler: unfortunately, multivals will turn out to be somewhat underwhelming for use in code you care about. They’re awkward to work with, have numerous gotchas, are a maintenance hazard, and don’t perform notably better (that I know of) than their primary alternative, tables. This is not a very useful article, unless you have a use for knowing minutiae about Lua’s behavior.

Let’s dive in!

An Introduction to Multivals - 1, 2, 3, unpack, and ...

There are three primary ways to represent a multival in Lua. Each of these has their own distinct uses. Each of the following code snippets produces the same multival:

• A multival literal: 1, 2, 3. (Note that this does not include table literals like {1, 2, 3}).
• Unpacking a table: table.unpack({1, 2, 3})
• The vararg, ..., in a function: local function x(...) return ... end x(1, 2, 3)

Multivals can be packed into a table by simply calling them:

local function two_values() return "first", "second" end
local tab = {two_values()}
print(tab[1], tab[2])
  -- prints "first second"

To manipulate multivals, we can use two techniques: function signatures and select:

local function multival_first(x, ...) return x end
print(multival_first(1, 2, 3))
  -- prints "1"

local function multival_rest(x, ...) return ... end
print(multival_rest(1, 2, 3))
  -- prints "2 3"

print(select(2, 1, 2, 3))
  -- prints "2 3"; identical to the last line

In order to get values from the end of a multival, we have three options. The first is recursion:

local function increment_each_value(first, ...)
  if first then
    return first + 1, increment_each_value(...)
  end
end

print(increment_each_value(1, 2, 3))
  -- prints "2 3 4"

The second is packing the multival into a table. There are two ways to do this, both of which are demonstrated below:

-- 1. You can pack a multival directly into a table literal
local function pack_multival(...)
  return {...}
end

-- Lua's default printing for tables only shows identity; we'll just
-- print the length instead.
print(#pack_multival(1, 2, 3))
  -- prints "3"

-- 2. You can use table.pack, which is built in to Lua 5.2 and later
local tab = table.pack(1, 2, 3)

print(#tab)
  -- prints "3"

-- table.pack also sets the "n" field on the table it returns. This is
-- more efficient than using #tab, which iterates through the table.
print(tab.n)
  -- prints "3"

The final way of getting values from the end of a multival is select combined with select("#", ...) (which we’ll dig into more later on):

local function get_end_of_multival(...)
  local count = select('#', ...)
  return select(count, ...)
end

print(get_end_of_multival(1, 2, 3, 4, 5))
  -- prints "5"

The Vararg: A Second-Class Value

When you start working with multivals, they might seem pretty straightforward. As long as you’re working on something that uses a list with fixed length or that can be expressed by iterating through the head and tail of a list, it seems like working with multivals should be nice and predictable.

The first issue you’ll likely run into if you try to use multivals heavily comes from expecting them to work the same way the rest of Lua does. In short, Lua’s local variables use what’s called lexical scope. This means that the scope of a variable definition is the block of code that you see it contained in in your source code file. It doesn’t matter what order other functions are called in, or whether they’re called at all - the definitions of local variables can be determined just by looking at the structure of the code. This is as opposed to dynamic scope, where the variables defined at a given point are dependent on the runtime of the program itself. Lua’s global variables effectively use dynamic scope.

As discussed above, there’s only three ways to refer to a multival: calling a function for its return values, literally inserting multiple values separated by commas, or using the vararg symbol ... as a function argument. It’s this last case that we’re interested in.

When you use ... in the signature of a function, you must place it at the end of the argument list. (This is what prevents you - syntactically, at least - from accessing the end of ... without either assigning the whole thing to individual variables, recursing through it, or packing it into a table.)

You also can’t assign the whole thing to a variable, because assigning a ... to a variable means unpacking the multival and assigning the first value to the variable - or multiple values, if that’s specified. For example:

-- Assigning the vararg to a single variable will assign that variable
-- to the _first value_ of the vararg.
local function try_to_assign_vararg(...)
  local x = ...
  return x
end

print(try_to_assign_vararg(1, 2, 3))
  -- prints "1"

There’s one additional rule about the vararg, however, that’s very unlike the behavior of the rest of Lua. You can only use a vararg within the function where it is defined. This prevents you from, for instance, saving the vararg into a closure:

local function try_to_return_vararg_from_closure(...)
  return function()
    return ...
  end
end

This fails to run with the following error: cannot use '...' outside a vararg function near '...'. This means that, while you can use ... in two nested functions, each ... can only refer to the vararg of the function that’s currently running. You can’t capture ... within a closure to persist it. Effectively, ... is not lexically scoped, but rather a dynamically scoped variable with certain extra rules like not being reassignable and not being usable outside a function that defined it.

Another way to put this would be to say that Lua’s multivals are “second class” values. You might be familiar with languages that have second class functions, which can’t be assigned to variables or passed as a parameter to a functions. This is similar, with the notable exception that we can pass the vararg to another function. We just can’t save it in a variable, save it in a closure, or manipulate it in certain ways.

Cutoff Multivals

Passing multivals to functions and packing them into tables is pretty straightforward, but there’s one major gotcha about it we haven’t gone over yet. Take a look at the following example:

local function returns_three_values()
  return 1, 2, 3
end

print(returns_three_values())
  -- prints "1 2 3"
print(#{returns_three_values()})
  -- prints "3"

print(returns_three_values(), 4, 5)
  -- prints "1 4 5"
print(#{returns_three_values(), 4 5})
  -- prints "3"

As the second pair of print expressions demonstrates, a function call can only return multiple values into a multival if it is the last thing in the multival. Following a function call in a multival with any other value, even nil, will cut off its return values at the first item.

This also applies to the vararg, ..., in function definitions:

local function f(...)
  return ..., 4, 5
end

print(f(1, 2, 3))
  -- prints "1 4 5"

Lua’s Most Jarring Feature: select("#", ..)

One of Lua’s nicer properties is how a key set to nil and a non-existent key are indistinguishable in a table. In Javascript, for instance, you have both obj.x = undefined to set a property to undefined and delete obj.x to actually remove it from the object. (There’s also obj.x = null, but let’s ignore that for now.) In Lua, there’s just tab.x = nil, and there’s no way to distinguish between a property that was set to nil and a property that was never set at all.

With that in mind, let’s look at the following example:

local y = {nil,nil,nil}

print(#y)
   -- prints "0" - nils at the end of a table don't matter

local function multival_length(...) select("#", ...) end

print(multival_length(table.unpack(y)))
  -- prints "0"

print(multival_length(nil, nil, nil))
  -- prints "3". wtf?

As this example demonstrates, multivals throw that nice property out the window. Just to be clear:

• There is no distinction between {nil, nil} and {} in Lua.
• There is one distinction between multival_length(nil, nil) and multival_length() in Lua. That distinction is made byselect("#", ...).

This becomes even more confusing when combined with a function that returns zero values, as opposed to nil:

print(select("#", print("a"), print("b"), print("c")))
  -- first prints each of "a", "b", and "c" on their own lines
  -- next, prints "2". wait, what?

What in the world? What’s going on here?

As it turns out, a function that returns zero values creates an empty multival, as you’d expect. It also doesn’t add anything to the end of a multival if you call it at the end. However, if a call to a zero-return-value function appears before the end of a multival, it results in a nil being inserted into the multival in place of its zero return values. Thus, instead of collapsing the multival made up of all three of the print calls into a single zero-length multival, Lua instead collapses the last call to print, but turns the other two calls into nil. This all doesn’t apply to tables, because, unlike in multivals, nils in tables are truly identical to absent values.

At first, this all seems like something that doesn’t belong in Lua. Without the select("#", ...) form, there’d be no way of telling a nil at the end of a multival from the end of the multival itself. However, this makes certain abstractions much easier to create.

One notable situation where select("#", ...) becomes particularly useful is when wrapping functions that return multiple values. select("#", ...) lets you easily tell if you can simply save the first return value from the function or if you need to create a table to save all the values.

Are You Sure That’s a Tail Call?

When first looking at multivals in Lua, a programmer who’s determined to use them somehow might first be inclined to combine them with tail recursion to neatly express different functions. For instance, we could implement range as follows:

local function range1(acc, n)
  if n < 1 then return
  elseif n == 1 then return acc
  else return acc, range1(acc+1, n-1)
  end
end

local function range(n)
  if n < 1 then error("n must be >= 1") end
  return range1(1, n)
end

print(range(3))
  -- prints "1 2 3"

print(#{range(10)})
  -- prints "10"

This works great! We can call range(n) when we want a multival of length n, and surround it with braces like {range(n)} when we want the equivalent table. This should let us avoid allocating extra tables when we don’t need them. And since Lua has tail call elimination, the recursive call range1 makes to itself should never blow the stack. Right?

print(#{range(1000 * 1000)})
  -- throws a stack overflow error

Hmm.

As it turns out, multivals break tail call elimination. If you’re returning multiple values, Lua needs to collect all those values before it can actually begin to return them. This means that it can’t throw away the stack frames of the recursing function. It’s very similar to what would happen if you returned {range1(acc, n)} within range1. That case is clearly not a tail call. It’s the particular syntax of multivals that makes this confusing, since it doesn’t look like the faux-tail-call is “surrounded” by anything.^[2]

In order to avoid the stack overflow and properly recurse the function, we can implement range with a table as follows:

local function range1(tab, acc, n)
  if n < 1 then return tab
  else
    tab[acc] = acc
    return range1(tab, acc+1, n-1)
  end
end

local function range(n)
  if n < 1 then error("n must be >= 1") end
  return range1({}, 1, n)
end

print(#range(1000 * 1000))
  -- prints "1000000"

It’s perhaps less elegant, mixing recursion and mutation, but its runtime characteristics are vastly better.

Maximum Occupancy

It makes sense that recursing to much could cause issues by blowing the stack. What’s less intuitive is that multivals have a cap on their size even when you make them without recursion. Consider this use of the range function we defined above:

local t = range(1000 * 1000)
print(t)
  -- prints a table identity

Great! Now, let’s try unpacking that into a multival. No recursion is involved, so this should work fine, right?

local t = range(1000 * 1000)
print(table.unpack(t))
  -- throws "too many results to unpack" error

As it turns out, multivals simply have a cap on the number of values they can contain. There’s no way to work around this - if you need to handle a list of items without worrying about how big it is, use a table, not a multival.

Multival literals in function arguments and return values are even more restricted:

print((function() return 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
            14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
            28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
            42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
            56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,
            70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
            84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97,
            98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
            110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120,
            121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131,
            132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
            143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153,
            154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
            165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175,
            176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186,
            187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
            198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,
            209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
            220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230,
            231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241,
            242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252,
            253, 254, 255 end)())
  -- throws "function or expression needs too many registers" error

print(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
      19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
      35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
      51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
      67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,
      83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98,
      99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,
      112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124,
      125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
      138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150,
      151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
      164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176,
      177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189,
      190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202,
      203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215,
      216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228,
      229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241,
      242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254,
      255)
  -- throws "function or expression needs too many registers" error

There is, however, no limit on table literals, reinforcing the fact that the contents of table literals are not multivals, despite their similarity in appearance:

-- generate.lua
print("print(#{")
for i = 1, 1000 * 1000 do
    print(i ..",")
end
print("})")

lua generate.lua > generated.lua && lua generated.lua
# prints "1000000"

Multivals are Data Structures, Just Bad Ones

What all these different examples of multival quirks demonstrate is that multivals are just another data structure. (I find this case is made best by how tail call elimination doesn’t work when recursing within a multival, just like it wouldn’t work within a table literal.) They aren’t exceptions to the rules that data structures follow - in fact, they have extra rules that tables don’t. To sum up:

• Multivals cannot be assigned to variables. They can only referred to as literals, function call expressions, or the vararg ....
• The vararg cannot be used outside the function that creates it (including within closures made within that function).
• Multivals are cut off at the first value when inserting them into another multival before its end.
• Unlike tables, multivals have a built-in length that is unrelated to the arrangement of nils within the multival. This length can be retrieved with select("#", ...).
• When a multival contains a call to a zero-return-value function before the end of the multival, a nil is inserted where the function’s return value would go.
• When a function makes a recursive call within a multival, tail call elimination is not applied. Recursing too many times within a multival will thus blow the stack.
• Unpacking too many values from a table into a multival will result in an error.
• Trying to call or return from a function with too many arguments will result in an error. This limit is much lower than the previously-mentioned limit on unpacking tables, being just below 255 items.

A Last Tiny Nitpick

Finally, one incredibly subjective thing that bugs me about multivals is the syntax used for the vararg. In my opinion, the fact that ... is valid Lua makes it unnecessarily hard to insert an easily-understood placeholder into example code.

Conclusion

While multivals remain a strange and sharp-edged corner of Lua, I hope that they’re a little bit easier to understand thanks to this post. While they’re rarely the best solution to a given problem, it’s still helpful for a Lua user to understand their strengths and limitations, even if it’s only to justify avoiding them.

If you’d like to leave a comment, please email benaiah@mischenko.com