I’ve been spending a lot of time lately writing Fennel and contributing to its compiler. Since it’s a language that compiles to Lua, that also means spending a lot of time getting to know Lua. This article will be focused on Lua (though I’ll address Fennel from time to time in the footnotes).

Lua is a neat, elegant, and relatively simple language. I find it particularly notable for its embeddability into other programs; capability of hosting many different styles of programming; and the excellent performance of one of its implementations, LuaJIT. In many ways, it feels like a more elegant and restrained JavaScript - it supports tail call elimination^[1] and coroutines, it doesn’t have the confusing var/let distinction (though it lacked a feature like const until Lua 5.4), and it doesn’t privilege class syntax over other styles of programming (object oriented programming can be done in Lua with metatables, but it’s not given special syntax like it is in JS).

That said, it has aspects that are undoubtedly quirky - arrays and dictionaries are unified into "tables", whose integer indices start at 1; the standard library is unusually minimal, a nod to its embeddability; it, like JavaScript, makes the immense error of making variables global by default and local only when you ask. Curiously, it also supports a quite uncommon language feature: multiple return values (which we’ll call “multivals” for brevity’s sake). It is this last subject that we’ll be unpacking more about today.

Since Lua is a dynamic language (as opposed to, say, Go) multiple returns are implemented in a rather odd way. This isn’t explored in as much detail as I’d like in Lua’s documentation, so I wanted to provide a one-stop resource that explains everything I can think to write down about this idiosyncratic part of the language.

A spoiler: unfortunately, multivals will turn out to be somewhat underwhelming for use in code you care about. They’re awkward to work with, have numerous gotchas, are a maintenance hazard, and don’t perform notably better (that I know of) than their primary alternative, tables. This is not a very useful article, unless you have a use for knowing minutiae about Lua’s behavior.

Let’s dive in!

An Introduction to Multivals - 1, 2, 3, unpack, and ...

There are three primary ways to represent a multival in Lua. Each of these has their own distinct uses. Each of the following code snippets produces the same multival:

• A multival literal: 1, 2, 3. (Note that this does not include table literals like {1, 2, 3}).
• Unpacking a table: table.unpack({1, 2, 3})
• The vararg, ..., in a function: local function x(...) return ... end x(1, 2, 3)

Multivals can be packed into a table by simply calling them:

local function two_values() return "first", "second" end
local tab = {two_values()}
print(tab[1], tab[2])
  -- prints "first second"

To manipulate multivals, we can use two techniques: function signatures and select:

local function multival_first(x, ...) return x end
print(multival_first(1, 2, 3))
  -- prints "1"

local function multival_rest(x, ...) return ... end
print(multival_rest(1, 2, 3))
  -- prints "2 3"

print(select(2, 1, 2, 3))
  -- prints "2 3"; identical to the last line

In order to get values from the end of a multival, we have three options. The first is recursion:

local function increment_each_value(first, ...)
  if first then
    return first + 1, increment_each_value(...)
  end
end

print(increment_each_value(1, 2, 3))
  -- prints "2 3 4"

The second is packing the multival into a table. There are two ways to do this, both of which are demonstrated below:

-- 1. You can pack a multival directly into a table literal
local function pack_multival(...)
  return {...}
end

-- Lua's default printing for tables only shows identity; we'll just
-- print the length instead.
print(#pack_multival(1, 2, 3))
  -- prints "3"

-- 2. You can use table.pack, which is built in to Lua 5.2 and later
local tab = table.pack(1, 2, 3)

print(#tab)
  -- prints "3"

-- table.pack also sets the "n" field on the table it returns. This is
-- more efficient than using #tab, which iterates through the table.
print(tab.n)
  -- prints "3"

The final way of getting values from the end of a multival is select combined with select("#", ...) (which we’ll dig into more later on):

local function get_end_of_multival(...)
  local count = select('#', ...)
  return select(count, ...)
end

print(get_end_of_multival(1, 2, 3, 4, 5))
  -- prints "5"

The Vararg: A Second-Class Value

When you start working with multivals, they might seem pretty straightforward. As long as you’re working on something that uses a list with fixed length or that can be expressed by iterating through the head and tail of a list, it seems like working with multivals should be nice and predictable.

The first issue you’ll likely run into if you try to use multivals heavily comes from expecting them to work the same way the rest of Lua does. In short, Lua’s local variables use what’s called lexical scope. This means that the scope of a variable definition is the block of code that you see it contained in in your source code file. It doesn’t matter what order other functions are called in, or whether they’re called at all - the definitions of local variables can be determined just by looking at the structure of the code. This is as opposed to dynamic scope, where the variables defined at a given point are dependent on the runtime of the program itself. Lua’s global variables effectively use dynamic scope.

As discussed above, there’s only three ways to refer to a multival: calling a function for its return values, literally inserting multiple values separated by commas, or using the vararg symbol ... as a function argument. It’s this last case that we’re interested in.

When you use ... in the signature of a function, you must place it at the end of the argument list. (This is what prevents you - syntactically, at least - from accessing the end of ... without either assigning the whole thing to individual variables, recursing through it, or packing it into a table.)

You also can’t assign the whole thing to a variable, because assigning a ... to a variable means unpacking the multival and assigning the first value to the variable - or multiple values, if that’s specified. For example:

-- Assigning the vararg to a single variable will assign that variable
-- to the _first value_ of the vararg.
local function try_to_assign_vararg(...)
  local x = ...
  return x
end

print(try_to_assign_vararg(1, 2, 3))
  -- prints "1"

There’s one additional rule about the vararg, however, that’s very unlike the behavior of the rest of Lua. You can only use a vararg within the function where it is defined. This prevents you from, for instance, saving the vararg into a closure:

local function try_to_return_vararg_from_closure(...)
  return function()
    return ...
  end
end

This fails to run with the following error: cannot use '...' outside a vararg function near '...'. This means that, while you can use ... in two nested functions, each ... can only refer to the vararg of the function that’s currently running. You can’t capture ... within a closure to persist it. Effectively, ... is not lexically scoped, but rather a dynamically scoped variable with certain extra rules like not being reassignable and not being usable outside a function that defined it.

Another way to put this would be to say that Lua’s multivals are “second class” values. You might be familiar with languages that have second class functions, which can’t be assigned to variables or passed as a parameter to a functions. This is similar, with the notable exception that we can pass the vararg to another function. We just can’t save it in a variable, save it in a closure, or manipulate it in certain ways.

Cutoff Multivals

Passing multivals to functions and packing them into tables is pretty straightforward, but there’s one major gotcha about it we haven’t gone over yet. Take a look at the following example:

local function returns_three_values()
  return 1, 2, 3
end

print(returns_three_values())
  -- prints "1 2 3"
print(#{returns_three_values()})
  -- prints "3"

print(returns_three_values(), 4, 5)
  -- prints "1 4 5"
print(#{returns_three_values(), 4 5})
  -- prints "3"

As the second pair of print expressions demonstrates, a function call can only return multiple values into a multival if it is the last thing in the multival. Following a function call in a multival with any other value, even nil, will cut off its return values at the first item.

This also applies to the vararg, ..., in function definitions:

local function f(...)
  return ..., 4, 5
end

print(f(1, 2, 3))
  -- prints "1 4 5"

Lua’s Most Jarring Feature: select("#", ..)

One of Lua’s nicer properties is how a key set to nil and a non-existent key are indistinguishable in a table. In Javascript, for instance, you have both obj.x = undefined to set a property to undefined and delete obj.x to actually remove it from the object. (There’s also obj.x = null, but let’s ignore that for now.) In Lua, there’s just tab.x = nil, and there’s no way to distinguish between a property that was set to nil and a property that was never set at all.

With that in mind, let’s look at the following example:

local y = {nil,nil,nil}

print(#y)
   -- prints "0" - nils at the end of a table don't matter

local function multival_length(...) select("#", ...) end

print(multival_length(table.unpack(y)))
  -- prints "0"

print(multival_length(nil, nil, nil))
  -- prints "3". wtf?

As this example demonstrates, multivals throw that nice property out the window. Just to be clear:

• There is no distinction between {nil, nil} and {} in Lua.
• There is one distinction between multival_length(nil, nil) and multival_length() in Lua. That distinction is made byselect("#", ...).

This becomes even more confusing when combined with a function that returns zero values, as opposed to nil:

print(select("#", print("a"), print("b"), print("c")))
  -- first prints each of "a", "b", and "c" on their own lines
  -- next, prints "2". wait, what?

What in the world? What’s going on here?

As it turns out, a function that returns zero values creates an empty multival, as you’d expect. It also doesn’t add anything to the end of a multival if you call it at the end. However, if a call to a zero-return-value function appears before the end of a multival, it results in a nil being inserted into the multival in place of its zero return values. Thus, instead of collapsing the multival made up of all three of the print calls into a single zero-length multival, Lua instead collapses the last call to print, but turns the other two calls into nil. This all doesn’t apply to tables, because, unlike in multivals, nils in tables are truly identical to absent values.

At first, this all seems like something that doesn’t belong in Lua. Without the select("#", ...) form, there’d be no way of telling a nil at the end of a multival from the end of the multival itself. However, this makes certain abstractions much easier to create.

One notable situation where select("#", ...) becomes particularly useful is when wrapping functions that return multiple values. select("#", ...) lets you easily tell if you can simply save the first return value from the function or if you need to create a table to save all the values.

Are You Sure That’s a Tail Call?

When first looking at multivals in Lua, a programmer who’s determined to use them somehow might first be inclined to combine them with tail recursion to neatly express different functions. For instance, we could implement range as follows:

local function range1(acc, n)
  if n < 1 then return
  elseif n == 1 then return acc
  else return acc, range1(acc+1, n-1)
  end
end

local function range(n)
  if n < 1 then error("n must be >= 1") end
  return range1(1, n)
end

print(range(3))
  -- prints "1 2 3"

print(#{range(10)})
  -- prints "10"

This works great! We can call range(n) when we want a multival of length n, and surround it with braces like {range(n)} when we want the equivalent table. This should let us avoid allocating extra tables when we don’t need them. And since Lua has tail call elimination, the recursive call range1 makes to itself should never blow the stack. Right?

print(#{range(1000 * 1000)})
  -- throws a stack overflow error

Hmm.

As it turns out, multivals break tail call elimination. If you’re returning multiple values, Lua needs to collect all those values before it can actually begin to return them. This means that it can’t throw away the stack frames of the recursing function. It’s very similar to what would happen if you returned {range1(acc, n)} within range1. That case is clearly not a tail call. It’s the particular syntax of multivals that makes this confusing, since it doesn’t look like the faux-tail-call is “surrounded” by anything.^[2]

In order to avoid the stack overflow and properly recurse the function, we can implement range with a table as follows:

local function range1(tab, acc, n)
  if n < 1 then return tab
  else
    tab[acc] = acc
    return range1(tab, acc+1, n-1)
  end
end

local function range(n)
  if n < 1 then error("n must be >= 1") end
  return range1({}, 1, n)
end

print(#range(1000 * 1000))
  -- prints "1000000"

It’s perhaps less elegant, mixing recursion and mutation, but its runtime characteristics are vastly better.

Maximum Occupancy

It makes sense that recursing to much could cause issues by blowing the stack. What’s less intuitive is that multivals have a cap on their size even when you make them without recursion. Consider this use of the range function we defined above:

local t = range(1000 * 1000)
print(t)
  -- prints a table identity

Great! Now, let’s try unpacking that into a multival. No recursion is involved, so this should work fine, right?

local t = range(1000 * 1000)
print(table.unpack(t))
  -- throws "too many results to unpack" error

As it turns out, multivals simply have a cap on the number of values they can contain. There’s no way to work around this - if you need to handle a list of items without worrying about how big it is, use a table, not a multival.

Multival literals in function arguments and return values are even more restricted:

print((function() return 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
            14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
            28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
            42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
            56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,
            70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
            84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97,
            98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109,
            110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120,
            121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131,
            132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
            143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153,
            154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
            165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175,
            176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186,
            187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
            198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,
            209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
            220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230,
            231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241,
            242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252,
            253, 254, 255 end)())
  -- throws "function or expression needs too many registers" error

print(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
      19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
      35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
      51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
      67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,
      83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98,
      99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,
      112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124,
      125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
      138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150,
      151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163,
      164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176,
      177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189,
      190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202,
      203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215,
      216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228,
      229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241,
      242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254,
      255)
  -- throws "function or expression needs too many registers" error

There is, however, no limit on table literals, reinforcing the fact that the contents of table literals are not multivals, despite their similarity in appearance:

-- generate.lua
print("print(#{")
for i = 1, 1000 * 1000 do
    print(i ..",")
end
print("})")

lua generate.lua > generated.lua && lua generated.lua
# prints "1000000"

Multivals are Data Structures, Just Bad Ones

What all these different examples of multival quirks demonstrate is that multivals are just another data structure. (I find this case is made best by how tail call elimination doesn’t work when recursing within a multival, just like it wouldn’t work within a table literal.) They aren’t exceptions to the rules that data structures follow - in fact, they have extra rules that tables don’t. To sum up:

• Multivals cannot be assigned to variables. They can only referred to as literals, function call expressions, or the vararg ....
• The vararg cannot be used outside the function that creates it (including within closures made within that function).
• Multivals are cut off at the first value when inserting them into another multival before its end.
• Unlike tables, multivals have a built-in length that is unrelated to the arrangement of nils within the multival. This length can be retrieved with select("#", ...).
• When a multival contains a call to a zero-return-value function before the end of the multival, a nil is inserted where the function’s return value would go.
• When a function makes a recursive call within a multival, tail call elimination is not applied. Recursing too many times within a multival will thus blow the stack.
• Unpacking too many values from a table into a multival will result in an error.
• Trying to call or return from a function with too many arguments will result in an error. This limit is much lower than the previously-mentioned limit on unpacking tables, being just below 255 items.

A Last Tiny Nitpick

Finally, one incredibly subjective thing that bugs me about multivals is the syntax used for the vararg. In my opinion, the fact that ... is valid Lua makes it unnecessarily hard to insert an easily-understood placeholder into example code.

Conclusion

While multivals remain a strange and sharp-edged corner of Lua, I hope that they’re a little bit easier to understand thanks to this post. While they’re rarely the best solution to a given problem, it’s still helpful for a Lua user to understand their strengths and limitations, even if it’s only to justify avoiding them.

If you’d like to leave a comment, please email benaiah@mischenko.com

So you’ve been writing an application in Go, and you’re getting to the point where you have a lot of different options in your program. You’ll likely want a configuration file, as specifying every option on the command-line can get difficult and clunky, and launching applications from a desktop environment makes specifying options at launch even more difficult.

This post will cover configuring Go apps using a simple, INI-like configuration language called TOML, as well as some related difficulties and pitfalls.

TOML has quite a few implementations, including several libraries for Go. I particularly like BurntSushi’s TOML parser and decoder, as it lets you marshal a TOML file directly into a struct. This means your configuration can be fully typed and you can easily do custom conversions (such as parsing a time.Duration) as you read the config, so you don’t have to do them in the rest of your application.

Configuration location

The first question you should ask when adding config files to any app is "where should they go?". For tools that aren’t designed to be run as a service, as root, or under a custom user (in other words, most of them), you should be putting them in the user’s home directory, so they’re easily changed. A few notes:

• Even if you currently have only one file, you should use a folder and put the config file within it. That way, if and when you do need other files there, you won’t have to clutter the user’s home directory or deal with loading config files that could be in two different locations (Emacs, for instance, supports both ~/.emacs.d/init.el and ~/.emacs for historical reasons, which ends up causing confusing problems when both exist).

• You should name your configuration directory after your program.

• You should typically prefix your config directory with a . (but see the final note for Linux, as configuration directories within XDG_CONFIG_HOME should not be so prefixed).

• On most OSs, putting your configuration files in the user’s “home” directory is typical. I recommend the library go-homedir, rather than the User.Homedir available in the stdlib from os/user. This is because use of os/user uses cgo, which, while useful in many situations, also causes a number of difficulties that can otherwise be avoided - most notably, cross-compilation is no longer simple, and the ease of deploying a static Go binary gets a number of caveats.

• On Linux specifically, I strongly encourage that you do not put your configuration directory directly in the user’s home directory. Most commonly-used modern Linux distributions use the XDG Base Directory Specification from freedesktop.org, which specifies standard locations for various directories on an end-user Linux system. (Despite this, many applications don’t respect the standard and put their configurations directly in ~ anyway). By default, this is ~/.config/, but it can also be set with the XDG_CONFIG_HOME environment variable. Directories within this should not use a leading ., as the directory is already hidden by default.

The following function should get you the correct location for your config directory on all platforms (if there’s a platform with a specific convention for config locations which I’ve missed, I’d appreciate you letting me know so I can update the post - my email is at the bottom of the page).

import (
    "path/filepath"
    "os"
    "runtime"

    "github.com/mitchellh/go-homedir"
)

var configDirName = "example"

func GetDefaultConfigDir() (string, error) {
    var configDirLocation string

    homeDir, err := homedir.Dir()
    if err != nil {
        return "", err
    }

    switch runtime.GOOS {
    case "linux":
        // Use the XDG_CONFIG_HOME variable if it is set, otherwise
        // $HOME/.config/example
        xdgConfigHome := os.Getenv("XDG_CONFIG_HOME")
        if xdgConfigHome != "" {
            configDirLocation = xdgConfigHome
        } else {
            configDirLocation = filepath.Join(homeDir, ".config", configDirName)
        }

    default:
        // On other platforms we just use $HOME/.example
        hiddenConfigDirName := "." + configDirName
        configDirLocation = filepath.Join(homeDir, hiddenConfigDirName)
    }

    return configDirLocation, nil
}

Within the config folder, you can use any filename you want for your config - I suggest config.toml.

Loading the config file

To load a config file, you’ll first want to define the what config values you’ll use. burntsushi/toml will ignore options in the TOML file that you don’t use, so you don’t have to worry about that causing errors. For instance, here’s the proposed configuration for a project I’m maintaining, wuzz (the keybindings aren’t currently implemented, but I’ve left them in for the sake of demonstration):

type Config struct {
    General GeneralOptions
    Keys    map[string]map[string]string
}

type GeneralOptions struct {
    FormatJSON             bool
    Insecure               bool
    PreserveScrollPosition bool
    DefaultURLScheme       string
}

It’s pretty simple. Note that we use a named struct for GeneralOptions, rather than making Config.General an anonymous struct. This makes nesting options simpler and aids tooling.

Loading the config is quite easy:

import (
    "errors"
    "os"
    
    "github.com/BurntSush/toml"
)

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }
    
    var conf Config
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

toml.DecodeFile will automatically populate conf with the values set in the TOML file. (Note that we pass &conf to toml.DecodeFile, not conf - we need to populate the struct we actually have, not a copy). Given the above Config type and the following TOML file…

[general]
defaultURLScheme = "https"
formatJSON = true
preserveScrollPosition = true
insecure = false

[keys]

  [keys.general]
  "C-j" = "next-view"
  "C-k" = "previous-view"
  
  [keys.response-view]
  "<down>" = "scroll-down"

…we’ll get a Config like the following:

Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        PreserveScrollPosition: true,
        Insecure:               false,
    },
    Keys: map[string]map[string]string{
        "general": map[string]string{
            "C-j": "next-view",
            "C-k": "previous-view",
        },
        "response-view": map[string]string{
            "<down>": "scroll-down",
        },
    },
}

Automatically decoding values

wuzz actually uses another value in its config - a default HTTP timeout. In this case, though, there’s no native TOML value that cleanly maps to the type we want - a time.Duration. Fortunately, the TOML library we’re using supports automatically decoding TOML values into custom Go values. To do so, we’ll need a type that wraps time.Duration:

type Duration struct {
    time.Duration
}

Next we’ll need to add an UnmarshalText method, so we satisfy the toml.TextUnmarshaler interface. This will let toml know that we expect a string value which will be passed into our UnmarshalText method.

func (d *Duration) UnmarshalText(text []byte) error {
    var err error
    d.Duration, err = time.ParseDuration(string(text))
    return err
}

Finally, we’ll need to add it to our Config type. This will go in Config.General, so we’ll add it to GeneralOptions:

type GeneralOptions struct {
    Timeout                Duration
    // ...
}

Now we can add it to our TOML file, and toml.DecodeFile will automatically populate our struct with a Duration value!

Input:

[general]
timeout = "1m"
# ...

Equivalent output:

Config{
    General: GeneralOptions{
        Timeout: Duration{
            Duration: 1 * time.Minute
        },
        // ...
    }
}

Default config values

We now have configuration loading, and we’re even decoding a text field to a custom Go type - we’re nearly finished! Next we’ll want to specify defaults for the configuration. We want values specified in the config to override our defaults. Fortunately, toml makes really easy to do.

Remember how we passed in &conf to toml.DecodeFile? That was an empty Config struct - but we can also pass one with its values pre-populated. toml.DecodeFile will set any values that exist in the TOML file, and ignore the rest. First we’ll create the default values:

import (
    "time"
)
var DefaultConfig = Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        Insecure:               false,
        PreserveScrollPosition: true,
        Timeout: Duration{
            Duration: 1 * time.Minute,
        },
    },
    // You can omit stuff from the default config if you'd like - in
    // this case we don't specify Config.Keys
}

Next, we simply modify the LoadConfig function to use DefaultConfig:

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }

    conf := DefaultConfig
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

The important line here is conf := DefaultConfig - now when conf is passed to toml.DecodeFile it will populate that.

Summary

I hope this post helped you! you should now be able to configure Go apps using TOML with ease.

If this post was helpful to you, or you have comments or corrections, please let me know! My email address is at the bottom of the page. I’m also looking for work at the moment, so feel free to get in touch if you’re looking for developers.

Complete code

package config

import (
    "errors"
    "path/filepath"
    "os"
    "runtime"
    "time"

    "github.com/BurntSushi/toml"
    "github.com/mitchellh/go-homedir"
)

var configDirName = "example"

func GetDefaultConfigDir() (string, error) {
    var configDirLocation string

    homeDir, err := homedir.Dir()
    if err != nil {
        return "", err
    }

    switch runtime.GOOS {
    case "linux":
        // Use the XDG_CONFIG_HOME variable if it is set, otherwise
        // $HOME/.config/example
        xdgConfigHome := os.Getenv("XDG_CONFIG_HOME")
        if xdgConfigHome != "" {
            configDirLocation = xdgConfigHome
        } else {
            configDirLocation = filepath.Join(homeDir, ".config", configDirName)
        }

    default:
        // On other platforms we just use $HOME/.example
        hiddenConfigDirName := "." + configDirName
        configDirLocation = filepath.Join(homeDir, hiddenConfigDirName)
    }

    return configDirLocation, nil
}

type Config struct {
    General GeneralOptions
    Keys    map[string]map[string]string
}

type GeneralOptions struct {
    DefaultURLScheme       string
    FormatJSON             bool
    Insecure               bool
    PreserveScrollPosition bool
    Timeout                Duration
}

type Duration struct {
    time.Duration
}

func (d *Duration) UnmarshalText(text []byte) error {
    var err error
    d.Duration, err = time.ParseDuration(string(text))
    return err
}

var DefaultConfig = Config{
    General: GeneralOptions{
        DefaultURLScheme:       "https",
        FormatJSON:             true,
        Insecure:               false,
        PreserveScrollPosition: true,
        Timeout: Duration{
            Duration: 1 * time.Minute,
        },
    },
}

func LoadConfig(configFile string) (*Config, error) {
    if _, err := os.Stat(configFile); os.IsNotExist(err) {
        return nil, errors.New("Config file does not exist.")
    } else if err != nil {
        return nil, err
    }

    conf := DefaultConfig
    if _, err := toml.DecodeFile(configFile, &conf); err != nil {
        return nil, err
    }

    return &conf, nil
}

If you’d like to leave a comment, please email benaiah@mischenko.com

I’ve been meaning to learn TypeScript for some time, so I’ve decided to build a roguelike (or at least begin to build a roguelike) in order to do some canvas programming as well.

This isn’t a guide, it’s a journal of my experiments, so follow at your own risk. I can’t guarantee I’m going to make the best design decisions right off the the bat.

I’ll be omitting most of the glue code from the articles, and I’m not planning a writeup of the build system, but you can see the full source code at the GitHub repo.

Designing the tile data

I want the game to be tile-based, so I’ll need some sort of generic tile container for doing things like rendering the map, generating levels, pathfinding, and so forth. For now, I’ll just represent the individual levels as tiles:

// src/Tile.ts

enum Tile {
  Nothing = 0,
  Floor = 1
}

Eventually I’m planning on making this an object or bitmask that stores additional information, but this will suffice for writing the container.

Row Major Ordering

Tilemaps should be as simple as possible; I want to make them cheap and reusable. The simplest way to store a grid of tiles would be an array of rowsq, where a hypothetical tileMap.at function would look like (x, y) => this[y][x]. This introduces a lot of overhead because of all the arrays. Fortunately it’s quite easy to turn this into a single array by arranging the elements like this:

const tileMap = [ 0, 0, 0, 0,
                  0, 0, 0, 0,
                  0, 0, 0, 0 ];

This is called a row-major order, and a tiles.at function would look like (width, x, y) => this[(y * width) + x].

I’d like to avoid tying higher-level code to this specific representation, so I’ve abstracted the coordinate system in case I want to swap it out. This is the code for a coordinate system:

// coords.ts

interface CoordinateSystem {
  getX(i: number): number;
  getY(i: number): number;
  getI(x: number, y: number): number;
}

class RowFirstCoordinates {
  public static GetX(i: number, width: number): number {
    return i % width;
  };
  public static GetY(i: number, width: number): number {
    return Math.floor(i / width);
  };
  public static GetI(x: number, y: number, width: number): number {
    return (y * width) + x;
  };

  // Instance methods allow you to store the width
  constructor(private width: number) { }
  public getX(i: number): number {
    return RowFirstCoordinates.GetX(i, this.width);
  }
  public getY(i: number): number {
    return RowFirstCoordinates.GetY(i, this.width);
  }
  public getI(x: number, y: number): number {
    return RowFirstCoordinates.GetI(x, y, this.width);
  }
}

This will allow us to convert in between coordinates and array indices. Now I have everything I need to start building the tilemaps.

Designing the TileMap API

I’d like tilemaps to have a fluent interface, but one that avoids mutating existing references. Here’s an example of the API I’m imagining:

const tiles = [ 0, 0, 0, 0, 0,
                0, 0, 0, 0, 0,
                0, 0, 0, 0, 0,
                0, 0, 0, 0, 0,
                0, 0, 0, 0, 0 ];
const mapA = new TileMap(tiles, 5);

// Create a region with x, y, width, and height
const regionB = {
  x:     1, y:      1,
  width: 2, height: 2
};

// Design note: Since each change creates and returns a new TileMap,
// it would be ideal to be able to batch these into some sort of
// execution-plan object that could be applied all at once, for cases
// where I don't care about the intermediate states.
const mapB = mapA
  .fillRegion(regionB, 1)
  .setAt(0, 3, 2)
  .setAt(4, 3, 2);

mapB.tiles === [ 0, 0, 0, 2, 0,
                 0, 1, 1, 0, 0,
                 0, 1, 1, 0, 0,
                 0, 0, 0, 0, 2,
                 0, 0, 0, 0, 0 ];

For now, I’m just going to create and implement the following interface, so I can get something on the screen:

// tileMaps.ts

import * as coords from "./coords";
import Tile from "./Tile";

// IRegion is used to represent an area without having to carry around
// a tile array.
interface IRegion {
  x: number;
  y: number;
  width: number;
  height: number;
}

type TileMapProcessor = { (tileMap: ITileMap): ITileMap }

interface ITileMap {
  width: number;
  height: number;
  tiles: Tile[];
  coords: coords.CoordinateSystem;
  getRegion: (region: IRegion) => ITileMap;
  fillRegion: (region: Iregion, contents: Tile) => ITileMap;
  applyProcessors: (processors: TileMapProcessor[]) => ITileMap;
}

You can see the beginnings of my desired interface in the signatures of getRegion, fillRegion, and applyProcessors - they return TileMaps so you can contine chaining operations. The methods do the following:

• getRegion takes an IRegion and returns an ITileMap containing the size of that region, containing the tiles in that region.
• fillRegion takes an IRegion and a Tile and returns a new ITileMap that is the result of setting all of the tiles in this ITileMap to contents.
• applyProcessors takes a list of functions that transform an ITileMap to another ITileMap. It passes this map through those functions in the order given.

The implementation for this is fairly straightforward:

// tileMaps.ts

import { assert, zeroedArray } from "./util"

class TileMap {
  public static NewEmpty(width: number, height: number) {
    const length = width * height;
    const tiles: Tile[] = zeroedArray(length);
    return new TileMap(tiles, width);
  }

  public readonly tiles: Tile[];
  public coords: coords.RowFirstCoordinates;
  public readonly height: number;
    public readonly width: number;
  constructor(tiles: Tile[], width: number) {
    assert(tiles.length % width === 0);
    this.width = width;
    this.height = tiles.length / width;
    this.coords = new coords.RowFirstCoordinates(width);
    this.tiles = tiles;
  }

  public getRegion(region: IRegion): TileMap {
    const { x, y, width, height } = region;
    assert((x + width <= this.width) && (y + height <= this.height));
    const regionTiles = this.tiles.filter((tile, i) => {
      const tileX = this.coords.getX(i);
      const tileY = this.coords.getY(i);
      if ((tileX >= x && tileX < x + width) && (tileY >= y && tileY < y + height)) {
        return true;
      } else {
        return false;
      }
    });

    return new TileMap(regionTiles, width);
  }
  public fillRegion(region: IRegion, contents: number): TileMap {
    const { x, y, width, height } = region;
    assert((x + width <= this.width) && (y + height <= this.height));
    const newTiles = this.tiles.map((tile, i) => {
      const tileX = this.coords.getX(i);
      const tileY = this.coords.getY(i);
      if ((tileX >= x && tileX < x + width) && (tileY >= y && tileY < y + height)) {
        return contents;
      } else {
        return tile;
      }
    });

    return new TileMap(newTiles, this.width);
  }
  public applyProcessors(processors: TileMapProcessor[]): ITileMap {
    return processors.reduce((map, processor) => processor(map), this as ITileMap);
  }
}

I’ll make another type to represent renderers, and then I can move on to actually drawing something

// tileMaps.ts

type TileMapRenderer = { (tileMap: ITileMap): void };

Drawing a map

Finally, I’m ready to draw my first map. First I’ll get the canvas and create a game loop:

// main.ts

import * as tileMaps from "./tileMaps";

const canvas = <HTMLCanvasElement>document.getElementById("canvas");
const ctx = canvas.getContext("2d");
// Nothing's changing yet, so I only need to draw once
let drawn = false;
let time;

const gameLoop = (ctx: CanvasRenderingContext2D,
                  map: tileMaps.ITileMap,
                  renderer: tileMaps.TileMapRenderer) => {
  const now = new Date().getTime();
  const dt = now - (time || now);
  time = now;

  console.log(dt);

  const recurse = (delta) => gameLoop(delta, ctx, map, drawer);
  if (!rendered) {
    renderer(map);
    rendered = true;
  }
  requestAnimationFrame(recurse);
};

Now I just need a map and a renderer. I’ll generate a map with some simple rooms:

// main.ts

class Room extends tileMaps.TileMap {
  public readonly x;
  public readonly y;
  constructor(public readonly parent: tileMaps.ITileMap, region: tileMaps.IRegion) {
    super(parent.getRegion(region).tiles, region.width);

    this.x = region.x;
    this.y = region.y;
  }
}

function getRandomRooms(tileMap: tileMaps.ITileMap): Room[] {
  return zeroedArray(getRandomInt(3, 9)).map(() => {
    const width = getRandomInt(10, 20);
    const height = getRandomInt(10, 20);
    const x = getRandomInt(0, tileMap.width - width);
    const y = getRandomInt(0, tileMap.height - height);
    return new Room(tileMap, { x, y, width, height });
  });
}

function createRooms(tileMap: tileMaps.ITileMap): tileMaps.ITileMap {
  const rooms = getRandomRooms(tileMap, 10);

  return rooms.reduce(
    (map, room) => map.fillRegion(room, Tile.Floor), tileMap);
}

const mapCreationLayers = [
  createRooms,
];

You’ll notice the signature of createRooms: (tileMap: tileMaps.ITileMap): tileMaps.ITileMap. That’s the same type for which I defined the alias tileMaps.TileMapProcessor - meaning I can create an empty map and pass it through mapCreationLayers as follows:

// main.ts

const mapWidth = 120;
const mapHeight = 80;

const map = tileMaps.TileMap.NewEmpty(mapWidth, mapHeight)
  .applyProcessors(mapCreationLayers);

And finally, I write a function to render the tiles to the canvas:

// main.ts

const render = (tileMap: tileMaps.ITileMap) => {
  tileMap.tiles.map((tile, i) => {
    const x = map.coords.getX(i);
    const y = map.coords.getY(i);
    ctx.fillStyle = (tile === Tile.Floor) ? "#FFF" : "#000";
    ctx.fillRect(x * tileWidth, y * tileHeight, 10, 10);
  });
};

Eventually the tile colors will be drawn with a pixel buffer which is scaled up to the full size of the canvas, making drawing a grid of solid colors much faster. At the moment, however, I’m only rendering the map once - in the starting frame.

It’s all set - I can now kick off the game loop to start it off:

// main.ts

gameLoop(ctx, map, render);

Doesn’t look like much, but it’s a success! In the next post I’ll be adding more types of tiles and exploring ways to generate a level. You can also check out the GitHub repo, or try a running version at roguelike.benaiah.me.

If you’d like to leave a comment, please email benaiah@mischenko.com

Visit the Github Repo

Update

At some point, ido-hacks became no longer required, and ido-ubiquitous was folded into another package - ido-completing-read+. I’ve also somewhat modified my config since I wrote this post. Here’s the ido config I’m using today:

(use-package ido :quelpa
  :config
  (setq ido-everywhere t
    ido-virtual-buffers t
    ido-use-faces t
    ido-default-buffer-method 'selected-window
    ido-auto-merge-work-directories-length -1)
  (ido-mode))
(use-package flx-ido :quelpa :requires ido :config (flx-ido-mode))
(use-package ido-vertical-mode :quelpa :requires ido :config (ido-vertical-mode))
(use-package ido-completing-read+ :quelpa :requires ido
  :config
  (setq ido-ubiquitous-max-items 50000
    ido-cr+-max-items 50000)
  (ido-ubiquitous-mode +1))

If you’re wondering what the :quelpa keyword means or what it does, see my other post “Bootstrapping an Emacs config with quelpa-use-package”.

Original post

I’m a heavy user of ido – it handles all of my completion, whether execute-extended-command, find-file, or whatever else. However, ido’s default configuration leaves much to be desired:

• The way it displays results by default is quite difficult to read, as you can’t quickly skip your eyes to the next item when they’re all listed in a single wrapped line.
• The default completion is prefix-only
• It’s not obvious how to make ido actually complete everything

Fortunately this is all quite fixable with a few packages and a little configuration:

• ido-everywhere, when set to t, puts ido in more places than the default.
• ido-virtual-buffers doesn’t fix anything per se, but it does add recentf files to find-file completion, which is really handy.
• ido-vertical-mode makes ido display each result on its own line, with a configurable amount of lines to display at a time (though I just use the default).
• flx-ido fixes the completion by using the flx completion engine.
• ido-hacks and ido-ubiquitous together make ido complete everything I’ve wanted so far.

My full ido configuration is as follows:

(use-package ido :quelpa :config
  (setq ido-auto-merge-delay-time 99999999)
  (setq ido-everywhere t)
  (setq ido-virtual-buffers t)
  (ido-mode))
(use-package flx-ido :quelpa :requires ido :config (flx-ido-mode))
(use-package ido-vertical-mode :quelpa :requires ido :config (ido-vertical-mode))
(use-package ido-hacks :quelpa (:stable nil) :requires ido :config (ido-hacks-mode))
(use-package ido-ubiquitous :quelpa :requires ido :config (ido-ubiquitous-mode))

Enjoy!

If you’d like to leave a comment, please email benaiah@mischenko.com

Emacs configuration is already a hairy enough beast as it is, but package management introduces a whole new set of confusing, difficult problems. What I want from an Emacs package management system is the following:

• Every package I use should be recorded in my config
• My init file should ensure all my packages are installed when I start emacs.
• If any packages are not installed, the init should automatically download and install them.
• MELPA Stable should be used, not MELPA, unless I specify that a package should be installed from regular MELPA (for packages that don’t have a stable version).
• I don’t want to have to manually trigger package updates.

It’s a fairly simple set of requirements, but for a long time it was all-but-impossible to impossible to actually get all of them with Emacs’ packaging functionality. Fortunately a couple projects have come out that allow this vision to become a reality:

• use-package allows you to specify your packages in your init and keep their configuration isolated and tidy.
• quelpa allows you to build Emacs packages directly from source, using MELPA recipes. It also allows you to default to MELPA Stable, only using MELPA when specified.
• quelpa-use-package combines the two, allowing you to specify how a package should be installed as part of its use-package declaration.

I’ve created a minimal example of how to use these three packages together in such a way that you don’t need anything installed except for Emacs and the init.el. Once you start Emacs, everything will be automatically installed for you. You can find that at quelpa-use-package-bootstrap-config. Please note that I haven’t tested this with any versions of Emacs prior to 25.1.1 (the version I’m currently running), nor on Windows, but I expect it should work as far back as 24.3.1 or 24.4, and there’s no OS-specific code in the init.

Some caveats:

• You do have to pay attention to your packages and their dependencies. Some packages require you to grab other packages explicitly before they will work. In addition, MELPA packages that come from EmacsWiki usually fail to build on my machine, so I rehost the wiki packages I need to GitHub and install them like so:

  (use-package goto-chg :quelpa
    (goto-chg :stable nil :fetcher github :repo "benaiah/goto-chg"))
  

• If you’re not on a network and you don’t have your packages installed, this won’t work.
• While I’m writing this, no MELPA Stable version of quelpa-use-package exists, so you do technically have at least one unstable dependency with this setup. However, quelpa-use-package currently works well, and is rarely updated.

This setup has been working very well for me, and I hope it will be useful to you as well. It’s solid, easy to use (once you have it set up), and should theoretically make it trivial to set up Emacs on a new computer.

If you’d like to leave a comment, please email benaiah@mischenko.com

Visit the Github Repo

Update - changed the tool used and watch code

Originally, this post used a tool called fswatch to monitor the file system. Unfortunately, fswatch has a bug on some Linux systems where it detects file reads as file changes. The bug has gone unaddressed for several months. It causes a lot of problems - in my static site generator, the -w option will build once and wait for an event fine, but as soon as it sees a change it will start building the site over and over again without any changes, as it detects the generator reading the source files and triggers a new build, which repeats ad infinitum.

I’ve thus replaced the tool used for monitoring directory changes with watchexec, a similar tool, also cross-platform, written in Rust. While I haven’t been able to test this on platforms other than Linux, it doesn’t have any known bugs of that variety, and fswatch's bug made it completely unusable on what has become my primary platform.

Updated Post

I’ve been building a new site lately, which will likely be inaugurated with this very post. My tech decisions for the site are a whole post of their own, but suffice it to say I’m using a static site generator I wrote in Racket, and it’s going swimmingly.

As I’m working on both the generator and the site at the same time, most of the time I have a pain point while writing I can fix it myself. (This is not necessarily a good thing, as I’d likely get more writing done if I wasn’t spending so much time on the generator). One of the biggest things I want out of the generator is live file watching, so when I change files it automatically updates the site.

There aren’t any existing libraries that I could find for watching a filesystem for changes, but this presented an opportunity to learn how to implement an oft-used pattern of mine: shelling out.

You see, there’s a cross-platform, very simple file change monitor called watchexec. Simple as in "given a directory (searched recursively), whenever a file is changed, run a command". Simple is particularly nice in this case because it means it will be easy to interpret the output – all I need to do is figure out how to watch a path.

Spawning the process

Looking at the documentation for process, it appears that we can spawn a process and get its input and output ports^[1]. Let’s write a function that will let us watch a list of paths (this requires racket/string and racket/system):

;; `watch' requires that the `watchexec' command is installed.
(define (watch-paths paths)
  (let ((paths-string
         (string-join
          (map (lambda (path)
                 ;; Quote paths to avoid escaping problems
                 (string-append
                  "-w \""
                  ;; Accept either strings or path objects
                  (cond [(string? path) path]
                        [(path? path) (path->string path)]
                        [else ""])
                  "\" "))
               paths)
          " ")))
    (process (string-append
              "watchexec " paths-string "-- echo \"File changed.\""))))

We can run watchexec at the command line to check the output[^2]:

$ watchexec -w ~/dev/personal-site -- echo "File changed."
File changed.
File changed.

Now we need to kick off a build when the watch tools produces output. Fortunately Racket makes it very easy to call a callback when a port produces output, using handle-evt. It’s as simple as (handle-evt <event object> <event handler). We do need to explicitly create a loop and a channel we can use to break from it, as well.

(define (watch site-directory)
  (letrec (;; Start the watch process
           (watcher (watch-paths (list (build-path site-directory "src"))))

           ;; This is the port we use to read the watch tool's output
           (watcher-output (list-ref watcher 0))

           ;; Putting a message to this channel allows us to exit the
           ;; watch loop programatically
           (exit-channel (make-channel))

           ;; `process' requires that its IO ports are manually
           ;; closed, so we do so in the exit channel's handler
           (exit-handler
            (lambda (val)
              (close-input-port watcher-output)
              (close-output-port (list-ref watcher 1))))

           ;; This function handles the events fired by the watcher
           ;; output port
           (input-handler
            (lambda (val)
              (let ((output (read-line watcher-output)))
                (cond
                 ;; If the watch tool stops, we'll get an EOF in the
                 ;; port. When this happens we need to quit
                 [(eof-object? output)
                  (channel-put exit-channel 'done)
                  (loop "")]
                 [else (loop output)]))))

           ;; This is the function that will loop and wait for events
           (loop
            (lambda ([files "No"])

              ;; We don't want to build the site when exiting, where
              ;; we'll be passed an empty string
              (and (not (string=? files ""))
                   ;; Build the site
                   (displayln "")
                   (displayln (string-append files " files changed, rebuilding..."))
                   (build site-directory)
                   (displayln "Built!"))

              ;; `sync' waits for either the input port or the exit
              ;; channel to send an event
              (sync (handle-evt watcher-output input-handler)
                    (handle-evt exit-channel exit-handler))))

           ;; Kick off the loop
           (watch-thread (thread (lambda () (loop)))))

    ;; Callers may need to be able to access both the thread (so they
    ;; can wait on it if necessary), and the exit channel (so they can
    ;; programatically stop the loop).
    `((thread . ,watch-thread)
      (exit-ch . ,exit-channel))))

Watching from the command line

We’re almost done! At this point we can call (watch <site-directory>) and it will call the build function every time a file changes. However, we’d rather be able to do this without having to enter the Racket REPL every time.

Fortunately creating a command line wrapper for this function is quite easy. All we need to do is create a new file with no .rkt extension and put a hashbang line that points to the racket executable at the top:

#!/usr/local/bin/racket
#lang racket

(require "build.rkt")

(define watch? (make-parameter #f))

(define args
  (command-line
   #:once-each
   [("-w") "watch" (watch? #t)]
   #:args
   (str) str))

(if (watch?)
    (let ((watcher (watch args)))
      (thread-wait (dict-ref watcher 'thread)))
    (build args))

The thread-wait waits for the loop thread we spawn before exiting the process. Without this, the script would exit immediately, which defeats the purpose.

If you’d like to leave a comment, please email benaiah@mischenko.com