Goodbye, Lua

As of May 19th, I've unsubscribed from the Lua mailing list. Thus ends an era.

For years I have been a strong proponent of Lua. I've used it in my games, and I've used it in servers with Nginx. I've used Lua to write command line tools, and I've used it to write extensions for existing games. Lua is a great language to play with for a number of reasons. The syntax is easy to learn and somewhat forgiving. It includes the concept of tables, which double as arrays, and even though it's extremely simple, it's also very fast (extremely fast when using LuaJIT).

So why did I leave?

Five factors combined in a perfect storm to cause me to drop Lua as a game scripting language, as a server language, and as a language for my command line tools:

More details on each point below.

Future of LuaJIT

Mike Pall, the author of LuaJIT, single-handedly created the most amazing dynamic language JIT compiler ever. He may hold that title for years, despite the fact that he's no longer extending LuaJIT. He resigned from the project to move on.

LuaJIT will continue to be an awesome example of what a single brilliant developer can create, of course, but it means that the people who replace him have huge shoes to fill. Odds are good that anyone who does step up (and I'm not following the status; people may have stepped up already, I don't know) will probably just maintain LuaJIT, fixing bugs, and at best may bring LuaJIT 2.1 to a final release state.

But without Mike Pall, we probably won't be seeing amazing new features and performance enhancements any more. That amazing new garbage collector that he had planed to create will likely never come to pass, and there's little chance that LuaJIT will catch up to the new 5.3 syntax, meaning using Lua will forever be a choice between LuaJIT @ 5.1 with a few 5.2 upgrades, or using 5.3. leading me to….

Fragmentation of Lua

The major breaking changes in Lua in 5.3 seem to be fragmenting the already sparse Lua library ecosystem. Anyone who creates a library needs to now support LuaJIT (a 5.1/5.2 hybrid), 5.2, and 5.3. A lot of Lua libraries aren't actively updated, so in reality existing libraries are a crapshoot of 5.1, 5.2, and 5.3 support.

Lua dependency management is also very hit-or-miss. The "luarocks" package manager has about a 60% chance of working with any particular package on Windows, for instance. This has always been the case, but with the additional Lua version fragmentation, it's only making things worse.

JavaScript's Ecosystem

In contrast to Lua, JavaScript (and transpiled languages like TypeScript) have a massive, vibrant ecosystem of available libraries and frameworks available through a package manager that (usually) works.

I have encountered the occasional bug on Windows where, after I say npm install, it doesn't manage to finish installing everything. This doesn't inspire confidence. But in general it's easy enough to run the install command again, or nuke the packages folder and re-run. I don't run Windows on my servers, so this is really just a development environment issue; npm install has been completely reliable for me on Linux.

Intermittent Windows issues aside, the sheer quantity of tools available through an npm install command humbles me. There are so many that people talk of "JavaScript Fatigue," where developers, especially new ones, get overwhelmed by the number of options there are to accomplish a thing in JavaScript. No one ever talked about Lua Fatigue.

On Coroutines

The biggest feature that kept Lua head-and-shoulders above JavaScript, in my opinion, was coroutines. In case you're not familiar with the concept, it is a feature that lets you write code that can pause in the middle and resume later when you're ready for it to do more work.

It was useful for game AI ("do this, then this, and then this!"), and it was amazing for writing simple and amazingly fast web servers ("grab THIS from the database, THIS from this other server, and then send a response"). By fast I mean: If you compare to a connection-per-thread server (like one based on PHP or Ruby), you can see a 50-100x performance improvement. In games, it simply makes some kinds of code possible to write out in an easy to follow fashion.

JavaScript on the server, using NodeJS, could achieve similar levels of performance, but the code was not nearly as clean. Instead of writing out the steps in a simple "A,B,C" fashion, you instead would write "do A, then when it's done call this other function that does B, which calls this other function that does C." There was a common name for this pattern: Callback Hell.

In the current version of JavaScript, called variously ES5 or ES2015, which is available as part of Node 4.x, there's a new feature called "generators". Combined with another feature called "Promises," and using a library named "co", you can now write code in JavaScript that looks more like the "A,B,C" approach. Technically speaking, you can write asynchronous code in an imperative style.

A future version of JavaScript is supposed to include a feature called async/await, which gives you even better imperative style code. This feature can be used today with transpilers (Babel and TypeScript both support async/await).

So Lua's Killer Feature is now available in JavaScript.

Type Annotations

Lua is an easy language to write code in, in part because the types of variables isn't specified. Instead the interpreter needs to figure out the type at runtime. This trait is rather common in a lot of popular languages right now, including Ruby, PHP, and Python. It's totally in vogue.

But I found that, in writing several games and server projects in Lua, 80%+ of the bugs that I ended up having to fix were type errors.

What is a type error, you ask? It's when I try to use one thing as another. It can be as simple as trying to use a string where I need a number. Lua and JavaScript can both be somewhat forgiving about that, but there are times when it will cause an error. But that error won't happen until you run your code and execute that exact line.

A more annoying type error is when you have an object that has fields. Say you have a thing that has a color, but you accidentally type thing.Color instead of thing.color. If you're trying to set the color, you'll create a new field Color, and the actual color won't change. No runtime error, no evidence that anything is wrong – except the color isn't what you expect.

Languages like C++, on the other hand, require that every single variable and parameter have a type specified. This can be tedious, and can involve changing several files when the type of a structure changes.

TypeScript is a language that is like JavaScript, but with optional type annotations. I can use them as often as I want, and the more I use them, the more protections I get at compile time from things like misspelled field names. Even better, when I type thing. in my editor, it will know that thing has a "color" field and will offer to complete it, along with any other fields thing might have.

I swear that my productivity is almost twice what it was before.

Bonus Reasons

LuaJIT is my preferred Lua scripting engine because of its raw speed. That speed is even more important when running on mobile, where CPUs aren't as fast.

But due to an architectural decision made by Apple, you cannot JIT compile code for iOS. Period. LuaJIT is still faster than vanilla Lua, but only like 3x, not 5-100x.

But as of iOS 7, Apple has exposed JavaScriptCore, which is a full JIT-accelerated JavaScript environment. So JavaScript is now officially faster than Lua on iOS devices. which apparently doesn’t support JIT. Apple, you really should fix this! Grr.

Finally, JavaScript turns out to not suck. This was actually somewhat of a surprise to me. Yes there are blogs that describe how JavaScript is, in fact, broken. But it turns out that most of the problems can be avoided by using a few sane rules (enforced by linting), and the remainder can be avoided by using TypeScript. In fact, JavaScript and Lua share a surprising amount of architectural similarity: Both use Prototypal Inheritance, for instance.

Goodbye Lua

I honestly don't see a reason, in the foreseeable future, to ever return to the Lua ecosystem or language. I was there initially because it was awesome and small. Once upon a time I needed games to fit in a 20Mb download. Adding 10Mb to support JavaScript was unthinkable, but Lua was 0.1Mb. Today, some of my games use browsers, which come with JavaScript, so I get it for free, and other games can survive with a 10Mb bump – many games are 200Mb or more on mobile, and multiple gigabytes on PC. As I mentioned above, iOS also gets JavaScript for free.

Lua's unique advantages are no longer relevant (or no longer unique), and Lua's drawbacks (1-based arrays? Double-dash comments? Really?) will likely prevent me from ever considering using the language again. Lua does embed really nicely, so it's not impossible. But JavaScript is so much more widely known that it would be better for users.

Lua experienced quite a strong popularity as a game scripting engine. World of Warcraft famously used it. There are other games that use it, and that will likely continue for the intermediate term, if only out of inertia, but I'm guessing that Lua has passed its popularity peak.

There were some elegant things about Lua's design, and I will miss using a language as concise as Lua. But the time has come to move on, and JavaScript and TypeScript (and Go!) have replaced Lua in my heart.


comments powered by Disqus