Lua 5.4.0

(lua.org)

256 points | by smlckz 5 days ago

11 comments

  • pansa2 4 days ago

    It would be interesting to know how many projects are using regular Lua vs LuaJIT, as the two implementations have diverged.

    When both were on version 5.1, there didn't seem to be any downsides to LuaJIT. Many Lua projects could benefit from it as long as they weren't trying to run on a platform that disallows JIT compilation (e.g. iOS or a game console).

    However, now that LuaJIT is 2-3 language versions out-of-date, and has different best-practices (e.g. around FFI), it's almost like the two are different languages.

    • as-j 4 days ago

      We chose to go with Lua on our embedded project ~2 years ago, and chose lua 5.3 with the following rational:

      1. Lua embeds well on a small system and gives us a higher level language than C. lua or luajit is ok

      2. Lua isn't being used for performant code. If we have performance issues use C.

      3. LuaJit is looking for a maintainer and is stagnant. This isn't desirable.

      Not sure if this is "best practice" but it summarized how we worked through it.

      Lua 5.4 has some appealing benefits for us, we'll give it a few patch releases, but I look forward to moving to it.

      Re Performance: We're actually running from 50Hz code in Lua on an old embedded CPU. We haven't re-written that in C yet since it runs just fine. We're been constantly surprised at how well it runs and works for us.

      • fwsgonzo 4 days ago

        Performance-wise Lua is not too far behind LuaJIT that you should just "never" use Lua. I used Lua for a long time without problems, and in the cases where you need that extra performance you are just one "system call" away from native performance for that specific operation. I used to benchmark these things (for my use cases) for a long time, and Lua was always somewhere between 50-100% slower than LuaJIT, which is NOT a lot. It's a mistake to think that people implement Mandelbrot as a script callback. :)

        It's unfortunate that LuaJIT fell behind so much, but as far as game engines who need a scripting language goes: You are either safe with Lua, or you have other alternatives: JavaScript, WASM, dynamic libraries and so on.

        • WaltPurvis 4 days ago

          100% slower seems like a lot.

        • I recently redid the graphs of the luajit benchmark results to highlight performance across architectures: https://gist.github.com/cellularmitosis/55ff55b55008b2f1c42b...

          • fwsgonzo 4 days ago

            Nice work, man. I was doing microbenchmarks where I among other things measure the 90ns (JIT) and 120ns (Regular) function call overhead of Lua. My benchmarks was compared to my own scripting backend where the call overhead is 4ns, and it always feels a little bit like cheating when you have a gigantic headstart with each test.

            I'm not surprised the JIT isn't that good, but this is crazy almost! Did you compare against the latest version of Lua?

            https://gist.github.com/fwsGonzo/f874ba58f2bab1bf502cad47a9b...

            In my microbenchmarks LuaJIT was always faster than Lua, but not so much that I wouldn't stop using Lua. In my mind once you stop developing a language it stops being interesting, because you are always looking for ways to simplify and remove whole classes of bugs.

            EDIT: I just built lua5.4 from sources and ran my benchmarks again, and it's markedly worse than LuaJIT still, but it's better than before for sure. My conclusion is that Lua5.4 is the fastest PUC-Lua yet.

            https://gist.github.com/fwsGonzo/2f4518b66b147ee657d64496811...

            • anonymoushn 3 days ago

              Some of these benchmarks are maybe not how one would do things if they were trying to go fast. An unfortunate idiom I wrote a thousand times at a company that uses Lua and tries to go fast is

                local n = 0
                local t = {}
                -- to do an insert
                n = n + 1
                t[n] = blorp
                -- to delegate some inserts to another function
                n = insert_some_stuff(t,n)
              
              A less messy thing you could do is

                local table_insert = table.insert
              
              but this is not as great, because lua will still do a binary search to find the end of the array each time you insert something.
              • megameter 3 days ago

                Lua's weak point is around doing things with arrays, IME. It's just not very fast when you actually need something simple.

                I actually started poking around at a language design over the past month that tries to leverage Lua as the compiler for something that can run in its own separate bytecode interpreter, with more of an emphasis on low level primitives. It started off as a simple adaptation of Forth ideas, but yesterday I started playing with a revision that shifts the data structure from plain old stack towards growable arrays containing three cursors(thus, "tricursor array") which can be purposed to describe bounds, destinations, insertion points, read and write, top of stack, etc. In Forth the top three values of the stack tend to get dedicated words for their manipulation because they are used quite often; this model runs with that idea plus methods I've often used for array data(text edits, sound sample loops, blotting sprites) and tries to extrapolate that into a language that can ease certain forms of array programming, while falling back on using the array as a data stack. Still just sketching it now.

                • fwsgonzo 3 days ago

                  That might be, but the C++ array append is bounds-checked which makes it half as fast as it could be too. It would be 16ns if there is no checking. So, you are right but I am trying to do a balanced approach here. These benchmarks are made for me, and I want to write (more or less) normal code.

                  I don't want to write local table_insert = ... I'd rather drop Lua for something else then. That said, it's cool that there are things you can do to speed things up if you really have to.

                  • anonymoushn 3 days ago

                    Fair enough! I think t[#t+1] = x is not as offensive and will get you a few nanos less than the way with two table lookups, but if that's also not to your taste then that's fine.

          • alexhutcheson 4 days ago

            LuaJIT:

            + Super easy FFI to C

            + Very fast (although profile to make sure this matters)

            PUC Lua:

            + Possible to run untrusted code (e.g. user scripts) in a sandbox

            + Will run on a huge variety of platforms, including many microcontrollers (only limiting factor is a C compiler and a mostly-complete libc)

            + Has some additional language features (although some features, including integers, are controversial)

            + Actively maintained, expected to receive new releases and updates

          • If you use LuaJIT, you can generate bindings to C libraries, versus handwriting bindings for PUC Lua. The productivity difference is staggering. Additions to Lua since 5.1.5 have not helped me write more or better software, as much as I love Lua.

            • otabdeveloper4 4 days ago

              > When both were on version 5.1, there didn't seem to be any downsides to LuaJIT.

              LuaJIT has (had?) weird memory limits due to its GC implementation. This matters if you're running scripts on a server.

              • kzrdude 4 days ago

                Every major release of Lua has always been a different language, and embedding in general encourages not upgrading. If you ship something based on Lua 5.2, how to handle all existing user scripts if you upgrade Lua?

                • upofadown 4 days ago

                  I think we do have to call this a legitimate language fork at this point. Prosody (XMPP server) runs on 5.2 and does not require LuaJIT. The LuaJIT thing is more of a symptom than a cause.

                • ufo 4 days ago

                  This release brings many performance improvements, including a new garbage collector.

                  In terms of language features, the biggest new feature are the "to be closed" variables. They can be used to ensure that cleanup routines are always called, similarly to RAII in C++ or try-finally blocks.

                  • anonymoushn 4 days ago

                    <close> is equivalent to golang's defer (either can be implemented in terms of the other) except at the block level. imo calling it RAII mostly leads to confusion. One mailing list user recently asked a bunch of questions around using <close> for RAII. They expected RAII to work even for resources that are never bound to a particular type of local variable, for example if one writes do_thing(io.open("foo.txt")), where do_thing is not responsible for closing files because sometimes it is used with files that will continue to be used. They eventually concluded that the closest thing to RAII available was __gc.

                    Some users presented separate complaints about resources kept in to-be-closed locals in coroutines not necessarily being closed. You can do something to try to solve this, like setting the __gc and __close metamethods of coroutines to call coroutine.close. A "real" solution would look more like dynamic-wind. Notably golang doesn't attempt to solve this one, so maybe not solving it is fine.

                    • pansa2 4 days ago

                      > <close> is equivalent to golang's defer [...] except at the block level.

                      Does that mean it’s the same as C#'s `using` and Python's `with`?

                      • anonymoushn 4 days ago

                        I wasn't deeply familiar with python's `with`, so I looked it up[0]

                        <close> differs from `with` in at least the sense that it doesn't have any equivalent to __enter__ and doesn't create its own block. It creates a local variable whose __close metamethod will be called when it goes out of scope. Since Lua has lexical scope at the block level rather than the function level, this works similarly to the way Python calls __exit__.

                        These snippets of Python and Lua are roughly equivalent. They both open a file, read one line, and close it, or do nothing if there was some error opening the file.

                          try:
                            with open("foo.txt") as f:
                              print(f.readline())
                            # f is now closed.
                          except:
                            pass
                        
                          local f,err = io.open("foo.txt")
                          if not err then
                            local f <close> = f
                            print(f:read("*l"))
                          end
                          -- f is now closed.
                        
                        C#'s `using`[1] seems much closer, except that it handles nulls by not trying to close them and lua's <close> does not include any such handling.

                        [0]: https://docs.python.org/3/reference/compound_stmts.html#the-...

                        [1]: https://docs.microsoft.com/en-us/dotnet/csharp/language-refe...

                • pansa2 4 days ago

                  There are many lightweight, embeddable scripting languages available, yet Lua remains dominant. This is despite the language being "quirky" at best (and "dreaded" at worst [0]).

                  Is this just inertia, or is there a technical reason for new projects to embed Lua rather than a different language?

                  [0] https://insights.stackoverflow.com/survey/2018#technology-_-...

                  • mhd 4 days ago

                    Tcl and various scheme dialects were popular, but they have a pretty unusual syntax, at least when viewed from a typical C/Algol-ish programmer. Lua just came at the right time and place and was more "usual" than that, also with a pretty small package and permissive license.

                    And I would say that this argument still holds true. For extending a game or application, I'd feel pretty bad forcing people to use JS, for example (unless it's EE, where the gloves are off). Wren might be an option, but it's certainly a lot less proven.

                    Also, there's LuaJIT, if you really need some performance in a small package.

                    StackOverflow might not be the best source for end-user scripting. (Never mind that I have my doubts about any statistic where Rust ends up winning the popularity contest.)

                    • TazeTSchnitzel 4 days ago

                      What better languages are there which are friendly to embedding? A significant part of Lua's popularity, AIUI, comes from it being easy to embed.

                      • pansa2 4 days ago

                        Squirrel [0] seems to have been used in a few places [1], but not many compared to Lua. There are also embeddable JavaScript implementations such as Duktape [2], and a lightweight Ruby in mruby [3].

                        [0] http://www.squirrel-lang.org/

                        [1] https://en.wikipedia.org/wiki/Squirrel_(programming_language...

                        [2] https://duktape.org/

                        [3] https://github.com/mruby/mruby

                        • I love Angelscript for that purpose (w/ C++). Very similar syntax to C++ (but w/ GC), very easy to expose classes, functions, and variables. All it takes is a quick compile & link. It also has a JIT compilation if you need it and ability to save/load the intermediate binary code for faster loading.

                          • bjoli 4 days ago

                            Guile scheme, if you have users that don't have the knee-jerk "ewww parentheses!" reaction. The threading model is vastly different from lua, but compared to regular lua you get a lot faster execution.

                            • srean 4 days ago

                              Guile is great and I enjoy how active it has been. However, it is no more that small embeddable lisp. Its embeddable of course, but not small by any means.

                              • bjoli 4 days ago

                                One you start wanting to run complex things over multiple interpreters in Lua (say, to use multiple threads) you might as well just use guile.

                                Sure, we are talking 2.5mb of library and about the same amount object code. Quite a bit larger than Lua. But that also gives you object code for working with texinfo (and quite a lot of completely unused, undocumented modules). I wonder how much could be stripped without anyone actually boticing

                            • rsecora 4 days ago

                              ChaiScript [1] For c++.

                              Almost trivial to add functions or calls in both directions.

                              - Header only project, with modern c++. Able to compile with and without mutex. Multiple context can run in parallel.

                              - Awesome integration and binding with c++ bars and functions.

                              There are two drawbacks. One is the maintenance and some performance quirks with returns and type conversions.

                              The other drawback, maintenance only [2]. the project accepts pull requests, but no recent activity on lenguaje or performance.

                              [1] http://chaiscript.com/

                              [2] https://github.com/ChaiScript/ChaiScript/commits?author=RobL...

                              • zeveb 4 days ago

                                Tcl leaps to mind. It is very easy to embed, too.

                                • salamanderman 4 days ago
                                • scythe 4 days ago

                                  Lua is weird, but it's competitors are also weird. Tcl, Scheme, Perl and Javascript are all at least as quirky if not more. JS in particular is at least as "dreaded". There are a few more "normal" alternatives, but they're less popular and have even smaller ecosystems than Lua, where package availability is already bad. Python and Ruby are huge by comparison and don't embed all that well.

                                  • bo1024 4 days ago

                                    Not sure why it’s dreaded, Lua is a joy to write code in.

                                    • RHSeeger 4 days ago

                                      > "quirky" at best (and "dreaded" at worst

                                      I think it depends heavily on the person. Tcl has this problem, too, where lots of people love it and lots of people are completely put off by it.

                                    • taf2 4 days ago

                                      I enjoy writing Lua... It's kind of somewhere between Javascript and Ruby in my mind...

                                      • leephillips 4 days ago

                                        The technical reason might be that a lot of people like working in this language, and consider it to be well designed.

                                        • IshKebab 4 days ago

                                          It's very easy to embed, and LuaJit is pretty fast.

                                        • anonymoushn 4 days ago

                                          One interesting change in this release is the move to a stackful VM. This creates a pretty small (edit: actually huge, 1978 stack frames in a lua 5.4 repl installed with luaver just now) limit on the depth of the Lua call stack. I wonder what motivated this decision.

                                          • shakna 4 days ago

                                            I believe, but am not certain, that the change was made to help speed up Lua. As an embedded language, and a language that runs on embedded devices, performance is one of the key criteria of design that goes into the language. The compile-to-bytecode step may have seen some improvement with the change.

                                            So instead of speculating on why the change happened, I've found a few bits of information around the limits that make me feel like it often simply won't be a problem.

                                            There's some interesting discussion around the limits across various platforms here [0].

                                            Whilst I'm not entirely clear on the motivation, even Lua running inside a browser under emcripten has a stack limit of over 4000, which is somewhat decent even for recursive functions.

                                            Whereas on Linux you seem to be able to tweak a config when building and safely have it in the region of 20,000, and quite possibly more.

                                            Even under a tight restriction, Lua seems to cope fairly well with it when recursing (the example crashed at 1758 recursions for a ulimit of 400).

                                            Whilst Python has a terrible limit of 1000 for the default recursion limit which is somewhat comparable to this aggressive test, Python also has no tail call elimination - but Lua does.

                                            I have a somewhat large Lua project (6000 lines of C, 10,000 lines of Lua) that is a recursive parser for a format that I passionately hate (think self-modifying LaTeX). With the default 5.4 limits it never hits a point where it crashes, though I had expected it to.

                                            [0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html

                                            • schwartzworld 4 days ago

                                              Doesn't Lua have tail call recursion built in to the language? Stack overflows shouldn't be an issue on properly written recursive functions.

                                              • shakna 4 days ago

                                                Yes, it has tail call elimination (which I think I mentioned). But you can still right badly behaved recursive functions that avoid tail call elimination, and that's when you'll hit your stack overflow.

                                                I have a few of those badly behaved recursive functions in the parser I mentioned at the end of my comment, and expected them to overflow, but they didn't hit the default limit.

                                              • anonymoushn 4 days ago

                                                Ah yeah, I checked just now and the number of allowed frames is quite large. I observed a smaller limit of around 200 with the default config of an earlier release candidate of 5.4.0 and just expected the final version to be similar.

                                              • wahern 4 days ago

                                                Citation? I don't think anything significant changed in that regard. Coroutines are still "stackless" in terms of the C stack, but "stackful" in terms of the language semantics. The virtual data stack is still clearly on the heap.

                                                Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.

                                              • moonchild 4 days ago

                                                Was it not a stack machine already?

                                                The embedding API explicitly exposes a stack, so I assumed that it was implemented that way.

                                                • anonymoushn 4 days ago

                                                  it had a stack, but that stack was on the heap. In lua 5.4 under normal circumstances, 20 frames of Lua call stack corresponds to 20ish frames of C call stack. In lua 5.3 the number of C stack frames was some smaller number unrelated to the depth of the Lua call stack.

                                                  • moonchild 4 days ago

                                                    Ah - so it uses the C native stack.

                                              • loustak 4 days ago

                                                I really like lua. Especially when it comes to game with Love2d beeing so wonderful. I wish more projects adopted it as a scripting interface...

                                                • donquichotte 4 days ago

                                                  Love2D is indeed excellent, and integrates well with the Tiled map editor. It makes making games almost as easy as Macromedia Flash did!

                                                  • slmjkdbtl 4 days ago

                                                    in my knowledge most tools with scripting support uses lua, there's not a lot of options (i'm also a big fan of love2d)

                                                    • pkukkapalli 4 days ago

                                                      I wanted to like Lua and Love2D, but struggled with writing code quickly because there's no editor autocompletion that I know of, so I spent a lot of time having to reference the docs. Do others just use it enough that they know the API by heart now?

                                                    • grillorafael 4 days ago

                                                      "const variables" makes me giggle a little

                                                      • shakna 4 days ago

                                                        The new random library implementation should stop me dragging in a Mersenne Twister for every other thing.

                                                        It was libc's random, which is... Trash.

                                                        Now it use's xoshiro256 [a], which I hadn't heard of. (Not my area). [0] It seems somewhat similar to the MT, as it isn't cryptographically safe, but should give you a fast and fairly random set of integers back. It's also significantly faster than the MT.

                                                        [a] Followed by two asterisks. I give up trying to make them appear in HN's formatting.

                                                        [0] http://prng.di.unimi.it/

                                                        • JoshuaRLi 3 days ago

                                                          Yeah, I was happy to see the switch away from C random/srandom or rand/srand.

                                                          Not my area either, but a little surprised why a PCG-family non-CSPRNG algorithm wasn’t used: https://www.pcg-random.org/posts/a-quick-look-at-xoshiro256....

                                                          PCG XSL RR 128/64 (MCG) could be used for 64-bit values (albeit with period 2^126 compared to xoroshiro256’s 2^256 − 1) with 128 bits. Half the state size than xoshiro256, and the performance might be comparable (would need benchmarking). LCG version if you want to trade PRNG quality for even more speed.

                                                          • nestorD 4 days ago

                                                            To give a very short summary, xoshiro256 is much faster but also has a much smaller period (however the period of the Mersenne Twister generator is way too large for most applications anyway).

                                                          • I just recently started using Lua through the sol2 library. I'm basically providing some flexibility through my input file to my users. I'm very surprised by both library and Lua itself. For my use, I don't feel a lot of performance hit, and gets a lot of flexibility.

                                                            • ausjke 4 days ago

                                                              many projects I am aware of stuck with 5.1 and luajit still and so do I. I still use lua5.1 myself these days.

                                                              • catalogia 4 days ago

                                                                [Unfortunately, IMHO] Lua "minor versions" tend to break compatibility with older versions, making it problematic to upgrade to new versions without breaking scripts written for older versions. I believe mpv is stuck on 5.2 due to this. Here's the list of incompatibilities for 5.4: https://www.lua.org/manual/5.4/manual.html#8

                                                                • sitkack 4 days ago

                                                                  Yeah, Lua doesn't use semver, which is ok. Just multiply by 10 and pretend it does. So this would be Lua 54.

                                                                  • pansa2 4 days ago

                                                                    Yes, Lua doesn’t do “minor versions”. You can think of 5.4.0 as 54.0.0, and 5.3.5 as 53.0.5.

                                                                  • kzrdude 4 days ago

                                                                    The only thing you're saying is that it's lamentable that they use the "5.4" version number for something that's not a minor version.

                                                                    That's what this is, this is the next major version of Lua.

                                                                    • catalogia 4 days ago

                                                                      I think new versions of a language, major or minor, breaking compatibility with old code is generally lamentable regardless of version scheme. The version scheme they chose is not the root of my disappointment. Because so many programs will likely never upgrade their version of Lua, this leads to fragmentation of the community.

                                                                    • topspin 4 days ago

                                                                      Likewise OpenResty is "stuck" on 5.1.

                                                                      • bungle 4 days ago

                                                                        "Stuck" might be a wrong word. OpenResty actually dropped support for Lua and is now LuaJIT only. LuaJIT is a fork of Lua 5.1, but it has features from 5.2 [1] (and from later versions where it sees fit). Because of this, Lua 5.1 is a golden standard of Lua world at the moment.

                                                                        [1] http://luajit.org/extensions.html#lua52

                                                                        • xvilka 4 days ago

                                                                          There is more modern alternative to LuaJIT though - MoonJIT [1]. Unlike the former it supports modern Lua standards.

                                                                          [1] https://github.com/moonjit/moonjit

                                                                          • spacechild1 4 days ago

                                                                            AFAICT, MoonJIT is just a fork of LuaJIT. Both support many 5.2 extensions. Additionally, MoonJIT also has a few Lua 5.3 extensions, but it is not fully compatible with Lua 5.3 or even Lua 5.2. Quote:

                                                                            "Note: this provides only partial compatibility with Lua 5.2 at the language and Lua library level. moonjit is API+ABI-compatible with Lua 5.1, which prevents implementing features that would otherwise break the Lua/C API and ABI (e.g. _ENV)."

                                                                            Some internal language changes in 5.3, like the new integer type, would require deep surgery in the JIT. This also seems to be the main (but not only) reason why Mike Pall doesn't have plans to ever support Lua 5.3. (https://www.reddit.com/r/lua/comments/2zutj8/mike_pall_luaji...)

                                                                        • nicoster 4 days ago

                                                                          <close> seems to be a nice feature. wish it could be ported to luajit.

                                                                        • Rochus 4 days ago

                                                                          And doesn't seem to suffer from it.