This is not mentioned in the article, but since it has performance implications for a ton of BEAM applications, it's probably relevant: OTP 27 will also include a new builtin JSON module, and I did a quick and dirty benchmark [1] of it against Elixir's (excellent) Jason library. It seems like it'll be a pretty good performance win for BEAM applications that work with JSON.
Arbitrary precision integers aren't necessarily that expensive to implement. It does require overflow checking for operations on regular fixed integers, and keeping track of type, but if you're doing that anyway (many languages are) then it's almost free; the bignum code only runs when it doesn't fit in a fixed integer. Common Lisp, Python, Haskell use big integers for all integers, too. And others, I'm sure. It can be optimized away by a compiler to just fixed integers, sometimes.
I don't think there's an explicit to large integers, other than memory? Just the benchmark needs to pick a size, and 1_262_000 digits is used for the benchmark.
And Erlang/OTP follows a pattern of not setting limits on things unless needed, but not necessarily being well optimized at large sizes. Big integers have been there forever? with no explicit size limit, but pretty slow for the last several decades.
If you need mathematical performance and parallel computing (SIMD or matrix multiplication) it might be worth using the NX library which is basically the elixir equivalent of numpy. Otherwise you could probably write maths stuff in rustler if there really is a performance issue. Generally it’s immutability slowing down the maths not the calculations.
Using Karatsuba fixes a problem I've been complaining about for awhile when I'd run into certain fake-believe leetcode style problem or just doing weird non-sensical math benchmarks across different languages.
If you're not familiar with Karatsuba checkout the following
I had a problem last year with opentelemetry.js where it started concatenating numbers as strings instead of summing them. The backend, which I believe is Python? Did not blow up until we hit somewhere north of 300 digits, which is still pretty good.
But a million digits is a pretty good BigInteger. What is that? Half a megabyte?
“OpenTelemetry.js” is written in typescript and they thought putting argument types on functions would protect them from bad inputs and thus “1” + 1 problems. Only none of that happens when calling it from JavaScript.
Somewhere in our half million line (I actually don’t know, because we didn’t have a monorepo) application someone turned a stat into a string and I never did track it down.
When we migrated from StatsD that became a problem.
I was going to say that OTP was certainly done, but it got a patch last month [1]. I don't expect many more? But just because OTP 22 is done, doesn't mean OTP is done. There's obviously lots of optimization available that can be done, as well as ports to future architectures, and the requirements aren't set in stone; it's got a TLS stack, so that's never done for example.
[1] https://zeroclarkthirty.com/2024-04-21-benchmarking-erlangs-...
I'm legitimately surprised with their performance achievements there.
What's the logic behind this?
> Running an unpatched Erlang/OTP 26 on my Intel-based iMac from 2017, the benchmark finishes in about 10 seconds.
> The same benchmark run using Erlang/OTP 26.0.2 finishes in about 0.4 seconds.
I love that Erlang even supports an integer with 1,262,000 digits, but I imagine that's one reason it'll never be a mathematical speed daemon.
Yep, SBCL does range analysis and can do inline fixnum or unboxed arithmetic.
And Erlang/OTP follows a pattern of not setting limits on things unless needed, but not necessarily being well optimized at large sizes. Big integers have been there forever? with no explicit size limit, but pretty slow for the last several decades.
>> A brief history of recent optimizations
>> Erlang/OTP 22 introduced a new SSA-based intermediate representation in the compiler.
Using Karatsuba fixes a problem I've been complaining about for awhile when I'd run into certain fake-believe leetcode style problem or just doing weird non-sensical math benchmarks across different languages.
If you're not familiar with Karatsuba checkout the following
* https://www.youtube.com/watch?v=frT1UPiJUO0
* https://www.youtube.com/watch?v=JCbZayFr9RE
But a million digits is a pretty good BigInteger. What is that? Half a megabyte?
Somewhere in our half million line (I actually don’t know, because we didn’t have a monorepo) application someone turned a stat into a string and I never did track it down.
When we migrated from StatsD that became a problem.
[1] https://www.erlang.org/patches/otp-22.3.4.27