I was recently looking at porting a cool OpenBSD feature to FreeBSD and was wondering how the locking could possibly work in terms of ordering. It turns out that all the OpenBSD locks boiled down to a global netlock rw lock for the entire IPv4/IPv6 stack.
Usually when OpenBSD does something it's because there's a clean implementation.
OpenBSD is very seldom the first to actually do something; but they focus on making their code able to be understood by a single person.
Doing things is hard, doing them in a way that is clear, obvious and "simple" is even harder. So when OpenBSD finally manages to do something that others may have had for a while... it's a cause of celebration. Because if the implementation was a clean one it would have gone in much sooner.
Do you know of a write up that illustrates this extra simplicity by comparing OpenBSD’s implementation of something with a previous mainstream implementation?
I would say compare the OpenBSD documentation for ground up install to network serving to any other documentation source for the same process.
OpenBSD will definitely be straight forward and easy to follow all from a single "manual". I'm sure there are other OSs that compare favorable to OpenBSD but those other OSs would also be high quality.
It is insanely difficult to implement it. The idea sounds common sense, but the details are extremely difficult to get right and any mistake causes random crashes, or incorrect data (read security holes).
That is one way to solve it. However it isn't always a good answer. Mutexs - by definition - mean that you stop everyone else from doing something when you are doing it, which means if two threads are trying to do something one will be stopped. It is also very easy to forget to lock/unlock a mutex at the right time, both of which will cause problems. Mutexs also send a message out to other CPUs which means even if no other thread wants the mutex you still slowed your system down. In the real world mutexs become painfully slow when more than a few threads are accessing the same shared data.
There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics.
The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.
Right, but if done right you rarely are in a situation two threads want to share the thread, so in practice you are not. However doing this right is difficult.
Yeah, but most of the things that need exclusivity or consistency are relatively rare or very narrow.
You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept)
You need exclusivity when send or recieve on tcp sockets. (Very narrow)
You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though.
For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.
If it's insanely difficult to implement in a monolithic kernel, where all the data structures are easily available and shared, then it must be plain impossible to implement in a microkernel!
Since microkernels are not popular (at least not open source ones, and the closed source often don't give enough data to know) that isn't even meaningful. We would need a full-features microkernel based OS that tried to do it correctly. Otherwise it could be easier to do, but nobody tried.
I deeply appreciate that they look like they should be easier, but several decades of people trying to make them work have mostly failed which makes me think there's some deeper problem that we're not seeing.
The problem is difficult no matter what. Many many more people are working on monokernels and so it is within the data we have to say that micro kernels are easier. There are not enough people to work on microkernels and there is a lot to do.
I like openbsd for what it is, but they do have a tendency to imply or outright say that the thing they just did is something they invented.
Like when they launched W^X (which isn't even an accurate term, because that implies that read-only non-execute pages don't exist), they basically claimed to have invented it. And they said that it was "impossible" to do on 32bit x86. Which was a surprise for those of us who had been running that on 32bit x86 for years, using grsec patches.
They defended themselves for this mis-statement by saying that they don't look at what other OSs, like Linux, do. Fair enough, but don't say you're the first to do it, then.
OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded. And since they have fewer developers (and not a focus on performance), they've understandably been lagging behind in this space.
> "OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded."
Which seems like an entirely reasonable choice if they felt there were bigger priorities. OpenBSD received SMP support in November 2004, right at the same time as the first mainstream dual-core CPU arrived on the market. I get what you mean by "late to the game", but I also feel that it's not really a fitting statement.
Yes, I said they focus on other things. It's not criticism. But it is true that they dismissed SMP, and did not see dual-core coming. They were late because they were reacting to dual core launching.
> I also feel that it's not really a fitting statement.
In what way? Linux got it almost a decade before that. Windows NT got it more than a decade before OpenBSD.
SunOS approaching 15 years before OpenBSD.
ChatGPT claims Mac OS X got it in 1999 with Mac OS X Server, but I could not immediately confirm that one.
Again, this is not criticism. But I don't see how it would be inaccurate to say they were late, when I think they were literally last.
They were last, but not late. SMP in the x86 world was miniscule in terms of multi-CPU setups. It didn't "happen" until the multi-core era, and OpenBSD was well in time for that.
For OpenBSD it's significant and somewhat of a milestone because its network stack has been trailing a bit in terms of transfer speeds when compared to e.g. Linux. The kernel has had a lot of "big locks" that hamper executional performance in complex ways across many different venues, and the past few years the developers have been busy cutting these into smaller pieces.
And like OpenSSH, due to the BSD licensing, it will be quickly be ported to other operating systems. Which is an advantage to us all, a great thing for humanity.
...I was going to make a joke about Windows servers, but actually AFAIK Windows just uses OpenSSH for its SSH server so that kinda proves the point doesn't it?
Maybe this will be the return of an OpenBSD home router for me. I ran one for years because it was a fun project but when gigabit became available in my area I learned a low power, passively cooled cpu struggled with gigabit traffic on OpenBSD.
Throughput depends also on how large (and how well/poorly phrased) your PF rule-set is. Up until 2021 or so my own OpenBSD home router ran on some 14nm era low-power passively cooled Celeron I picked up in 2015, and I managed to get about 700 Mbit/s through that thing. These days anything just slightly snappier will get you 1 Gbit/s throughput.
Sure, something like ZFS would be nice, but in a router not needed as you'd mount the fs rw only during update, wouldn't you? Logs are either on a separate, RAM based fs (tmpfs) or (preferably) sent to a remote logging host.
FFS can crash quite safely. The chance of lost data is incredibly small, especially with "softdep" out of the way. It just doesn't come back online as fast as a journaled file system will.
It's literally "just" them making their kernel (finally) multi-core instead of locking everything every time a process does networking.
Pretty much every other OS out there has done that for years, if not decades.
(FreeBSD for almost 20 years, although arguably at first that release was quite unstable.)
OpenBSD is very seldom the first to actually do something; but they focus on making their code able to be understood by a single person.
Doing things is hard, doing them in a way that is clear, obvious and "simple" is even harder. So when OpenBSD finally manages to do something that others may have had for a while... it's a cause of celebration. Because if the implementation was a clean one it would have gone in much sooner.
OpenBSD will definitely be straight forward and easy to follow all from a single "manual". I'm sure there are other OSs that compare favorable to OpenBSD but those other OSs would also be high quality.
The big difference on those is a very limited scope.
There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics.
The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.
You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept)
You need exclusivity when send or recieve on tcp sockets. (Very narrow)
You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though.
For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.
Like when they launched W^X (which isn't even an accurate term, because that implies that read-only non-execute pages don't exist), they basically claimed to have invented it. And they said that it was "impossible" to do on 32bit x86. Which was a surprise for those of us who had been running that on 32bit x86 for years, using grsec patches.
They defended themselves for this mis-statement by saying that they don't look at what other OSs, like Linux, do. Fair enough, but don't say you're the first to do it, then.
OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded. And since they have fewer developers (and not a focus on performance), they've understandably been lagging behind in this space.
Which seems like an entirely reasonable choice if they felt there were bigger priorities. OpenBSD received SMP support in November 2004, right at the same time as the first mainstream dual-core CPU arrived on the market. I get what you mean by "late to the game", but I also feel that it's not really a fitting statement.
> I also feel that it's not really a fitting statement.
In what way? Linux got it almost a decade before that. Windows NT got it more than a decade before OpenBSD.
SunOS approaching 15 years before OpenBSD.
ChatGPT claims Mac OS X got it in 1999 with Mac OS X Server, but I could not immediately confirm that one.
Again, this is not criticism. But I don't see how it would be inaccurate to say they were late, when I think they were literally last.
They were last, but not late. SMP in the x86 world was miniscule in terms of multi-CPU setups. It didn't "happen" until the multi-core era, and OpenBSD was well in time for that.
Anyway, this is not a productive semantics debate. Have a nice weekend.
Hey, I’d personally love if something openbsd did got it some wider industrial use.
You might want to have a look at e.g. https://www.mimar.rs/blog/how-to-increase-openbsds-resilienc...