Reminded me of a game engine of mine. Years ago I was experimenting with a time travel game concept. It had a virtual machine with green threads whose state could be suspended and serialized into a single binary blob at any time. I basically repurposed the GC's reachability analysis to find all objects reachable from the roots (VM's stack and global variables). IIRC it had also some sort of topological sort when serializing so that most references could be restored in one pass. It also serialized the stack itself and the current instruction pointer. "Manual persistence" they mention is actually quite straightforward to do if you're in full control of the VM. Additionally, it also serialized the physics engine's state as well, so that the forces applied were fully restored (important in a game). The concept was about being able to return to any point in time and have N versions of yourself working together to solve a problem. NPCs and objects had running scripts attached and that's why I needed the whole thing. It also allowed to have proper game saves out of the box. Oh, those were the fun times, and now I'm writing boring enterprise stuff :)
People will feel free to comment on the project as a whole anyway, so why go to the bother of making a special thing out of it? :)
I am surprised you didn't find more discussion, though. I'm pretty sure I first heard of the project and have seen it here multiple times. I think it's more often (or in the past) just "Spritely". If your search included the word "Goblins" instead of just "Spritely", that might have narrowed it down excessively.
Why don't we see more general purpose computing environments where the whole interpreter state is persistent and serializable by default? The only ones I'm aware of is R's .RData mechanism, some Smalltalk environments, and solutions at the container/VM level.
What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?
There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state."
They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%."
[1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/...
You generally only want to serialize a part of the program state.
I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another.
It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.
You'd still need a mechanism for persisting separately, or you'd end up with a broken image/"data layer" due to post-crash persists.
It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.
This is great! I've been following the Spritely project on the sidelines for a while now and have built a couple toy applications with Goblins; persistence and IO were often my biggest pain points. Looking forward to giving this a try soon!
This blog post commits the common sin of showing a screencapture of a terminal emulator session that moves at the speed of thought of the creator rather than presenting things in a reasonable way for the person actually consuming it (i.e. the one in the audience who's trying to work out, without the creator's a priori understanding of what they're doing, the relative significance/insignificance of whatever's flashing on the screen).
This happens constantly with software demos. It shouldn't. It's such a silly thing to mess up and takes more effort than just not including a screencapture whatsoever.
The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow.
Granted, I've played the space shooter in question before, so I do have some context :)
> The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow.
Working out that that's all that's happening (not to mentioning ask the inevitable, "Why is there a screencapture included here at all?") takes about 2–10x the amount of time it takes to read this description that says that's what it's showing.
Depends, I work daily with console, so I just reasoned that a sequence of:
- exit game
- console shows prompt for new command
- command magically appears fully defined
- game starts again
is a normal "quit, recall with arrow up, restart". It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking why those things are represented so. It's the first time I've heard of spritely goblins or seen that video.
This comment reeks of sophomoric-douchebag-leaping-at-the-chance-to-talk-down-to-someone-he-assumes-is-a-noob laced with confirmation cum survivorship/selection bias ("the thing I assumed was being shown is what was being shown").
> I work daily with console
"with console"? Aside from that—
Okay? So what? Me too.
> It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking
Really? Did you perform a survey? That "most developers" who spend their days inside a terminal emulator immediately grokked what the author was trying to communicate on the first viewing?
It's not evident that that's what's going on, because you can't see an up arrow keystroke. You see a flash of a command—too briefly to read—and even then it's not evident (even if you assume that it's simply an instance of up-arrow-followed-by-enter) that the command is merely a command to relaunch the game. Are there arguments being passed at the command line—and is there maybe something we can glean from them—or is it just a bare command? If upon multiple playthroughs of the looping GIF we see that it's the latter, is that because it's a script that encapsulates the interesting stuff?
For context, the GIF in question is presented as the payload to a sentence that begins, "Here's an example of our persistence system in action:". And the section immediately prior to that reads:
> Spritely Goblins has the answer, incorporating a powerful new persistence mechanism (codenamed "Aurie") which empowers you to save relevant parts of a running Goblins program and wake them up later! But that's not all -- you can also use Goblins' persistence system to upgrade your saved objects as they evolve, and it even helps making live hacking even more powerful since you can change objects and reload them with upgraded behavior live!
That's the context; at the point where the GIF is dumped into the post, I'm primed to see a demo of something a lot more involved than what's actually there. I'm primed to see a hacking session in a live programming system with a brand new persistence mechanism—with its own codename ("Aurie"). What we're actually rewarded with is a screencapture of a game just like any other game we've seen in the last 30+ years that saves when you quit and starts off where you left it.
I know that guile/scheme is uniquely well suited to this, but I'd be interested in hearing why you're using it instead of, say, writing a python library? Is it like a smalltalk thing with really good message-passing, are you getting some guarantees about serializability that are impossible in a more mainstream language? Do you just like it?
There's rather a lot of new syntax, and, more importantly, exploration and experimentation with new syntax involved. Python syntax is... not easy to extend. Let alone on an experimental basis.
How fast is Goblins? I've been considering using it for a project, but I'm worried that it would be too slow. Would it, e.g., be fast enough to serve as a database?
https://www.gdcvault.com/play/1012210/The-Implementation-of-...
Spritely Goblins v0.11.0, time travel distributed debugger and more - https://news.ycombinator.com/item?id=35864365 - May 2023 (1 comment)
Growing a Networked Garden with Spritely Goblins - https://news.ycombinator.com/item?id=34031136 - Dec 2022 (2 comments)
Maybe we should convert this thread to a discussion of the project as a whole, rather than of a point release?
I am surprised you didn't find more discussion, though. I'm pretty sure I first heard of the project and have seen it here multiple times. I think it's more often (or in the past) just "Spritely". If your search included the word "Goblins" instead of just "Spritely", that might have narrowed it down excessively.
What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?
There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state."
They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%."
[1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/...
I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another.
It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.
It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.
This happens constantly with software demos. It shouldn't. It's such a silly thing to mess up and takes more effort than just not including a screencapture whatsoever.
Granted, I've played the space shooter in question before, so I do have some context :)
Working out that that's all that's happening (not to mentioning ask the inevitable, "Why is there a screencapture included here at all?") takes about 2–10x the amount of time it takes to read this description that says that's what it's showing.
- exit game
- console shows prompt for new command
- command magically appears fully defined
- game starts again
is a normal "quit, recall with arrow up, restart". It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking why those things are represented so. It's the first time I've heard of spritely goblins or seen that video.
> I work daily with console
"with console"? Aside from that—
Okay? So what? Me too.
> It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking
Really? Did you perform a survey? That "most developers" who spend their days inside a terminal emulator immediately grokked what the author was trying to communicate on the first viewing?
It's not evident that that's what's going on, because you can't see an up arrow keystroke. You see a flash of a command—too briefly to read—and even then it's not evident (even if you assume that it's simply an instance of up-arrow-followed-by-enter) that the command is merely a command to relaunch the game. Are there arguments being passed at the command line—and is there maybe something we can glean from them—or is it just a bare command? If upon multiple playthroughs of the looping GIF we see that it's the latter, is that because it's a script that encapsulates the interesting stuff?
For context, the GIF in question is presented as the payload to a sentence that begins, "Here's an example of our persistence system in action:". And the section immediately prior to that reads:
> Spritely Goblins has the answer, incorporating a powerful new persistence mechanism (codenamed "Aurie") which empowers you to save relevant parts of a running Goblins program and wake them up later! But that's not all -- you can also use Goblins' persistence system to upgrade your saved objects as they evolve, and it even helps making live hacking even more powerful since you can change objects and reload them with upgraded behavior live!
That's the context; at the point where the GIF is dumped into the post, I'm primed to see a demo of something a lot more involved than what's actually there. I'm primed to see a hacking session in a live programming system with a brand new persistence mechanism—with its own codename ("Aurie"). What we're actually rewarded with is a screencapture of a game just like any other game we've seen in the last 30+ years that saves when you quit and starts off where you left it.