Up until about GPT 2, EURISKO was arguably the most interesting achievement in AI. Back in the day on the SL4 and singularitarian mailing lists, it was spoken of in reverent tones, and I’m sure I remember a much younger Eliezer Yudkowsky cautioning that Doug Lenat should have perceived a non-zero chance of hard takeoff at the moment of its birth. I suspect its achievements were slightly overblown and heavily guided by a human hand, but it’s still fascinating and definitely worthy of study. Genetic programming hasn’t yielded many interesting results since, and the unreasonably effectiveness of differentiable programming and backpropagation has sucked up much of the oxygen in the room. But not everything is differentiable, the combination of the two still seems worth investigating, and EURISKO goes to show the power of heuristic approaches to some problems.
> the combination of the two still seems worth investigating
This.
Back in the late 1980's and early 90's the debate-du-jour was between deliberative and reactive control systems for robots. I got my Ph.D. for simply saying that the entire debate was based on the false premise that it had to be one or the other, that each approach had its strengths and weaknesses, and that if you just put the two together the whole would be greater than the sum of its parts. (Well, it was a little more than that. I had to actually show that it worked, which was more work that simply advancing the hypothesis, but in retrospect it seems kinda obvious, doesn't it?)
If I were still in the game today, combining generative-AI and old-school symbolic reasoning (which has also advanced a lot in 30 years) would be the first thing I would focus my attention (!) on.
People have advanced that argument a lot, and it's often worked for a short while; then the statistical models get better.
Chess was a game for humans.
It was very briefly a game for humans and machines (Kasparov had a go at getting "Advanced Chess" off the ground as a competitive sport), but soon enough having a human in the team made the program worse.
But at least the evaluation functions were designed by humans, right? That lasted a remarkably long time; first Stockfish became the strongest engine in the world by using distributed hyperparameter search to tweak its piece-square tables, then AlphaZero came along and used a policy network + MCTS instead of alpha-beta search, then (with an assist from the Shogi community) Stockfish struck back with a completely learned evaluation function via NNUE.
So the last frontier of human expertise in chess is search heuristics, and that's going to fall too: https://arxiv.org/abs/2402.04494.
The common theme with all of this is that the stuff which we used before are, fundamentally, hacks to get around _not having enough compute_, but which make the system worse once you don't have to make those tradeoffs around inductive biases. Empirical evidence suggests that raw scaling has a long way to run yet.
I find myself not wanting to agree with you, but deep down I think you're right.
AI greatly reminds me of the Library of Babel thought experiment. If we can imagine a library with every book that can possibly be written in any language, would it contain all human knowledge lost in a sea of noise? Is there merit or value in creating a system that sifts through such a library to attune hidden truths, or are we dooming ourselves to finding meaning in nothingness?
In a certain sense, there's immense value to developing concepts and ideas through intuition and thought. In another sense, a rose by any other name smells just as sweet; if an AI creates a perpetual motion device before a human does, that's not nothing. I don't expect AI to speed past human capability like some people do, but it's certainly displaced a lot of traditional computer-vision and text generation applications.
> If we can imagine a library with every book that can possibly be written in any language, would it contain all human knowledge lost in a sea of noise? Is there merit or value in creating a system that sifts through such a library to attune hidden truths, or are we dooming ourselves to finding meaning in nothingness?
The work that system would be required to find those "hidden truths" is equivalent to re-deriving those truths from scratch.
Similar argument: an image is just a number; if you take e.g. a 800x600 24bpp picture, that's a number 1 440 000 bytes long; you could hypothetically start from 0 and generate every 1 440 000-byte number, thus generating every possible 800x600 24bit image. In that set, you'd find every historical event, photographed at every moment from every angle, and even photos of every fragment of every book from the Library of Babel. But good luck finding anything particular in there.
Similar argument 2: any movie or song is contained somewhere within digital expansion of the number Pi. But again, it's worthless unless you know how to find such works, which basically requires you to have them in the first place.
Maybe. The statistical models are definitely better at natural language processing now, but they still fail on analytical tasks.
Of course, human brains are statistical models, so there's an existence proof that a sufficiently large statistical model is, well, sufficient. But that doesn't mean that you couldn't do better with an intelligently designed co-processor. Even humans do better with a pocket calculator, or even a sheet of paper, than they do with their unaided brains.
If human brains are statistical models, why are human brains so bad at statistics?
Edt: btw, same for probabilistic inference, same for logical inference, and same for any other thing anyone's tried as the one true path to AI since the 1950's. Humans have consistently proven bad at everything computers are good at, and that tells us nothing about why humans are good at anything (if, indeed, we are). Let's not assume too much about brains until we find the blueprint, eh?
That depends on what you mean by being "bad at statistics." What brains do on a conscious level is very different than what they do at a neurobiological level. Brains are "bad at statistics" on the conscious level, but at the level of neurobiology that's all they do.
As an analogy, consider a professional tennis or baseball player. At the neurobiological level those people are extremely good at finding solutions to kinematic equations, but that doesn't mean that they would ace a physics test.
That is a very big assumption -that brains have conscious and subconscious levels that are good and bad at different things- that needs to be itself proved, before it can be used to support any other line of inquiry.
I'm not well versed in the relevant literature at all but my understanding is that research in the area points to the completely opposite direction: that humans e.g. playing baseball do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball etc.
This makes a lot more sense, not only for humans playing tennis, but for animals surviving in the wild, finding sustenance and shelter, and mates, while avoiding becoming a meal. Consider the Portia spider [1], a spider-hunting spider, itself prey to other hunting spiders, with a brain consisting of a few tens of thousands of neurons and still perfectly capable not only of navigating complex environments in all three space dimensions but also making complex plans involving detours.
Just think of how quickly a spider must be able to think that hunts, and is hunted by other spiders -some of the most deadly predators in the animal kingdom. There is no chance of a snowball in hell that such an animal has the time to solve kinematic equations with a few KBs of neurons. Absolutely no chance at all.
For that and many other stuff like that it looks very unlikely to me that human brains, or any brains, are like you say. In any case, that sounds positively Freudian and I don't mean that as an insult, but I so could.
______________
[1] My favourite. No, I don't mean meal. I just love this paper; it's almost the best paper in autonomous robotics and planning that I've ever read:
> That is a very big assumption -that brains have conscious and subconscious levels that are good and bad at different things- that needs to be itself proved, before it can be used to support any other line of inquiry.
You can't be serious. Do you really doubt that hand-eye coordination and solving systems of kinematic equations on paper using math are disjoint skills? That one can be good at one without being good at the other? That there is in actual fact an inverse correlation between these skills? How do you account for the fact that even people who have never studied math or physics can learn to throw and catch a ball?
Sorry about that, I'm dealing with a troll on another thread so I'm on a bit of a hair trigger.
I think we have a fundamental disconnect somewhere, so let's try to diagnose it. Where do you start to disagree in the following series of claims:
1. People can have kinematic skills, like throwing and catching balls, without having math or physics skills, like solving kinematic equations.
2. In order to have kinematic skills, something in your brain must be doing something that can be equated by some mapping to solving kinematic equations, because the actions that your muscles perform when performing kinematic skills are the solutions to kinematic equations, so your brain must be producing those (things that map to) solutions somehow.
3. As far as we can tell, brains don't operate symbolically at the neurobiological level. Individual neurons operate according to laws having to do with electrical impulses, synapse firings, neurotransmitters, etc. none of which have anything to do with kinematics.
4. People with kinematic skills generally have only limited insight into how they do what they do when they apply those skills. Being able to catch a ball doesn't by itself give you enough insight to be able to describe to someone how to build a machine that would catch a ball. But someone with math and physics and engineering skills but no kinematic skills (your streotypical geek) could plausibly build a machine that could catch a ball much better than they themselves could. But the workings of a machine built using knowledge of math would almost certainly operate in a very different manner than the brain of a human with kinematic skills.
I think I'll stop there and ask if there is anything you disagree with so far.
It's great to read conversation of towering HN experts in the field.
Lisper, as I understand this part -
> In order to have kinematic skills, something in your brain must be doing something that can be equated by some mapping to solving kinematic equations
you're talking about an equivalent of YeGoblynQueenne's
> that humans ... do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball
So to me the question is, is it correct? Can "mapping to solve kinematic equation" be the same as "simple heuristic... like placing hands in from of eyes"?
Physically this equivalence seems at least plausible.
Now, about
> neurons operate according to laws having to do with electrical impulses
- can't we have those kinematic equations solving, or, in other words, applying simple heuristics, as a trained combination of such neuronal activity?
Let's go back to the original formulation so we don't lose the plot here:
Me: As an analogy, consider a professional tennis or baseball player.
YeGoblynQueenne: humans e.g. playing baseball do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball etc.
At the risk of stating the obvious, being a professional tennis or baseball player involves a lot more than "simple heuristics ... like placing their hands in front of their eyes so that they line up with the ball." That simple heuristic might work for one specific skill -- catching a ball that happens to be heading in your direction. But it won't help much for moving a bat or a raquet in such a way that it will hit a ball moving past you at close to 100mph in such a way that the ball ends up traveling on some desired trajectory.
But even just moving your hand in front of your eyes is nowhere near as trivial as YeGoblynQueenne implies. To do that you have to control seven degrees of freedom: two at your shoulder, two at your elbow, and two at your wrist. Solving those kinematic equations even to find a static solution is elementary but non-trivial, a skill that is solidly at the undergraduate level.
Now consider running to catch a ball. That involves controlling about 20 or 30 degrees of freedom (two arms, two legs, neck, waist, two eyes...) in real time in a situation that involves not just kinematics but also dynamics. Solving that analytically was an unsolved research problem for a long time (maybe still is, I haven't been keeping up with recent developments). A child can learn to do it. But they do have to learn to do it. It's not a skill humans are born with.
It seems pretty obvious to me that the process of learning how to catch a ball while running is very different than the process of learning how to do math. And yet, there must be a mapping between them because the movements required for catching a ball are the solutions to kinematic equations.
> being a professional tennis or baseball player involves a lot more than "simple heuristics ...
Mmm, a combination of simple heuristics, all of which are of course learned, but still simple heuristics, could in itself be a simple heuristic. Yet it could allow performing pretty complex-looking actions, including those you described. Simple heuristic here could be a linear or low degree polynomial approximation of a good solution to kinematic equation - not precise, but enough to get to the goal, while learnable and explainable. But still without actual full-blown abstract, mathematically correct complete solution.
What's meant by heuristics is some times unclear. I wonder if by heuristics you mean a shortcut. In CS and AI our model of a shortcut is the heuristic cost functions in heuristic search algorithms like A* and its variants.
It's interesting because I've thought along the following lines. A* is a pathfinding algorithm so it's a natural choice for path planning -the process of planning a path through some environment for an autonomous agent to follow. The funny thing is, as it turns out, pathfinding can be abstracted as finding a "path" through a graph: a set of nodes connected by edges; and that's a great abstraction for general task planning - the task of achieving any arbitrary objective - so A* is also widely used for task planning.
Well, isn't path planning an almost universal ability of intelligent animals? Most animals are motile for some part of their lives and they seem to use their intelligence at the very least to navigate their environment. So is it that far-fetched to think that an ancestral ability for path planning, essentially identical to a heuristic search algorithm like A*, evolved into general intelligence? And wouldn't that mean that general intelligence can be, ultimately, modeled as some kind of heuristic search?
The answer I think is: no, and that's a dangerous way to think. A model is a model, it's not the process it models. And I think that's my fundamental disagreement with lisper, disregarding my confusion about the meaning of "kinematics".
>> But even just moving your hand in front of your eyes is nowhere near as trivial as YeGoblynQueenne implies.
Yeah, I was actually thinking of kinematics as in classical mechanics. I think you were speaking about kinematic equations as in robotics. My bad, I misunderstood.
I agree that moving your hand in the right place is not a simple problem, and I don't actually have an insight into that, but I think it's easier than calculating the trajectory of an object, let alone many at once (think juggling). Maybe that's another source of our disagreement- but see my comment about having multiple models for a process.
> Yeah, I was actually thinking of kinematics as in classical mechanics. I think you were speaking about kinematic equations as in robotics.
What do you see as the relevant difference?
> I think it's easier than calculating the trajectory of an object, let alone many at once
Well, yeah, but there isn't anything fundamentally more difficult about juggling. It all boils down to Newton's laws.
My point is that there are two different ways that human brains can apply Newton's laws. We can do it intuitively, without even being consciously aware of Newton's laws, which is why humans were able to throw and catch objects before 1687. Or we can do it consciously by manipulating symbolic representations of the equations of motion. Those two activities are in some sense equivalent because they both involve producing a model of a physical system in our brains and using that model to make accurate predictions about that system. But they are also obviously radically different in other ways, and being skilled at one in now way implies being skilled at the other.
I'm not an expert in either, so I'm possibly overemphasizing the difference.
[Edit: As far as I understand it, one is useful in predicting the movement of objects outside the body, the other the position of the limbs etc.]
>> My point is that there are two different ways that human brains can apply Newton's laws. We can do it intuitively, without even being consciously aware of Newton's laws, which is why humans were able to throw and catch objects before 1687. Or we can do it consciously by manipulating symbolic representations of the equations of motion. Those two activities are in some sense equivalent because they both involve producing a model of a physical system in our brains and using that model to make accurate predictions about that system. But they are also obviously radically different in other ways, and being skilled at one in now way implies being skilled at the other.
I totally agree with that. Can we agree that we can model whatever our brains do with kinematic equations, but we have no idea what is the true process that is being modeled?
OK, I'm an expert in both, so I can say the following with some authority:
No, we cannot model what our brains do with kinematic equations. Our brains operate according to the laws of neurobiology, which we do not yet fully understand, but which we know enough about to know that they bear absolutely no resemblance to the laws of kinematics. Your brain is not made of mechanical linkages.
Nonetheless, despite the fact that the laws of neurobiology and the laws of kinematics bear no resemblance to each other, our brains somehow manage to produce solutions to problems that require solving kinematic equations. Not only that, but our brains can do this in two completely different ways, one of which is conscious and deliberate (what we call "doing math") and the other of which is instinctive and subconscious (developing sensory-motor skills).
We get leverage out of doing math despite the fact that our brains can solve some of the same problems innately. Likewise, I believe that LLMs could get a lot of leverage if they were augmented with special-purpose modules for doing math and other specific tasks.
>> No, we cannot model what our brains do with kinematic equations.
I've confused you. My apologies. What I meant with this sentence:
"we can model whatever our brains do with kinematic equations"
Was that we can model whatever our brains do _while catching a ball etc_ by means of kinematic equations. I did not mean that we can model everything our brains do, i.e. the function of the brain, in general. If we could model an entire brain just by kinematic equations we wouldn't need any AI research, and I wouldn't be arguing that we don't know what our brains do when they solve problems that we solve using kinematic equations. Our disagreement is about the solutions our brain finds to that kind of problem.
>> Not only that, but our brains can do this in two completely different ways, one of which is conscious and deliberate (what we call "doing math") and the other of which is instinctive and subconscious (developing sensory-motor skills).
That's my problem with all this - the "subconscious" part. I don't really understand what it means. When I catch a ball, I do it entirely consciously, and I know exactly what I'm doing: I'm extending my hand to catch the ball. I may not be able to articulate every little muscle movement, or describe precisely the position of my arms, my hand, my fingers, the ball, etc, but I do know with great accuracy where those objects are in space, and where they are in relation with each other. I cannot introspect into the intellectual mechanisms by which I know those things, but I do know them, so they're not "subconscious".
The difference you point out, between doing maths with pen-and-paper (or computers) and performing a task without having to do maths-with-pen-and-paper, is, I think, the difference between having a formal language that is powerful enough to describe all the objects and functions I describe above (hand position, muscle movement etc), on the one hand, and not having such a language on the other hand. Somehow humans are able to come up with formal languages with the power to describe some of the things we do, like catching balls etc, and many other things besides. As a side note, we do not have a formal language -we do not have the mathematics- to describe our ability to come up with formal languages, yet. That was be one of the original goals of AI research, although it has now fallen by the wayside, in the process of chasing benchmark performance.
I digress. When I speak of "formal languages", I mean more broadly formal systems, like mathematics (of which logic is one branch, btw). When I speak of a "model" in my earlier comment, I mean a formalism that describes various kinds of human capability, like our catching-balls example. Kinematic equations, that's one such model. But a model is not the thing it, well, models. Is my claim.
I hope this is clear and apologies if it's not. Most of our discussion is not on things of my expertise so I'm trying to find the best way to say them. Also, this is a much less technical discussion and so much less precise, than I'm used to. I hope I'm not wasting your time with needless philosophising.
On the other hand, I think this kind of conversation would be made much easier if we didn't assume human brains. Our ability to navigate, and interact with, our environment, is shared to a greater or lesser extent with many animals that aren't humans and don't have human brains, so whatever we can do with our brains thanks to that shared ability, must also share an underlying system- because we all evolved from the same, very distant, animal ancestors, ultimately, and we must have inherited the same basic firmware as it were.
> Was that we can model whatever our brains do _while catching a ball etc_ by means of kinematic equations.
No, we can't even do that. All we can do is observe that the results of what our brains do happen to be the solutions to kinematic equations. It does not follow that we can model the process of producing those solutions by kinematic equations. It does not even follow that the process of producing those solutions bears any resemblance to what we do when we do math to find them.
Here is an analogy: we can observe that the motions of objects obeys the principle of least action [1] and that to compute the action we have to integrate the Lagrangian. It does not follow that there is anything happening in the physical mechanism that causes particles to move that is even remotely analogous to integrating a Lagrangian.
> When I catch a ball ... I know exactly what I'm doing
No, I don't think you do. If you did, you would be able to describe what you are doing to someone else, and they would be able to reproduce your actions based on that description alone. Alternatively, you would be able to render your knowledge into computer code and build a robot that could do it. But I doubt you can actually do either of those things if your only skill is catching a ball and you are not trained in math.
By way of very stark contrast, I am absolutely terrible at hand-eye coordination tasks, but I can build a machine that is much better at it than I am [2]. Just to be clear, I didn't actually build that particular machine, but I do know how. And so I can tell you that the process of learning how to build a machine that can catch a ball is radically different than the process of learning how to catch a ball yourself.
Sorry for the lag. Productive day yesterday and today my friendly neighbourhood
rock band was in a great mood early in the bloody morning.
>> No, we can't even do that. (...)
OK well I'm very confused. I thought our disagreement was on whether our brains
actually calculate actual kinematic equations, or just the same results by some
other means. It feels to me like we're arguing the same corner but we don't have
a common language.
>> No, I don't think you do. (...)
"I can't put my finger on it, but I know it when I see it". My claim is that
there is a difference between tacit knowledge, and articulable knowledge. I can
not articulate the knowledge I have of how I am catching a ball; but I certainly
know how I catch a ball, otherwise I wouldn't be able to do it. In machine
learning, we replace explicit, articulable knowledge with examples that
represent our tacit knowledge. I might not be able to manually define the
relation betwen a set of pixels and a class of objects that might be found in a
picture, but I can point to a picture that includes an image of a certain class
and label it, with the class. And so can everyone else, and that's how we get
tons of labelled examples to train image classifiers with, without having to
know how to hand-code an image classifier.
Here's a little thing I'm working on. Assume that, in order to learn any concept
we need two things: some inductive bias, background knowledge of the relevant
concepts; and "forward knowledge" of the target concept. In statistical machine
learning the inductive bias comes in the form of neural net architectures,
function kernels, Bayesian priors etc. and the knowledge of a target concept
comes in the form of labelled examples. Now, there are four learning settings;
tabulating:
Background Target Error
---------- -------- -----
Known Known Low
Known Unknown Moderate
Unknown Known Moderate
Unknown Unknown High
Where "Error" is the error of a learned hypothesis with respect to the target
theory. In the first setting, where we have knowledge of both the background and
the target, and the error is low, we're not even learning anything: just
calculating. We can equally well match the first three settings to deductive,
inductive, and abductive reasoning. You can also replace "known" and "unknown"
with "certain" and "uncertain".
Now, I'd say that the invention of kinematic equations by which we can model the
way we move our hands to catch balls etc is in the setting where the background
theory and the target are both known: the background being our theory of
mathematics, and the target being some obsrvations about the behaviour of humans
catching balls. I don't know if the kinematic equations you speak of where
really derived from such observations, but they could have. Humans are very good
at modelling the world in this way.
We're in deep trouble when we're in the last setting, where we have no idea of
the right background theory nor the target theory. And that's not a problem
solved by machine learning. We only make progress in that kind of problem very
slowly, with the scientific method, and it can take us thousands of years,
during which we're stuck with bad models. For 15 centuries, the model is
epicycles, until we have the laws of planetary motion and universal gravitation.
And, suddenly, there are no more epicycles.
This also adressses your earlier comment about betting against a scientific
upheaval in the science of computation.
Cool machine, btw, in that video. So you're a roboticist? I work on machine
learning of autonomous behaviour for mobile robotics.
> It feels to me like we're arguing the same corner but we don't have a common language.
That's possible. It's actually a deep philosophical question. Do planets "solve Newton's equations of motion" when they move? On the one hand, they move in ways that correspond to solutions to those equations, and so one could say that they "find solutions" to those equations. On the other hand, the process by which they do this is pretty clearly radically different than what a mathematician does when they solve equations.
> So you're a roboticist?
I used to be. I've been out of the field for over 20 years now. But back in the day I was pretty well known.
>> That's possible. It's actually a deep philosophical question. Do planets "solve Newton's equations of motion" when they move?
Yes, that's an interesting question- that I'm really not equipped to answer. Probably for the best.
>> I used to be. I've been out of the field for over 20 years now. But back in the day I was pretty well known.
I'm really new to the field so I don't know your work. In fact I wouldn't even say I am in the field as such. An academic sibling suggested I take a post doc job and now I'm collaborating with roboticists. I'm just working on autonomous behaviour- I'm not allowed near hardware.
It's an interesting field although I have to constantly be on my toes to avoid violating my principles. See I'm a peacenick, but it seems with the work I do, as soon as I got that stuff working, someone will want to put it on a drone, strap a gun on its back and send it to kill people. And I'm dead set against that sort of thing.
I had a quick look at your site and you've worked with NASA. Respect! We can send autonomous rovers to explore far away planets and people want to keep them here wreaking havoc and death. Unbelievable.
Do you have any pointers to your work? Something you are really proud of that you did in the past? I'm curious.
>> Sorry about that, I'm dealing with a troll on another thread so I'm on a bit of a hair trigger.
Hey, no worries. Thanks for being a gentleman and I'm sorry you're being harassed. Btw, just to be clear: I'm perfectly fine with robust disagreement, I just don't deal well with personal attacks; which you didn't do, I was just worried that's where this conversation was going.
So, thanks for the very detailed analysis of your argument. That indeed makes it much simpler to find common ground. Here's where I disagree: point number 2!
Here's why. It's obvious to me that it's entirely possible to have two distinct models of the same process that compute almost identical results, so it's entirely possible for humans to be using a completely different process to catch balls etc, than kinematic equations.
And here's why I think this is likely: first, because of the point I made above about computational complexity and second because of the observed wide variability in the uh, let's say kinematic capabilities of different humans. If we were all solving kinematic equations, we would all have the same skills. What's more: humans can be wildly inaccurate in their motions (I know I am; don't leave coffee cups on my desk), while robots for example, are distinctly not. That also points to a different computation.
So, to summarise my argument: what we do needs neither be the same computational process, nor be computing the same results, as kinematic equations.
Btw, I'm a bit confused because I thought you were talking about kinematics in classical mechanics, but now I think you're talking about kinematics in robotics, with muscle actions etc. But I think both apply, except the robotics equations are I think much easier to solve than the classical mechanics ones, which I suspect may veer off into the chaotic.
Edit: I had more here on my _agreement_ to your point number 4, but I'm cutting it down to shorten the comment. You don't have all day :)
In any case, I think we just can't say for sure what our brains do, until we can say for sure.
> it's entirely possible for humans to be using a completely different process to catch balls etc, than kinematic equations.
This all turns on what you mean by "completely different". Yes, obviously when you learn to actually catch a ball your brain is not doing anything that maps straightforwardly onto the kinds of symbolic manipulations that happen when you do math. On the other hand, it has to map onto doing math somehow even if that mapping is not straightforward. The only other possibility is that your brain is actually doing something that doesn't map onto math in any way, but still somehow produces the same results that math does by sheer coincidence. If you could actually demonstrate that, it would be one of the biggest breakthroughs in the history of science because it would refute the Church-Turing thesis.
To be honest I suspect the next scientific revolution would be a refutation of Church-Turing, or maybe something more like an extension of it to phenomena we are not closely studying yet, a bit like my understanding of the relation between Newtonian mechanics, and General Relativity and Quantum mechanics. Unfortunately that won't be me bringing that revolution about, so you won't get to say you exchanged views with a scientific legend :P
For the time being of course we can agree that our brains probably do some kind of maths, as far as we understand it. I'm guessing the way we understand maths has everything to do with the way our brains understand maths because, well, that's my position in our disagreement. But, see, I can do maths by counting on my fingers, so the question is really what kind of maths we're talking about and how complex can they realistically be. My argument is that if it's not the kind of maths a standard human being can calculate very quickly without pen or paper, then that's a no-go, because that leaves plenty of time to be eaten by a sabretooth, or what have you.
> That is a very big assumption -that brains have conscious and subconscious levels that are good and bad at different things- that needs to be itself proved, before it can be used to support any other line of inquiry.
Does this assumption itself need to be proven?
Besides, it's not true: you can simply define it as an assumption within a thought experiment and proceed merrily along, or you can just not bother to consider whether one's premises are true in the first place, and proceed merrily along.
The second option tends to be more popular in my experience, perhaps because it is so much easier, and perhaps for some other reasons also.
> If human brains are statistical models, why are human brains so bad at statistics?
If CPUs are made of silicon, why are they so bad at simulating semiconductors? Or why CPUs are so bad at emulating CPUs?
If JavaScript runs on a CPU, why is it so bad at doing bitwise stuff?
Etc.
What the runtime is made of is entirely separate of what's running on it. Same is with human brain (substrate) and human consciousness (software), or humans (substrate) and bureaucracy (runtime) and corporations (software).
Your question implies it is obvious that a system of statistical models would (or should) be good at statistics. And that the opposite is a paradox. I would ask why you think that is obvious?
Being good at statistics is more of a knowledge graph of understanding concepts than a statistical model, I think.
That's the "bitter lesson", right? Which is really a sour lesson- as in sour grapes. See, Rich Sutton's point with his Bitter Lesson is that encoding expert knowledge only improves performance temporarily, which is eventually surpassed by more data and compute.
There are only two problems with this: One, statistical machine learning systems have an extremely limited ability to encode expert knowledge. The language of continuous functions is alien to most humans and it's very difficult to encode one's intuitive, common sense knowledge into a system using that language [1]. That's what I mean when I say "sour grapes". Statistical machine learning folks can't use expert knowledge very well, so they pretend it's not needed.
Two, all the loud successes of statistical machine learning in the last couple of decades are closely tied to minutely specialised neural net architectures: CNNs for image classification, LSTMs for translation, Transformers for language, Difussion models and Ganns for image generation. If that's not encoding knowledge of a domain, what is?
Three, because of course three, despite point number two, performance keeps increasing only as data and compute increases. That's because the minutely specialised architectures in point number two are inefficient as all hell; the result of not having a good way to encode expert knowledge. Statistical machine learning folk make a virtue out of necessity and pretend that only being able to increase performance by increasing resources is some kind of achievement, whereas it's exactly the opposite: it is a clear demonstration that the capabilities of systems are not improving [2]. If capabilities were improving, we should see the number of examples required to train a state-of-the-art system either staying the same, or going down. Well, it ain't.
Of course the neural net [community] will complain that their systems have reached heights never before seen in classical AI, but that's an argument that can only be sustained by the ignorance of the continued progress in all the classical AI subjects such as planning and scheduling, SAT solving, verification, automated theorem proving and so on.
For example, and since planning is high on my priorities these days, see this video where the latest achievements in planning are discussed (from 2017).
See particularly around this point where he starts talking about the Rollout IW(1) symbolic planning algorithm that plays Atari from screen pixels with performance comparable to Deep-RL; except it does so online (i.e. no training, just reasoning on the fly):
[1] Gotta find where this paper was but none other than Vladimir Vapnik basically demonstrated this by trying the maddest experiment I've ever seen in machine learning: using poetry to improve a vision classifier. It didn't work. He's spent the last 20 years trying to find a good way to encode human knowledge into continuous functions. It doesn't work.
[2] In particular their capability for inductive generalisation which remains absolutely crap.
It sounds kinda crazy (is there really that much far transfer?), but you know, I think it would work... He just needed to use LLMs instead: https://arxiv.org/abs/2309.10668#deepmind
Yeah, that's one of the papers in that line of research by Vapnik. He's got a few with similar content. Visually, it's not the paper I remember, I'll have to read it again to be sure.
If I remember correctly, Vapnik's point is, we know that Big Data Deep Learning works; now, try to do the same thing with small data. Very much like my point that capabilities of models are not improving, only the scale increasing.
> The language of continuous functions is alien to most humans and it's very difficult to encode one's intuitive, common sense knowledge into a system using that language
In other words; machine learned models are octopus brains (https://www.scientificamerican.com/article/the-mind-of-an-oc...) and that creeps you out. Fair enough, it creeps me out too, and we should honour our emotions — I'm no rationalist – but we should also be aware of the risks of confusing our emotional responses with reality.
Please don't god mode me? Machine learning doesn't creep me out. I'm sorry it creeps you out. In my culture, octopus is a prized delicacy, my dad used to fish them out of the sea with his bare hands when I was a kid. If you wanna creep me out, you should try snake, not octopus.
>Two, all the loud successes of statistical machine learning in the last couple of decades are closely tied to minutely specialised neural net architectures: CNNs for image classification, LSTMs for translation, Transformers for vision, Difussion models and Ganns for image generation. If that's not encoding knowledge of a domain, what is?
Transformers, Diffusion for Vision, Image generation are really odd examples here. None of those architectures or training processes are tuned for Vision in mind lol. It was what? 3 years after Attention 2017 before the famous Vit paper. CNNs have lost a lot of favor to Vits, LSTMs are not the best performing translators today.
The bitter lesson is that less encoding of "expert" knowledge results in better performance and this has absolutely held up. The "encoding of knowledge" you call these architectures is nowhere near that of the GOFAI kind and even more than that, less biased NN architectures seem to be winning out.
>That's because the minutely specialised architectures in point number two are inefficient as all hell; the result of not having a good way to encode expert knowledge.
Inefficient is a whole lot better than can't even play the game, the story of GOFAI for the last few decades.
>If capabilities were improving, we should see the number of examples required to train a state-of-the-art system either staying the same, or going down. Well, they ain't.
The capabilities of models are certainly increasing. Even your example is blatantly wrong. Do you realize how much more data and compute it would take to train a Vanilla RNN to say GPT-3 level performance?
>> Inefficient is a whole lot better than can't even play the game, the story of GOFAI for the last few decades.
See e.g. my link above where GOFAI plays the game (Atari) very well indeed.
Also see Watson winning Jeopardy (a hybrid system, but mainly GOFAI - using frames and Prolog for knowledge extraction, encoding and retrieval).
And Deep Blue beating Kasparov. And MCTS still the SOTA search algo in Go etc.
And EURISCO playing Traveller as above.
And Pluribus playing Poker with expert game-playing knowledge.
And the recent neuro-symbolic DeepMind thingy that solves geometry problems from the maths olympiad.
etc. etc. [Gonna stop editing and adding more as they come to my mind here.]
And that's just playing games. As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems and neural nets suck at them. Ask Yan LeCun: "Machine learning sucks". He means it sucks in all the things that classical AI does best and he means he wants to do them with neural nets, and of course he'll fail.
It's often forgotten that Rich Sutton said the two things which work are learning (the AlphaGo/Leela Zero policy network) and search (MCTS). (I think the most interesting research in ML is around the circumstances in which large models wind up performing implicit search.)
That was a figure of speech. I didn't literally mean games (not that GOFAI performs better than NNs in those games anyway). I simply went off your own examples - Vision, Image generation, Translation etc.
>As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems
You can use NNs for all these things. It wouldn't make a lot of sense because GOFAI would be perfect and the former would be inefficient but you certainly could which is again more than I can say for GOFAI and the domains you listed.
As it is, your comment seems to tell me that neural nets are good at neural net things and GOFAI is good at GOFAI things, which is obvious, and is what I'm saying: neural nets can make only very limited use of expert knowledge and so suck in all domains where domain knowledge is abundant and abundantly useful, which are the same domains where GOFAI dominates. GOFAI can make very good use of expert knowledge but is traditionally not as good in domains where only tacit knowledge is available, because we don't understand the domain well enough yet, like in anything to do with pattern recognition, which is the same domains where neural nets dominate. If explicit, expert knowledge was available for those domains, then GOFAI would dominate, and neural nets would fall behind, completely contrary to what Sutton thinks.
So, the bitter lesson is only bitter for those who are not interested in what classical AI systems can do best. For those of us who are, the lesson is sweet indeed: we're making progress, algorithmic progress, progress in understanding, scientific progress, and don't need to burn through thousands of credit to train on server farms to do anything of note. That's even a running joke in my team: hey, do you need any server time? Nah, I'll run the experiment on my laptop over lunch. And then beat the RL algo (PPO) that needs three days training on GPUs. To solve mazes badly.
Hey, og_kalu, I vouched for your comment but it stays dead. It's not you, it was me who was out of line, with my comment: "wishful thinking"; that's not a very polite thing to say. And my original comment was a bit prissy, too.
To be honest, I'm always a bit jumpy around your comments because I've noticed them all over the place and they're often grayed-out. You kind of tend to go for the jugular. I don't mean that as a good thing. I think others have noticed it too and you get more reaction than you should. That's a shame, because it's clear there's lots of interesting conversations to be had, given you have such strong views and you seem to have done quite a bit of reading; though only on one side of things.
Anyway sorry for starting it this time around and that you got dead'ed, I hope we get to disagree more in the future.
Yeah, all of those architectures are _themselves_ hacks to get around having insufficient compute! They absolutely were encoding inductive biases into the network to get around not being able to train enough, and transformers (handwaving hard enough to levitate, the currently-trainable model family with the least inductive bias) have eaten the world in all domains.
This is evidence _for_ the Bitter Lesson, not against it.
They haven't (eaten the world etc). They just happen to be the models that trend hard right now. I bet if you could compare like for like you'd be able to see some improvement in performance from Transformers, but that 'd be extremely hard to separate from the expected improvement from the constantly increasing amounts of data and compute. For example, you could, today, train a much bigger and deeper Multi-Layered Perceptron than you could thirty years ago, but nodoy is trying because that's so 1990's, and in any case they have the data and compute to train much bigger, much more inefficient (contrary to what you say if I got that right) architectures.
Wait a few years and the Next Big Thing in AI will come along, hot on the heels of the next generation of GPUs, or tensor units or whatever the hardware industry can cook up to sell shovels for the gold rush. By then, Transfomers will have hit the plateau of diminishing returns, there'll be gold in them there other hills and nobody would talk of LLMs anymore because that's so 2020s. We've been there so many times before.
The tricky part here is that "efficiency" is not a single dimension! Transformers are much more "efficient" in one sense, in that they appear to be able to absorb much more data before they saturate; they're in general less computationally efficient in that you can't exploit symmetries as hard, for example, at implementation time.
Let's talk about that in terms of a concrete example: the big inductive bias of CNNs for vision problems is that CNNs essentially presuppose that the model should be translation-invariant. This works great — speeds up training and makes it more stable – until it doesn't and that inductive bias starts limiting your performance, which is in the large-data limit.
Fully-connected NNs are more general than transformers, but they have _so many_ degrees of freedom that the numerical optimization problem is impractical. If someone figures out how to stabilize that training and make these implementable on current or future hardware, you're absolutely right that you'll see people use them. I don't think transformers are magic; you're entirely correct in saying that they're the current knee on the implementability/trainability curve, and that can easily shift given different unit economics.
I think one of the fundamental disconnects here is that people who come at AI from the perspective of logic down think of things very differently to people like me who come at it from thermodynamics _up_.
Modern machine learning is just "applications of maximum entropy", and to someone with a thermodynamics background, that's intuitively obvious (not necessarily correct! just obvious) –in a meaningful sense the _universe_ is a process of gradient descent, so "of course" the answer for some local domain models is maximum-entropy too. In that world view, the higher-order structure is _entirely emergent_. I'm, by training, a crystallographer, so the idea that you can get highly regular structure emerging from merciless application of a single principle is just baked into my worldview very deeply.
Someone who comes at things from the perspective of mathematical logic is going to find that worldview very weird, I suspect.
>> Let's talk about that in terms of a concrete example: the big inductive bias of CNNs for vision problems is that CNNs essentially presuppose that the model should be translation-invariant. This works great — speeds up training and makes it more stable – until it doesn't and that inductive bias starts limiting your performance, which is in the large-data limit.
I don't know about that, I'll be honest. Do you have a reference? I suspect it won't disagree with what I'm saying, that neural nets just can't use strong enough bias to avoid overfitting. I didn't say that in so many words, above, but that's the point of having a good inductive bias, that you're not left, as a learner, to the mercy of the data.
>> Someone who comes at things from the perspective of mathematical logic is going to find that worldview very weird, I suspect.
No that's absolutely a standard assumption in logic :) Think of grammars; like Chomsky likes to say, human language "makes infinite use of finite means" (quoting Wilhelm von Humboldt). Chomsky of course believes that human language is the result of a simple set of rules, very much like logical theories. Personally, I have no idea, but Chomsky consistently and even today pisses off all the linguists and all the machine learning people, so he must be doing something right.
Btw, I'm not coming from the perspective of mathematical logic, only. It's complicated, but, e.g. my MSc was in data science and my PhD in a symbolic form of machine learning. See, learning and logic, or learning and reasoning, are not incompatible, they're fundamentally the same.
To clarify what I mean on this specific bit: the SOTA results in 2D and 3D vision, audio, translation, NLP, etc are all transformers. Past results do not necessarily predict future performance, and it would be absurd to claim that an immutable state of affairs, but it's certainly interesting that all of the domain-specific architectures have been flattened in a very short period of time.
Thanks for clarifying. Well, my argument is that the state of the art is more the result of trends in research than of the true capabilities of different approaches.
Take my little rant about Rich Sutton's (a god, btw) Bitter Lesson with respect to RL. So, there's AlphaGo, AlphaZero and μZero, yes? AlphaGo knows the rules of Go and starts with some expert knowledge, and beats very human Go player. AlphaZero knows the rules of Go but has no expert knowledge and it beats AlphaGo. And μZero neither knows the rules of Go, nor has expert knowledge, and it beats AlphaZero, and can also plays chess, shoggi and Atari games, with one hand while eating a banana. Do you know how hard it is to eat a banana with one hand? Unpeeled!
Easy to draw a conclusion from that. Except all those systems were developed and used by DeepMind, and there are very few entities besides DeepMind that can even train them, so all we know is what DeepMind claims and we have no way to check their claims. For example, can I test different configurations of μZero, with and without knowledge of the rules of the game and expert knowledge? Not really. And it's clear to me that DeepMind are pushing very, very hard a form of AI that relies on having gigantic resources, like the ones the just completely coincidentally happen to be among the few entities to have access to. So I remain unconvinced.
(I need to re-read the μZero paper, it's in my pdf buffer. I didn't get it the first time I read it, and it might well be that they did make sufficient ablation studies to convince even me and I just don't remember it).
> Up until about GPT 2, EURISKO was arguably the most interesting achievement in AI.
I'm really baffled by such statement and genuinely curious.
How come that studying GOFAI as undergraduate and graduate at many European universities, doing a PhD. and working in the field for several years _never_ exposed me to EURISKO up until last week (thanks to HN)?
I heard about Cyc, many formalism and algorithms that related to EURISKO, but never heard of its name.
It was featured in a BBC radio series on AI made by Colin Blakemore [1] around 1980, the papers on AM and EURISKO were in the library of the UK university that I attended.
So? There's a real possibility DART has still saved its customers more money over its lifetime than GPT has, and odds are basically 100% that your yoga teacher and IT colleagues haven't heard a thing about it either. The general public has all sorts of wrong impressions and unknown unknowns of facts that I don't see why they should ever be used as a technology industry benchmark by anyone not working in the UI department of a smartphone vendor.
"... I’m sure I remember a much younger Eliezer Yudkowsky cautioning that Doug Lenat should have perceived a non-zero chance of hard takeoff at the moment of its birth."
Also, in 2009 someone suggested re-implementing Eurisko[1], and Yudkowsky cautioned against it:
> This is a road that does not lead to Friendly AI, only to AGI. I doubt this has anything to do with Lenat's motives - but I'm glad the source code isn't published and I don't think you'd be doing a service to the human species by trying to reimplement it.
To my mind -- and maybe this is just the benefit of hindsight -- this seems way too overcautious on Yudkowsky's part.
Machinery can be a lot simpler than biology. Birds are incredibly complex systems: wing structure, musculature, feathers, etc. An airplane can be a vaguely wing-shaped piece of metal and a pulse jet. It doesn’t seem super implausible that there is some algorithm that is to human consciousness what a pulse jet with wings is to a bird. Maybe LLMs are that, but maybe they’re far more than is really needed because we don’t yet know what we are doing.
I would bet against it being possible to implement consciousness on a PDP, but I wouldn’t be very confident about it.
> a much younger Eliezer Yudkowsky cautioning that Doug Lenat should have perceived a non-zero chance of hard takeoff at the moment of its birth
Why is Yudkowsky taken seriously? This stuff is comparable to the "LHC micro black holes will destroy Earth" hysteria.
There are actual concerns around AI like deep fakes, a deluge of un-filterable spam, mass manipulation via industrial scale propaganda, mass unemployment created by widespread automation leading to civil unrest, opaque AIs making judgements that can't be evaluated properly, AI as a means of mass appropriation of work and copyright violation, concentration of power in large AI companies, etc. The crackpot "hard takeoff" hysteria only distracts from reasonable discourse about these risks and how to mitigate them.
Perhaps we can disagree on the shape of the curve, but it seems likely that ever more capable AI will enable ever more serious harms. Absolutely true that we should counter those harms in the present and not fixate on a theoretical future, but the medicine is much the same either way.
Trivialities Annoyances Immediate harm X-Risk
|------------------------------------------------------|
\----stuff you mention-------/
\---stuff Eliezer------/
wrote about
> The crackpot "hard takeoff" hysteria only distracts from reasonable discourse about these risks and how to mitigate them.
IDK, I feel endless hand-wringing about copyright and deepfakes distract from risks of actual, significant harm at scale, some of which you also mentioned.
> "LHC micro black holes will destroy Earth" hysteria.
I will be heavily downvoted for this, but here is how I remember it:
1) LHC was used to study blackholes and prove things like Hawking radiation
2) LHC was supposed to be safe due to Hawking radiation (that was only an unproven theory at the time)
So the unpopular question: what if Hawking radiation didnt actually exist?
Wouldnt there be a risk of us dying? A small risk, but still some risk? (especially as the potential micro black hole would have the same velocity as earth, so it wouldnt fly away somewhere into space)
On a side note: how would EURISCO evaluate this topic?
Since I read about this secretive CYC (why u can email asking for it, but source not hosted anywhere?): couldnt any current statistics based AI be used to feed this CYC program / database with information? Take a dictionary and ask ChatGPT to fill it with information for each word.
The fundamental reason that hysteria was silly is that Earth is bombarded by cosmic rays that are far stronger than anything done in the LHC. The reason we built the LHC is so we can do observable repeatable experiments at high energies, not to reach energies never reached on Earth before.
The AI hysteria I'm talking about here is the "foom" hysteria, the idea that a sufficiently powerful model will start self-improving without bound and become some kind of AI super-god. That's about as wild as the LHC will make a black hole that will implode the Earth. There are fundamental reasons to believe it's impossible, such as the question of "where would the information come from to drive that runaway intelligence explosion?"
There are legitimate risks with AI, but not because AI is somehow special and magical. All technologies have risks. If you make a sharper stick, someone will stab someone with it. Someday we may make a stick so sharp it stabs the entire world (cue 50s sci-fi theremin music).
Edit: for example... I would argue that the Internet itself has X-risks. The Internet creates an environment that incentivizes an arms race for attention grabbing, and the most effective strategies usually rely on triggering negative emotions and increasing division. This could run away to the point that it drives, say, civilizational collapse or a global thermonuclear war. Does this mean it would have been right to ban the Internet or require strict licensing to place any new system online?
You remember ... wrongly. It's just another particle accelerator, its intention was not to produce micro black holes for study.
You shouldn't use "theory" when it comes to science unless you know what that means. Gravity is a "theory." "Theory" means that it has a working model, comes with a ton of observational evidence in line with predictions, and it has yet to be replaced by anything better. Outside of math, nothing is ever proven. Any leading scientific theory is, at best, "yet to be disproven." And it stays in the lead until something better comes along: more accurate, extending over a greater domain, etc.
Hawking radiation has yet to be observed.
And if you're worried about micro black holes, well, even an iron atom has a non-zero chance of tunneling to a micro black hole state. No collider needed.
Cyc isn't secretive, it's proprietary, the way the Microsoft codebase is, the Adobe codebase is, and so on.
People like religion, particularly if it doesn't affect how they live their life _today_ too much. You get all of the emotional benefits of feeling like you're doing something virtuous without the effort of actually performing good works.
The confluence of happenstance that occurs to make this a reality is pretty amazing to witness.
Unfortunately it starts with the passing of Douglas Lenat. But that enabled Stanford to open up their 40 year old archive, which they still had, of Lenats work.
Somehow, someway, someone not only stumbled upon EURISKO, but also knew what it was. One of the most notorious AI research projects of the age that actually broke out of the research labs of Stanford and out into the public eye, with impactful results. Granted, for arguably small values of “public” and “impactful”, but for the small community it affected, it made a big splash.
Lenat used EURISKO to find a very unconventional winning configuration to go on to win a national gaming tournament. Twice.
In that community, it was a big deal. The publisher changed the rules because of it, but Lenat returned victorious again the next year. After a discussion with the game and tournament sponsors, he never came back.
Apparently EURISKO has quite a reputation in the symbolic AI world, but even there it was held close.
But now it has been made available. Not only made available, but made operational. EURISKO is written in an obsolete Lisp dialect, Interlisp. But, coincidentally, we have today machine simulators that can run versions of that Lisp on long lost, 40 year machines.
And someone was able to port it. And it seems to run.
The thought of the tendrils through time that had to twist their way for us to get here leaves, at least me, awestruck. So much opportunity for the wrong butterfly to have been stepped on to prevent this from happening.
But it didn’t, and here we are. Great job by the spelunkers who dug this up.
Enough of the Traveller tournament story is dodgy and inconsistent enough that it's very hard to say what actually happened beyond Lenat winning the tournament twice in a row with some kind of computer assistance,
Basically, with the Traveller tournament Lenat appears to have stumbled onto a story that caught the public's imagination, and then through the milked it for all he could to give his project publicity and to make it appear more
successful than it actually was. And if that required embellishing the story or just making shit up, well, no harm no foul.
Even when something is technically true, it often turns out that it's being told in a misleading way. For example, you say that "the publisher changed the ruleset". That was the entire gimmick of the Traveller TCS tournament rules! The printed rulebook had a preset progression of tournament rules for each year.
EURISKO is basically a series of genetic algorithms over lisp code - the homoiconic nature of lisp making it effectively a meta-optimizer. Amongst many problems was that the solution space, even for things like "be interesting and true", was way too large.
Eurisko (Gr., I discover) is a discovery system written by Douglas Lenat in RLL-1, a representation language itself written in the Lisp programming language. A sequel to Automated Mathematician, it consists of heuristics, i.e. rules of thumb, including heuristics describing how to use and change its own heuristics
It got a lot of kudos for winning a multi player naval wargame by building a bizarre but successful fleet that exploited all the loopholes and quirks in the rules.
IIRC it had at least one small (?) purely defensive boat that couldn’t be destroyed by typical weapons so its parent fleet couldn’t be defeated. It wasn’t like a modern drone swarm
It makes me think of the battles in Doc Smith’s Lensman series where the Galactic Patrol would develop a game-breaking fleet formation to use against Boskone in every major naval battle.
Ah yes, probably need to re-read them. I remember how virtually every book introduces a new super weapon that becomes part of the standard arsenal in the next, all the way up to entire 'negative matter' planets that are fired out of subspace (can't recall the in-universe name) at the planets hosting your opponent's base.
Can anyone give a clear example of how this can be used productively? Its description doesn't help much.
What can one do with EURISKO? The fact of its recovery after its authors passing is interesting, in and of itself - but why is EURISKO, specifically, worth the effort of understanding?
This.
Back in the late 1980's and early 90's the debate-du-jour was between deliberative and reactive control systems for robots. I got my Ph.D. for simply saying that the entire debate was based on the false premise that it had to be one or the other, that each approach had its strengths and weaknesses, and that if you just put the two together the whole would be greater than the sum of its parts. (Well, it was a little more than that. I had to actually show that it worked, which was more work that simply advancing the hypothesis, but in retrospect it seems kinda obvious, doesn't it?)
If I were still in the game today, combining generative-AI and old-school symbolic reasoning (which has also advanced a lot in 30 years) would be the first thing I would focus my attention (!) on.
Chess was a game for humans.
It was very briefly a game for humans and machines (Kasparov had a go at getting "Advanced Chess" off the ground as a competitive sport), but soon enough having a human in the team made the program worse.
But at least the evaluation functions were designed by humans, right? That lasted a remarkably long time; first Stockfish became the strongest engine in the world by using distributed hyperparameter search to tweak its piece-square tables, then AlphaZero came along and used a policy network + MCTS instead of alpha-beta search, then (with an assist from the Shogi community) Stockfish struck back with a completely learned evaluation function via NNUE.
So the last frontier of human expertise in chess is search heuristics, and that's going to fall too: https://arxiv.org/abs/2402.04494.
The common theme with all of this is that the stuff which we used before are, fundamentally, hacks to get around _not having enough compute_, but which make the system worse once you don't have to make those tradeoffs around inductive biases. Empirical evidence suggests that raw scaling has a long way to run yet.
AI greatly reminds me of the Library of Babel thought experiment. If we can imagine a library with every book that can possibly be written in any language, would it contain all human knowledge lost in a sea of noise? Is there merit or value in creating a system that sifts through such a library to attune hidden truths, or are we dooming ourselves to finding meaning in nothingness?
In a certain sense, there's immense value to developing concepts and ideas through intuition and thought. In another sense, a rose by any other name smells just as sweet; if an AI creates a perpetual motion device before a human does, that's not nothing. I don't expect AI to speed past human capability like some people do, but it's certainly displaced a lot of traditional computer-vision and text generation applications.
The work that system would be required to find those "hidden truths" is equivalent to re-deriving those truths from scratch.
Similar argument: an image is just a number; if you take e.g. a 800x600 24bpp picture, that's a number 1 440 000 bytes long; you could hypothetically start from 0 and generate every 1 440 000-byte number, thus generating every possible 800x600 24bit image. In that set, you'd find every historical event, photographed at every moment from every angle, and even photos of every fragment of every book from the Library of Babel. But good luck finding anything particular in there.
Similar argument 2: any movie or song is contained somewhere within digital expansion of the number Pi. But again, it's worthless unless you know how to find such works, which basically requires you to have them in the first place.
Maybe. The statistical models are definitely better at natural language processing now, but they still fail on analytical tasks.
Of course, human brains are statistical models, so there's an existence proof that a sufficiently large statistical model is, well, sufficient. But that doesn't mean that you couldn't do better with an intelligently designed co-processor. Even humans do better with a pocket calculator, or even a sheet of paper, than they do with their unaided brains.
Edt: btw, same for probabilistic inference, same for logical inference, and same for any other thing anyone's tried as the one true path to AI since the 1950's. Humans have consistently proven bad at everything computers are good at, and that tells us nothing about why humans are good at anything (if, indeed, we are). Let's not assume too much about brains until we find the blueprint, eh?
That depends on what you mean by being "bad at statistics." What brains do on a conscious level is very different than what they do at a neurobiological level. Brains are "bad at statistics" on the conscious level, but at the level of neurobiology that's all they do.
As an analogy, consider a professional tennis or baseball player. At the neurobiological level those people are extremely good at finding solutions to kinematic equations, but that doesn't mean that they would ace a physics test.
I'm not well versed in the relevant literature at all but my understanding is that research in the area points to the completely opposite direction: that humans e.g. playing baseball do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball etc.
This makes a lot more sense, not only for humans playing tennis, but for animals surviving in the wild, finding sustenance and shelter, and mates, while avoiding becoming a meal. Consider the Portia spider [1], a spider-hunting spider, itself prey to other hunting spiders, with a brain consisting of a few tens of thousands of neurons and still perfectly capable not only of navigating complex environments in all three space dimensions but also making complex plans involving detours.
Just think of how quickly a spider must be able to think that hunts, and is hunted by other spiders -some of the most deadly predators in the animal kingdom. There is no chance of a snowball in hell that such an animal has the time to solve kinematic equations with a few KBs of neurons. Absolutely no chance at all.
For that and many other stuff like that it looks very unlikely to me that human brains, or any brains, are like you say. In any case, that sounds positively Freudian and I don't mean that as an insult, but I so could.
______________
[1] My favourite. No, I don't mean meal. I just love this paper; it's almost the best paper in autonomous robotics and planning that I've ever read:
https://www.frontiersin.org/journals/psychology/articles/10....
You can't be serious. Do you really doubt that hand-eye coordination and solving systems of kinematic equations on paper using math are disjoint skills? That one can be good at one without being good at the other? That there is in actual fact an inverse correlation between these skills? How do you account for the fact that even people who have never studied math or physics can learn to throw and catch a ball?
And yes, I'm serious. Can you please be less confrontational?
I think we have a fundamental disconnect somewhere, so let's try to diagnose it. Where do you start to disagree in the following series of claims:
1. People can have kinematic skills, like throwing and catching balls, without having math or physics skills, like solving kinematic equations.
2. In order to have kinematic skills, something in your brain must be doing something that can be equated by some mapping to solving kinematic equations, because the actions that your muscles perform when performing kinematic skills are the solutions to kinematic equations, so your brain must be producing those (things that map to) solutions somehow.
3. As far as we can tell, brains don't operate symbolically at the neurobiological level. Individual neurons operate according to laws having to do with electrical impulses, synapse firings, neurotransmitters, etc. none of which have anything to do with kinematics.
4. People with kinematic skills generally have only limited insight into how they do what they do when they apply those skills. Being able to catch a ball doesn't by itself give you enough insight to be able to describe to someone how to build a machine that would catch a ball. But someone with math and physics and engineering skills but no kinematic skills (your streotypical geek) could plausibly build a machine that could catch a ball much better than they themselves could. But the workings of a machine built using knowledge of math would almost certainly operate in a very different manner than the brain of a human with kinematic skills.
I think I'll stop there and ask if there is anything you disagree with so far.
Lisper, as I understand this part -
> In order to have kinematic skills, something in your brain must be doing something that can be equated by some mapping to solving kinematic equations
you're talking about an equivalent of YeGoblynQueenne's
> that humans ... do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball
So to me the question is, is it correct? Can "mapping to solve kinematic equation" be the same as "simple heuristic... like placing hands in from of eyes"?
Physically this equivalence seems at least plausible.
Now, about
> neurons operate according to laws having to do with electrical impulses
- can't we have those kinematic equations solving, or, in other words, applying simple heuristics, as a trained combination of such neuronal activity?
Me: As an analogy, consider a professional tennis or baseball player.
YeGoblynQueenne: humans e.g. playing baseball do not find solutions to kinematic equations, but instead use simple heuristics that exploit our senses and body configuration, like placing their hands in front of their eyes so that they line up with the ball etc.
At the risk of stating the obvious, being a professional tennis or baseball player involves a lot more than "simple heuristics ... like placing their hands in front of their eyes so that they line up with the ball." That simple heuristic might work for one specific skill -- catching a ball that happens to be heading in your direction. But it won't help much for moving a bat or a raquet in such a way that it will hit a ball moving past you at close to 100mph in such a way that the ball ends up traveling on some desired trajectory.
But even just moving your hand in front of your eyes is nowhere near as trivial as YeGoblynQueenne implies. To do that you have to control seven degrees of freedom: two at your shoulder, two at your elbow, and two at your wrist. Solving those kinematic equations even to find a static solution is elementary but non-trivial, a skill that is solidly at the undergraduate level.
Now consider running to catch a ball. That involves controlling about 20 or 30 degrees of freedom (two arms, two legs, neck, waist, two eyes...) in real time in a situation that involves not just kinematics but also dynamics. Solving that analytically was an unsolved research problem for a long time (maybe still is, I haven't been keeping up with recent developments). A child can learn to do it. But they do have to learn to do it. It's not a skill humans are born with.
It seems pretty obvious to me that the process of learning how to catch a ball while running is very different than the process of learning how to do math. And yet, there must be a mapping between them because the movements required for catching a ball are the solutions to kinematic equations.
> being a professional tennis or baseball player involves a lot more than "simple heuristics ...
Mmm, a combination of simple heuristics, all of which are of course learned, but still simple heuristics, could in itself be a simple heuristic. Yet it could allow performing pretty complex-looking actions, including those you described. Simple heuristic here could be a linear or low degree polynomial approximation of a good solution to kinematic equation - not precise, but enough to get to the goal, while learnable and explainable. But still without actual full-blown abstract, mathematically correct complete solution.
It's interesting because I've thought along the following lines. A* is a pathfinding algorithm so it's a natural choice for path planning -the process of planning a path through some environment for an autonomous agent to follow. The funny thing is, as it turns out, pathfinding can be abstracted as finding a "path" through a graph: a set of nodes connected by edges; and that's a great abstraction for general task planning - the task of achieving any arbitrary objective - so A* is also widely used for task planning.
Well, isn't path planning an almost universal ability of intelligent animals? Most animals are motile for some part of their lives and they seem to use their intelligence at the very least to navigate their environment. So is it that far-fetched to think that an ancestral ability for path planning, essentially identical to a heuristic search algorithm like A*, evolved into general intelligence? And wouldn't that mean that general intelligence can be, ultimately, modeled as some kind of heuristic search?
The answer I think is: no, and that's a dangerous way to think. A model is a model, it's not the process it models. And I think that's my fundamental disagreement with lisper, disregarding my confusion about the meaning of "kinematics".
Yeah, I was actually thinking of kinematics as in classical mechanics. I think you were speaking about kinematic equations as in robotics. My bad, I misunderstood.
I agree that moving your hand in the right place is not a simple problem, and I don't actually have an insight into that, but I think it's easier than calculating the trajectory of an object, let alone many at once (think juggling). Maybe that's another source of our disagreement- but see my comment about having multiple models for a process.
What do you see as the relevant difference?
> I think it's easier than calculating the trajectory of an object, let alone many at once
Well, yeah, but there isn't anything fundamentally more difficult about juggling. It all boils down to Newton's laws.
My point is that there are two different ways that human brains can apply Newton's laws. We can do it intuitively, without even being consciously aware of Newton's laws, which is why humans were able to throw and catch objects before 1687. Or we can do it consciously by manipulating symbolic representations of the equations of motion. Those two activities are in some sense equivalent because they both involve producing a model of a physical system in our brains and using that model to make accurate predictions about that system. But they are also obviously radically different in other ways, and being skilled at one in now way implies being skilled at the other.
I'm not an expert in either, so I'm possibly overemphasizing the difference.
[Edit: As far as I understand it, one is useful in predicting the movement of objects outside the body, the other the position of the limbs etc.]
>> My point is that there are two different ways that human brains can apply Newton's laws. We can do it intuitively, without even being consciously aware of Newton's laws, which is why humans were able to throw and catch objects before 1687. Or we can do it consciously by manipulating symbolic representations of the equations of motion. Those two activities are in some sense equivalent because they both involve producing a model of a physical system in our brains and using that model to make accurate predictions about that system. But they are also obviously radically different in other ways, and being skilled at one in now way implies being skilled at the other.
I totally agree with that. Can we agree that we can model whatever our brains do with kinematic equations, but we have no idea what is the true process that is being modeled?
No, we cannot model what our brains do with kinematic equations. Our brains operate according to the laws of neurobiology, which we do not yet fully understand, but which we know enough about to know that they bear absolutely no resemblance to the laws of kinematics. Your brain is not made of mechanical linkages.
Nonetheless, despite the fact that the laws of neurobiology and the laws of kinematics bear no resemblance to each other, our brains somehow manage to produce solutions to problems that require solving kinematic equations. Not only that, but our brains can do this in two completely different ways, one of which is conscious and deliberate (what we call "doing math") and the other of which is instinctive and subconscious (developing sensory-motor skills).
We get leverage out of doing math despite the fact that our brains can solve some of the same problems innately. Likewise, I believe that LLMs could get a lot of leverage if they were augmented with special-purpose modules for doing math and other specific tasks.
I've confused you. My apologies. What I meant with this sentence:
"we can model whatever our brains do with kinematic equations"
Was that we can model whatever our brains do _while catching a ball etc_ by means of kinematic equations. I did not mean that we can model everything our brains do, i.e. the function of the brain, in general. If we could model an entire brain just by kinematic equations we wouldn't need any AI research, and I wouldn't be arguing that we don't know what our brains do when they solve problems that we solve using kinematic equations. Our disagreement is about the solutions our brain finds to that kind of problem.
>> Not only that, but our brains can do this in two completely different ways, one of which is conscious and deliberate (what we call "doing math") and the other of which is instinctive and subconscious (developing sensory-motor skills).
That's my problem with all this - the "subconscious" part. I don't really understand what it means. When I catch a ball, I do it entirely consciously, and I know exactly what I'm doing: I'm extending my hand to catch the ball. I may not be able to articulate every little muscle movement, or describe precisely the position of my arms, my hand, my fingers, the ball, etc, but I do know with great accuracy where those objects are in space, and where they are in relation with each other. I cannot introspect into the intellectual mechanisms by which I know those things, but I do know them, so they're not "subconscious".
The difference you point out, between doing maths with pen-and-paper (or computers) and performing a task without having to do maths-with-pen-and-paper, is, I think, the difference between having a formal language that is powerful enough to describe all the objects and functions I describe above (hand position, muscle movement etc), on the one hand, and not having such a language on the other hand. Somehow humans are able to come up with formal languages with the power to describe some of the things we do, like catching balls etc, and many other things besides. As a side note, we do not have a formal language -we do not have the mathematics- to describe our ability to come up with formal languages, yet. That was be one of the original goals of AI research, although it has now fallen by the wayside, in the process of chasing benchmark performance.
I digress. When I speak of "formal languages", I mean more broadly formal systems, like mathematics (of which logic is one branch, btw). When I speak of a "model" in my earlier comment, I mean a formalism that describes various kinds of human capability, like our catching-balls example. Kinematic equations, that's one such model. But a model is not the thing it, well, models. Is my claim.
I hope this is clear and apologies if it's not. Most of our discussion is not on things of my expertise so I'm trying to find the best way to say them. Also, this is a much less technical discussion and so much less precise, than I'm used to. I hope I'm not wasting your time with needless philosophising.
On the other hand, I think this kind of conversation would be made much easier if we didn't assume human brains. Our ability to navigate, and interact with, our environment, is shared to a greater or lesser extent with many animals that aren't humans and don't have human brains, so whatever we can do with our brains thanks to that shared ability, must also share an underlying system- because we all evolved from the same, very distant, animal ancestors, ultimately, and we must have inherited the same basic firmware as it were.
No, we can't even do that. All we can do is observe that the results of what our brains do happen to be the solutions to kinematic equations. It does not follow that we can model the process of producing those solutions by kinematic equations. It does not even follow that the process of producing those solutions bears any resemblance to what we do when we do math to find them.
Here is an analogy: we can observe that the motions of objects obeys the principle of least action [1] and that to compute the action we have to integrate the Lagrangian. It does not follow that there is anything happening in the physical mechanism that causes particles to move that is even remotely analogous to integrating a Lagrangian.
> When I catch a ball ... I know exactly what I'm doing
No, I don't think you do. If you did, you would be able to describe what you are doing to someone else, and they would be able to reproduce your actions based on that description alone. Alternatively, you would be able to render your knowledge into computer code and build a robot that could do it. But I doubt you can actually do either of those things if your only skill is catching a ball and you are not trained in math.
By way of very stark contrast, I am absolutely terrible at hand-eye coordination tasks, but I can build a machine that is much better at it than I am [2]. Just to be clear, I didn't actually build that particular machine, but I do know how. And so I can tell you that the process of learning how to build a machine that can catch a ball is radically different than the process of learning how to catch a ball yourself.
---
[1] https://en.wikipedia.org/wiki/Stationary-action_principle
[2] https://www.youtube.com/watch?v=FycDx69px8U
>> No, we can't even do that. (...)
OK well I'm very confused. I thought our disagreement was on whether our brains actually calculate actual kinematic equations, or just the same results by some other means. It feels to me like we're arguing the same corner but we don't have a common language.
>> No, I don't think you do. (...)
"I can't put my finger on it, but I know it when I see it". My claim is that there is a difference between tacit knowledge, and articulable knowledge. I can not articulate the knowledge I have of how I am catching a ball; but I certainly know how I catch a ball, otherwise I wouldn't be able to do it. In machine learning, we replace explicit, articulable knowledge with examples that represent our tacit knowledge. I might not be able to manually define the relation betwen a set of pixels and a class of objects that might be found in a picture, but I can point to a picture that includes an image of a certain class and label it, with the class. And so can everyone else, and that's how we get tons of labelled examples to train image classifiers with, without having to know how to hand-code an image classifier.
Here's a little thing I'm working on. Assume that, in order to learn any concept we need two things: some inductive bias, background knowledge of the relevant concepts; and "forward knowledge" of the target concept. In statistical machine learning the inductive bias comes in the form of neural net architectures, function kernels, Bayesian priors etc. and the knowledge of a target concept comes in the form of labelled examples. Now, there are four learning settings; tabulating:
Where "Error" is the error of a learned hypothesis with respect to the target theory. In the first setting, where we have knowledge of both the background and the target, and the error is low, we're not even learning anything: just calculating. We can equally well match the first three settings to deductive, inductive, and abductive reasoning. You can also replace "known" and "unknown" with "certain" and "uncertain".Now, I'd say that the invention of kinematic equations by which we can model the way we move our hands to catch balls etc is in the setting where the background theory and the target are both known: the background being our theory of mathematics, and the target being some obsrvations about the behaviour of humans catching balls. I don't know if the kinematic equations you speak of where really derived from such observations, but they could have. Humans are very good at modelling the world in this way.
We're in deep trouble when we're in the last setting, where we have no idea of the right background theory nor the target theory. And that's not a problem solved by machine learning. We only make progress in that kind of problem very slowly, with the scientific method, and it can take us thousands of years, during which we're stuck with bad models. For 15 centuries, the model is epicycles, until we have the laws of planetary motion and universal gravitation. And, suddenly, there are no more epicycles.
This also adressses your earlier comment about betting against a scientific upheaval in the science of computation.
Cool machine, btw, in that video. So you're a roboticist? I work on machine learning of autonomous behaviour for mobile robotics.
No worries.
> It feels to me like we're arguing the same corner but we don't have a common language.
That's possible. It's actually a deep philosophical question. Do planets "solve Newton's equations of motion" when they move? On the one hand, they move in ways that correspond to solutions to those equations, and so one could say that they "find solutions" to those equations. On the other hand, the process by which they do this is pretty clearly radically different than what a mathematician does when they solve equations.
> So you're a roboticist?
I used to be. I've been out of the field for over 20 years now. But back in the day I was pretty well known.
Yes, that's an interesting question- that I'm really not equipped to answer. Probably for the best.
>> I used to be. I've been out of the field for over 20 years now. But back in the day I was pretty well known.
I'm really new to the field so I don't know your work. In fact I wouldn't even say I am in the field as such. An academic sibling suggested I take a post doc job and now I'm collaborating with roboticists. I'm just working on autonomous behaviour- I'm not allowed near hardware.
It's an interesting field although I have to constantly be on my toes to avoid violating my principles. See I'm a peacenick, but it seems with the work I do, as soon as I got that stuff working, someone will want to put it on a drone, strap a gun on its back and send it to kill people. And I'm dead set against that sort of thing.
I had a quick look at your site and you've worked with NASA. Respect! We can send autonomous rovers to explore far away planets and people want to keep them here wreaking havoc and death. Unbelievable.
Do you have any pointers to your work? Something you are really proud of that you did in the past? I'm curious.
https://en.wikipedia.org/wiki/ATLANTIS_architecture
https://flownet.com/gat/papers/tla.pdf
I think I'm most proud of this:
https://link.springer.com/article/10.1007/BF00710855
https://flownet.com/gat/papers/tpesamr.pdf
though it didn't make nearly as much of a splash.
I was also the tech lead on the New Millennium Deep Space One Remote Agent Executive, which sounds cool, but was really kind of a disaster. See:
https://www.youtube.com/watch?v=_gZK0tW8EhQ
if you want the gory details.
Hey, no worries. Thanks for being a gentleman and I'm sorry you're being harassed. Btw, just to be clear: I'm perfectly fine with robust disagreement, I just don't deal well with personal attacks; which you didn't do, I was just worried that's where this conversation was going.
So, thanks for the very detailed analysis of your argument. That indeed makes it much simpler to find common ground. Here's where I disagree: point number 2!
Here's why. It's obvious to me that it's entirely possible to have two distinct models of the same process that compute almost identical results, so it's entirely possible for humans to be using a completely different process to catch balls etc, than kinematic equations.
And here's why I think this is likely: first, because of the point I made above about computational complexity and second because of the observed wide variability in the uh, let's say kinematic capabilities of different humans. If we were all solving kinematic equations, we would all have the same skills. What's more: humans can be wildly inaccurate in their motions (I know I am; don't leave coffee cups on my desk), while robots for example, are distinctly not. That also points to a different computation.
So, to summarise my argument: what we do needs neither be the same computational process, nor be computing the same results, as kinematic equations.
Btw, I'm a bit confused because I thought you were talking about kinematics in classical mechanics, but now I think you're talking about kinematics in robotics, with muscle actions etc. But I think both apply, except the robotics equations are I think much easier to solve than the classical mechanics ones, which I suspect may veer off into the chaotic.
Edit: I had more here on my _agreement_ to your point number 4, but I'm cutting it down to shorten the comment. You don't have all day :)
In any case, I think we just can't say for sure what our brains do, until we can say for sure.
This all turns on what you mean by "completely different". Yes, obviously when you learn to actually catch a ball your brain is not doing anything that maps straightforwardly onto the kinds of symbolic manipulations that happen when you do math. On the other hand, it has to map onto doing math somehow even if that mapping is not straightforward. The only other possibility is that your brain is actually doing something that doesn't map onto math in any way, but still somehow produces the same results that math does by sheer coincidence. If you could actually demonstrate that, it would be one of the biggest breakthroughs in the history of science because it would refute the Church-Turing thesis.
For the time being of course we can agree that our brains probably do some kind of maths, as far as we understand it. I'm guessing the way we understand maths has everything to do with the way our brains understand maths because, well, that's my position in our disagreement. But, see, I can do maths by counting on my fingers, so the question is really what kind of maths we're talking about and how complex can they realistically be. My argument is that if it's not the kind of maths a standard human being can calculate very quickly without pen or paper, then that's a no-go, because that leaves plenty of time to be eaten by a sabretooth, or what have you.
I'll give you long odds against. That would be tantamount to discovering a physical phenomenon that could not be described mathematically.
> a bit like my understanding of the relation between Newtonian mechanics, and General Relativity and Quantum mechanics.
Those relationships are well understood: Newton is a low-order approximation of GR in the weak-field limit.
https://en.wikipedia.org/wiki/Post-Newtonian_expansion
The relationship between Newton and QM is explained, at least operationally if not philosophically, by decoherence:
https://en.wikipedia.org/wiki/Quantum_decoherence
Does this assumption itself need to be proven?
Besides, it's not true: you can simply define it as an assumption within a thought experiment and proceed merrily along, or you can just not bother to consider whether one's premises are true in the first place, and proceed merrily along.
The second option tends to be more popular in my experience, perhaps because it is so much easier, and perhaps for some other reasons also.
If CPUs are made of silicon, why are they so bad at simulating semiconductors? Or why CPUs are so bad at emulating CPUs?
If JavaScript runs on a CPU, why is it so bad at doing bitwise stuff?
Etc.
What the runtime is made of is entirely separate of what's running on it. Same is with human brain (substrate) and human consciousness (software), or humans (substrate) and bureaucracy (runtime) and corporations (software).
Being good at statistics is more of a knowledge graph of understanding concepts than a statistical model, I think.
Just like understanding a car engine.
There are only two problems with this: One, statistical machine learning systems have an extremely limited ability to encode expert knowledge. The language of continuous functions is alien to most humans and it's very difficult to encode one's intuitive, common sense knowledge into a system using that language [1]. That's what I mean when I say "sour grapes". Statistical machine learning folks can't use expert knowledge very well, so they pretend it's not needed.
Two, all the loud successes of statistical machine learning in the last couple of decades are closely tied to minutely specialised neural net architectures: CNNs for image classification, LSTMs for translation, Transformers for language, Difussion models and Ganns for image generation. If that's not encoding knowledge of a domain, what is?
Three, because of course three, despite point number two, performance keeps increasing only as data and compute increases. That's because the minutely specialised architectures in point number two are inefficient as all hell; the result of not having a good way to encode expert knowledge. Statistical machine learning folk make a virtue out of necessity and pretend that only being able to increase performance by increasing resources is some kind of achievement, whereas it's exactly the opposite: it is a clear demonstration that the capabilities of systems are not improving [2]. If capabilities were improving, we should see the number of examples required to train a state-of-the-art system either staying the same, or going down. Well, it ain't.
Of course the neural net [community] will complain that their systems have reached heights never before seen in classical AI, but that's an argument that can only be sustained by the ignorance of the continued progress in all the classical AI subjects such as planning and scheduling, SAT solving, verification, automated theorem proving and so on.
For example, and since planning is high on my priorities these days, see this video where the latest achievements in planning are discussed (from 2017).
https://youtu.be/g3lc8BxTPiU?si=LjoFITSI5sfRFjZI
See particularly around this point where he starts talking about the Rollout IW(1) symbolic planning algorithm that plays Atari from screen pixels with performance comparable to Deep-RL; except it does so online (i.e. no training, just reasoning on the fly):
https://youtu.be/g3lc8BxTPiU?si=33XSM6yK9hOlZJnf&t=1387
Bitter lesson my sweet little ass.
____________
[1] Gotta find where this paper was but none other than Vladimir Vapnik basically demonstrated this by trying the maddest experiment I've ever seen in machine learning: using poetry to improve a vision classifier. It didn't work. He's spent the last 20 years trying to find a good way to encode human knowledge into continuous functions. It doesn't work.
[2] In particular their capability for inductive generalisation which remains absolutely crap.
The main paper: https://gwern.net/doc/reinforcement-learning/exploration/act...
It sounds kinda crazy (is there really that much far transfer?), but you know, I think it would work... He just needed to use LLMs instead: https://arxiv.org/abs/2309.10668#deepmind
If I remember correctly, Vapnik's point is, we know that Big Data Deep Learning works; now, try to do the same thing with small data. Very much like my point that capabilities of models are not improving, only the scale increasing.
In other words; machine learned models are octopus brains (https://www.scientificamerican.com/article/the-mind-of-an-oc...) and that creeps you out. Fair enough, it creeps me out too, and we should honour our emotions — I'm no rationalist – but we should also be aware of the risks of confusing our emotional responses with reality.
Transformers, Diffusion for Vision, Image generation are really odd examples here. None of those architectures or training processes are tuned for Vision in mind lol. It was what? 3 years after Attention 2017 before the famous Vit paper. CNNs have lost a lot of favor to Vits, LSTMs are not the best performing translators today.
The bitter lesson is that less encoding of "expert" knowledge results in better performance and this has absolutely held up. The "encoding of knowledge" you call these architectures is nowhere near that of the GOFAI kind and even more than that, less biased NN architectures seem to be winning out.
>That's because the minutely specialised architectures in point number two are inefficient as all hell; the result of not having a good way to encode expert knowledge.
Inefficient is a whole lot better than can't even play the game, the story of GOFAI for the last few decades.
>If capabilities were improving, we should see the number of examples required to train a state-of-the-art system either staying the same, or going down. Well, they ain't.
The capabilities of models are certainly increasing. Even your example is blatantly wrong. Do you realize how much more data and compute it would take to train a Vanilla RNN to say GPT-3 level performance?
See e.g. my link above where GOFAI plays the game (Atari) very well indeed.
Also see Watson winning Jeopardy (a hybrid system, but mainly GOFAI - using frames and Prolog for knowledge extraction, encoding and retrieval).
And Deep Blue beating Kasparov. And MCTS still the SOTA search algo in Go etc.
And EURISCO playing Traveller as above.
And Pluribus playing Poker with expert game-playing knowledge.
And the recent neuro-symbolic DeepMind thingy that solves geometry problems from the maths olympiad.
etc. etc. [Gonna stop editing and adding more as they come to my mind here.]
And that's just playing games. As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems and neural nets suck at them. Ask Yan LeCun: "Machine learning sucks". He means it sucks in all the things that classical AI does best and he means he wants to do them with neural nets, and of course he'll fail.
It's often forgotten that Rich Sutton said the two things which work are learning (the AlphaGo/Leela Zero policy network) and search (MCTS). (I think the most interesting research in ML is around the circumstances in which large models wind up performing implicit search.)
>As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems
You can use NNs for all these things. It wouldn't make a lot of sense because GOFAI would be perfect and the former would be inefficient but you certainly could which is again more than I can say for GOFAI and the domains you listed.
As it is, your comment seems to tell me that neural nets are good at neural net things and GOFAI is good at GOFAI things, which is obvious, and is what I'm saying: neural nets can make only very limited use of expert knowledge and so suck in all domains where domain knowledge is abundant and abundantly useful, which are the same domains where GOFAI dominates. GOFAI can make very good use of expert knowledge but is traditionally not as good in domains where only tacit knowledge is available, because we don't understand the domain well enough yet, like in anything to do with pattern recognition, which is the same domains where neural nets dominate. If explicit, expert knowledge was available for those domains, then GOFAI would dominate, and neural nets would fall behind, completely contrary to what Sutton thinks.
So, the bitter lesson is only bitter for those who are not interested in what classical AI systems can do best. For those of us who are, the lesson is sweet indeed: we're making progress, algorithmic progress, progress in understanding, scientific progress, and don't need to burn through thousands of credit to train on server farms to do anything of note. That's even a running joke in my team: hey, do you need any server time? Nah, I'll run the experiment on my laptop over lunch. And then beat the RL algo (PPO) that needs three days training on GPUs. To solve mazes badly.
To be honest, I'm always a bit jumpy around your comments because I've noticed them all over the place and they're often grayed-out. You kind of tend to go for the jugular. I don't mean that as a good thing. I think others have noticed it too and you get more reaction than you should. That's a shame, because it's clear there's lots of interesting conversations to be had, given you have such strong views and you seem to have done quite a bit of reading; though only on one side of things.
Anyway sorry for starting it this time around and that you got dead'ed, I hope we get to disagree more in the future.
>> Do you realize how much more data and compute it would take to train a Vanilla RNN to say GPT-3 level performance?
Oh, good point. And what would GPT-3 do with the typical amount of data used to train an LSTM? Rhetorical.
This is evidence _for_ the Bitter Lesson, not against it.
Wait a few years and the Next Big Thing in AI will come along, hot on the heels of the next generation of GPUs, or tensor units or whatever the hardware industry can cook up to sell shovels for the gold rush. By then, Transfomers will have hit the plateau of diminishing returns, there'll be gold in them there other hills and nobody would talk of LLMs anymore because that's so 2020s. We've been there so many times before.
The tricky part here is that "efficiency" is not a single dimension! Transformers are much more "efficient" in one sense, in that they appear to be able to absorb much more data before they saturate; they're in general less computationally efficient in that you can't exploit symmetries as hard, for example, at implementation time.
Let's talk about that in terms of a concrete example: the big inductive bias of CNNs for vision problems is that CNNs essentially presuppose that the model should be translation-invariant. This works great — speeds up training and makes it more stable – until it doesn't and that inductive bias starts limiting your performance, which is in the large-data limit.
Fully-connected NNs are more general than transformers, but they have _so many_ degrees of freedom that the numerical optimization problem is impractical. If someone figures out how to stabilize that training and make these implementable on current or future hardware, you're absolutely right that you'll see people use them. I don't think transformers are magic; you're entirely correct in saying that they're the current knee on the implementability/trainability curve, and that can easily shift given different unit economics.
I think one of the fundamental disconnects here is that people who come at AI from the perspective of logic down think of things very differently to people like me who come at it from thermodynamics _up_.
Modern machine learning is just "applications of maximum entropy", and to someone with a thermodynamics background, that's intuitively obvious (not necessarily correct! just obvious) –in a meaningful sense the _universe_ is a process of gradient descent, so "of course" the answer for some local domain models is maximum-entropy too. In that world view, the higher-order structure is _entirely emergent_. I'm, by training, a crystallographer, so the idea that you can get highly regular structure emerging from merciless application of a single principle is just baked into my worldview very deeply.
Someone who comes at things from the perspective of mathematical logic is going to find that worldview very weird, I suspect.
I don't know about that, I'll be honest. Do you have a reference? I suspect it won't disagree with what I'm saying, that neural nets just can't use strong enough bias to avoid overfitting. I didn't say that in so many words, above, but that's the point of having a good inductive bias, that you're not left, as a learner, to the mercy of the data.
>> Someone who comes at things from the perspective of mathematical logic is going to find that worldview very weird, I suspect.
No that's absolutely a standard assumption in logic :) Think of grammars; like Chomsky likes to say, human language "makes infinite use of finite means" (quoting Wilhelm von Humboldt). Chomsky of course believes that human language is the result of a simple set of rules, very much like logical theories. Personally, I have no idea, but Chomsky consistently and even today pisses off all the linguists and all the machine learning people, so he must be doing something right.
Btw, I'm not coming from the perspective of mathematical logic, only. It's complicated, but, e.g. my MSc was in data science and my PhD in a symbolic form of machine learning. See, learning and logic, or learning and reasoning, are not incompatible, they're fundamentally the same.
To clarify what I mean on this specific bit: the SOTA results in 2D and 3D vision, audio, translation, NLP, etc are all transformers. Past results do not necessarily predict future performance, and it would be absurd to claim that an immutable state of affairs, but it's certainly interesting that all of the domain-specific architectures have been flattened in a very short period of time.
Take my little rant about Rich Sutton's (a god, btw) Bitter Lesson with respect to RL. So, there's AlphaGo, AlphaZero and μZero, yes? AlphaGo knows the rules of Go and starts with some expert knowledge, and beats very human Go player. AlphaZero knows the rules of Go but has no expert knowledge and it beats AlphaGo. And μZero neither knows the rules of Go, nor has expert knowledge, and it beats AlphaZero, and can also plays chess, shoggi and Atari games, with one hand while eating a banana. Do you know how hard it is to eat a banana with one hand? Unpeeled!
Easy to draw a conclusion from that. Except all those systems were developed and used by DeepMind, and there are very few entities besides DeepMind that can even train them, so all we know is what DeepMind claims and we have no way to check their claims. For example, can I test different configurations of μZero, with and without knowledge of the rules of the game and expert knowledge? Not really. And it's clear to me that DeepMind are pushing very, very hard a form of AI that relies on having gigantic resources, like the ones the just completely coincidentally happen to be among the few entities to have access to. So I remain unconvinced.
(I need to re-read the μZero paper, it's in my pdf buffer. I didn't get it the first time I read it, and it might well be that they did make sufficient ablation studies to convince even me and I just don't remember it).
I'm really baffled by such statement and genuinely curious.
How come that studying GOFAI as undergraduate and graduate at many European universities, doing a PhD. and working in the field for several years _never_ exposed me to EURISKO up until last week (thanks to HN)?
I heard about Cyc, many formalism and algorithms that related to EURISKO, but never heard of its name.
Is EURISKO famous in US only?
It was featured in a BBC radio series on AI made by Colin Blakemore [1] around 1980, the papers on AM and EURISKO were in the library of the UK university that I attended.
[1] https://en.wikipedia.org/wiki/Colin_Blakemore#Public_engagem...
I discussed ChatGPT with my yoga teacher recently, but I bet not even my IT colleagues would have a clue about EURISKO. :-)
https://www.lesswrong.com/posts/rJLviHqJMTy8WQkow/recursion-...
> This is a road that does not lead to Friendly AI, only to AGI. I doubt this has anything to do with Lenat's motives - but I'm glad the source code isn't published and I don't think you'd be doing a service to the human species by trying to reimplement it.
To my mind -- and maybe this is just the benefit of hindsight -- this seems way too overcautious on Yudkowsky's part.
[1]: https://www.lesswrong.com/posts/t47TeAbBYxYgqDGQT/let-s-reim...
I would bet against it being possible to implement consciousness on a PDP, but I wouldn’t be very confident about it.
Why is Yudkowsky taken seriously? This stuff is comparable to the "LHC micro black holes will destroy Earth" hysteria.
There are actual concerns around AI like deep fakes, a deluge of un-filterable spam, mass manipulation via industrial scale propaganda, mass unemployment created by widespread automation leading to civil unrest, opaque AIs making judgements that can't be evaluated properly, AI as a means of mass appropriation of work and copyright violation, concentration of power in large AI companies, etc. The crackpot "hard takeoff" hysteria only distracts from reasonable discourse about these risks and how to mitigate them.
IDK, I feel endless hand-wringing about copyright and deepfakes distract from risks of actual, significant harm at scale, some of which you also mentioned.
I will be heavily downvoted for this, but here is how I remember it:
1) LHC was used to study blackholes and prove things like Hawking radiation
2) LHC was supposed to be safe due to Hawking radiation (that was only an unproven theory at the time)
So the unpopular question: what if Hawking radiation didnt actually exist? Wouldnt there be a risk of us dying? A small risk, but still some risk? (especially as the potential micro black hole would have the same velocity as earth, so it wouldnt fly away somewhere into space)
On a side note: how would EURISCO evaluate this topic?
Since I read about this secretive CYC (why u can email asking for it, but source not hosted anywhere?): couldnt any current statistics based AI be used to feed this CYC program / database with information? Take a dictionary and ask ChatGPT to fill it with information for each word.
The AI hysteria I'm talking about here is the "foom" hysteria, the idea that a sufficiently powerful model will start self-improving without bound and become some kind of AI super-god. That's about as wild as the LHC will make a black hole that will implode the Earth. There are fundamental reasons to believe it's impossible, such as the question of "where would the information come from to drive that runaway intelligence explosion?"
There are legitimate risks with AI, but not because AI is somehow special and magical. All technologies have risks. If you make a sharper stick, someone will stab someone with it. Someday we may make a stick so sharp it stabs the entire world (cue 50s sci-fi theremin music).
Edit: for example... I would argue that the Internet itself has X-risks. The Internet creates an environment that incentivizes an arms race for attention grabbing, and the most effective strategies usually rely on triggering negative emotions and increasing division. This could run away to the point that it drives, say, civilizational collapse or a global thermonuclear war. Does this mean it would have been right to ban the Internet or require strict licensing to place any new system online?
You shouldn't use "theory" when it comes to science unless you know what that means. Gravity is a "theory." "Theory" means that it has a working model, comes with a ton of observational evidence in line with predictions, and it has yet to be replaced by anything better. Outside of math, nothing is ever proven. Any leading scientific theory is, at best, "yet to be disproven." And it stays in the lead until something better comes along: more accurate, extending over a greater domain, etc.
Hawking radiation has yet to be observed.
And if you're worried about micro black holes, well, even an iron atom has a non-zero chance of tunneling to a micro black hole state. No collider needed.
Cyc isn't secretive, it's proprietary, the way the Microsoft codebase is, the Adobe codebase is, and so on.
People like religion, particularly if it doesn't affect how they live their life _today_ too much. You get all of the emotional benefits of feeling like you're doing something virtuous without the effort of actually performing good works.
[1] https://news.ycombinator.com/item?id=28343118
I agree.
> I suspect its achievements were slightly overblown and heavily guided by a human hand
So do I. We'll find out how much of its performance was real, and how much bullshit.
> the unreasonably effectiveness of differentiable programming and backpropagation has sucked up much of the oxygen in the room
The Bitter Lesson -- http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Unfortunately it starts with the passing of Douglas Lenat. But that enabled Stanford to open up their 40 year old archive, which they still had, of Lenats work.
Somehow, someway, someone not only stumbled upon EURISKO, but also knew what it was. One of the most notorious AI research projects of the age that actually broke out of the research labs of Stanford and out into the public eye, with impactful results. Granted, for arguably small values of “public” and “impactful”, but for the small community it affected, it made a big splash.
Lenat used EURISKO to find a very unconventional winning configuration to go on to win a national gaming tournament. Twice.
In that community, it was a big deal. The publisher changed the rules because of it, but Lenat returned victorious again the next year. After a discussion with the game and tournament sponsors, he never came back.
Apparently EURISKO has quite a reputation in the symbolic AI world, but even there it was held close.
But now it has been made available. Not only made available, but made operational. EURISKO is written in an obsolete Lisp dialect, Interlisp. But, coincidentally, we have today machine simulators that can run versions of that Lisp on long lost, 40 year machines.
And someone was able to port it. And it seems to run.
The thought of the tendrils through time that had to twist their way for us to get here leaves, at least me, awestruck. So much opportunity for the wrong butterfly to have been stepped on to prevent this from happening.
But it didn’t, and here we are. Great job by the spelunkers who dug this up.
Basically, with the Traveller tournament Lenat appears to have stumbled onto a story that caught the public's imagination, and then through the milked it for all he could to give his project publicity and to make it appear more successful than it actually was. And if that required embellishing the story or just making shit up, well, no harm no foul.
Even when something is technically true, it often turns out that it's being told in a misleading way. For example, you say that "the publisher changed the ruleset". That was the entire gimmick of the Traveller TCS tournament rules! The printed rulebook had a preset progression of tournament rules for each year.
I wrote a bit more about this a few years ago with some of the other details: https://news.ycombinator.com/item?id=28344379
Doug Lenat's sources for AM (and EURISKO+Traveller?) found in public archives - https://news.ycombinator.com/item?id=38413615 - Nov 2023 (9 comments)
Eurisko Automated Discovery System - https://news.ycombinator.com/item?id=37355133 - Sept 2023 (1 comment)
Why AM and Eurisko Appear to Work (1983) [pdf] - https://news.ycombinator.com/item?id=28343118 - Aug 2021 (17 comments)
Early AI: “Eurisko, the Computer with a Mind of Its Own” (1984) - https://news.ycombinator.com/item?id=27298167 - May 2021 (2 comments)
Some documents on AM and EURISKO - https://news.ycombinator.com/item?id=18443607 - Nov 2018 (10 comments)
Why AM and Eurisko Appear to Work (1983) [pdf] - https://news.ycombinator.com/item?id=9750349 - June 2015 (5 comments)
Why AM and Eurisko Appear to Work (1984) [pdf] - https://news.ycombinator.com/item?id=8219681 - Aug 2014 (2 comments)
Eurisko, The Computer With A Mind Of Its Own - https://news.ycombinator.com/item?id=2111826 - Jan 2011 (9 comments)
Let's reimplement Eurisko - https://news.ycombinator.com/item?id=656380 - June 2009 (25 comments)
Eurisko, The Computer With A Mind Of Its Own - https://news.ycombinator.com/item?id=396796 - Dec 2008 (13 comments)
This is amusing: https://www.saildart.org/D.SAI[1,DBL]
And it looks like he wrote a story called "Lethe" as a grad student: https://www.saildart.org/LETHE.DOC[1,DBL]
4 months ago: https://news.ycombinator.com/item?id=38413615
- https://en.wikipedia.org/wiki/Eurisko
Traveller was (and is) a space-based RPG, although the original publisher is long out of business.
https://en.wikipedia.org/wiki/Traveller_(role-playing_game)
What can one do with EURISKO? The fact of its recovery after its authors passing is interesting, in and of itself - but why is EURISKO, specifically, worth the effort of understanding?