Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!)
I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app.
If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it.
Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).
High-level details about how you tackled the various sub-problems would be useful.
I assume, at minimum, there's:
- Segmentation of moving elements
- 3d reconstruction from 2d video
- Reverse kinematics
- Identification of holds
- Search for potential climbing path
- Generation of kinematics for path
I've been working on the same thing, there is an interesting quote from back in the day and its either make your program faster or wait 6 months for the computers to get faster. I look at it the same way now with Key Point Analysis and Pose Estimation libraries.
I feel like I’m the right audience for this - a not very good climber with unusual distribution of limbs.
But I’m not interested - part of the fun in climbing is on sight climbs and solving problems yourself.
Man, where is the hacking spirit? OP posts a very cool, impressive hack and all everyone can talk about in the comments is how it actually isn't interesting to them.
It's cool as an academic exercise, but there are things in life where the struggle is kind of the point. The messy learning process, asking other humans for help, is better for your growth.
If somebody solves the 'hacking' part for you, a lot of fun disappears. Yes they probably had tons of fun, but you lose some of it.
Figuring a route for the first time is by far the most rewarding part of climbing (although I love all of it). Same with kids - if they grok something on their own, what a reaction and reward compared to being told how to do it.
It's great if someone wants to use the "hacking spirit" to help themself climb, it doesn't obligate everyone else to want to use the same technique; and frankly it's (slightly) more interesting hearing people say why they would or wouldn't use it than to have a bunch of comments just saying "awesome that you're doing this" and ignoring whether they'd find it interesting themselves or not.
I would say (opinion) that it’s vastly more interesting to hear whether other people would use Thing I Built. I can nerd snipe myself all day on projects. By that I mean I can easily endlessly entertain myself building things, both useless and useful. Whether they are interesting AND useful for other people is quite a large driver for me both for personal and business projects
If you ever Moonboard, you'll see the app gives easy access to beta videos from Instagram. That's because many people want help on really hard problems. OP app is like those videos.
If you never feel like you need help from watching someone else climb, then I think you are not trying hard enough problems.
As an experienced, very poor climber myself, I enjoy the process of problem solving. However, knowing many experienced, very good climbers - studying beta is how many of them excel. As an aside, the term beta also came from the use of beta tapes by climbers to record themselves climbing so they could study the minutiae of their movement to find improvements. In higher levels of bouldering especially, a nuanced and firm understanding of beta is everything - so careful analysis of your movement can help identify areas of improvement.
I believe this is consistent with most elite (or elite-aspiring) athletes from many sports.
While I personally enjoy the problem-solving aspect of climbing (when I rarely do get out), I absolutely see the value in this project (and other climbing apps that thrive on beta sharing)
Came here hoping to see something like this. Glad to see it’s the top comment.
Part of the beauty of climbing is solving a puzzle and figuring things out.
Edit: Unsurprised to see the demo on a gym route instead of actual rock.
It would be really interesting to build an instance of this model trained on world cup footage for a few reasons.
- Like all of these things, your training data matters and the internet is awash with videos of people climbing badly. A lot of people specifically post "I can't climb this, what am I doing wrong?" videos. World cup climbers are, by the nature of the competition, extremely talented and technically proficient climbers. Even when they fail, they fail in smart interesting ways.
- There's lots of high quality video footage out there. Heck, the problems are even set with visual clarity in mind which would help when parsing that footage. There's potentially enough video to train instances on individual climbers. You could run side by sides like "How would Tamoa climb this and how would Janja climb this?".
- World cup problems are stylistically distinct. They involve lots of moves "typical" climbers will never ever encounter. Many climbers will look at a typical gym problem and think "I have an idea of how to climb this" but will look at a world cup problem and just think "????????". An app that told you how a problem like that should be climbed might be useful.
There are drawbacks too.
- World cup climbers are outliers, whose physical ability (strength, flexibility, etc.) give them access to kinds of movement that other climbers just don't have. No amount of "knowing the sequence" will get me up a climb that requires a full bat hang (look it up) because I just don't have the ankle strength to do the movement.
- World cup "style" is only commonly used at high level comps and in very large commercial gyms. It's probably not extremely relevant to a typical climbing session.
- World cup problems are very hard. Mostly v10 and up? It would be hilarious to watch a model trained on genetic monsters crushing the world's hardest boulder problems try to tell a doughy office worker (me) how to climb v2.
This was exactly my thought. See https://www.instagram.com/p/Cvg3bJkp30a/ on a super-hard slab problem at Innsbruck 2023 (Sorato Anraku couldn't do it, so you know it's hard...)
Of course everyone on HN is focused on the climbing / beta / technique aspect of this.
It is ok that it is a solution looking for a problem. There is obviously no 'business' or 'product' here. It isn't like there is a payment link on the page or anything.
What I'd like to see the comments focus on is that we should just be happy that someone is making the effort to learn more about AI and building tooling around it. Experimentation is king.
They've put their work out into the open (soon to be open sourced even!), not to be criticized over whether it is useful or not, but just that they created something that could spawn other interesting things that solve real world use cases.
I spent 8 years lead climbing at Mission Cliffs in SF 2-5 days a week (5.12 range was my top). They had excellent route setters there that I spent many hours just watching between climbs.
The route setters would effectively do what you're saying, while they were setting the route. They knew exactly what would work and what wouldn't. The holds/wall are really what dictate things and if you're not able to climb it... it isn't really their problem.
As far as I can tell, while this project is cool, it really has nothing to do with climbing or route setting. It is an AI project where the developer just kind of made up some task and had AI follow it.
It looks really interesting, but as an experienced climber I'm not sure if just watching a video of my avatar climbing would really help with skill acquisition.
Also, this claims that the wall type or video quality doesn't matter, but I have a hard time understanding how the model would be able to understand that a small crimp could possibly be dual textured and therefore has only a few specific ways of approaching it.
So it seems that this is more for visualizing a climb (which is a skill most climbers should develop) and not really for dialing in some sort of microbeta for a problem.
I suspect that, absent that information about the exact right way to grab a hold, or the exact way to put a foot on a hold, you'll be limited to beta suggestions, which is fine, I think. It'd be like having a group of other climbers nearby to suggest different beta, even if you don't have any friends.
So, in terms of solving complicated beta faster, I see real utility to this.
It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
> It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
I would think this is actually a Bad Thing. It's very easy to get stuck trying to make a sequence fit your style of climbing. The better approach (especially for long term skill acquisition) is a willingness to learn new styles. That's to say that every sequence is only solvable via one particular style, but I think long term development is hindered if you approach every crux with the one thing you are good at.
> So, in terms of solving complicated beta faster, I see real utility to this.
I can agree with this. But, to the point that others have made, I do wonder what this and the availability of beta videos for many, many routes and blocs does to climbing skill overall. Perhaps I'm just a grumpy old man, but, particularly when bouldering, sorting out the beta should be part of the journey toward eventually sending. Last fall, I visited Hueco Tanks after a six year absence. I suppose I was a bit disappointed to see so many people watching YouTube beta videos of nearly every problem they tried.
>I can agree with this. But, to the point that others have made, I do wonder what this and the availability of beta videos for many, many routes and blocs does to climbing skill overall.
That's a fair concern. That said, there are certain sequences that I'm physically incapable of doing without a dedicated multi-month program of stretching beforehand. Turns out falling on a thumbs-down jam is close enough to a shoulder dislocation that I maybe should've done some PT about it. The extreme is, obviously, the Edlinger vs. American example, but I think the middle ground actually addresses people's peculiar body geometry and/or range of motion. Alex Honnold's exact hip or elbow position might not be as meaningful for Ashima, even if they're on the exact same route.
Such nuances have made a difference in some cases (the one that springs to mind is Todd Skinner's observation that Steve Petro's hips sagged just a little mid-crux on Fiddler on the Roof, which, until corrected, had prevented Petro from nabbing the first ascent of one of the hardest cracks ever climbed to that point). Probably a net-bad for folks projecting Midnight Lightning or similar, but definitely useful for somebody looking to repeat Silence or whatever.
Agreed - so much about the detail of how you would climb something comes down to details that would be hard to measure with a camera, like textures, your estimate of friction, etc. Very cool idea though, looks fun to test.
And this is deep into "sport climbing", borderline gym rat territory imho. It doesn't model all the other core aspects such as protection, rope management, exposure and rest stops. I imaging if you pointed this at a real cliff and recorded several assents it would quickly become a blurry-twitchy mess as all the movements not touching the rock spoiled the data. Maybe for bouldering, but not for real climbing.
There are lots more variables to consider, particularly in lead climbing, even when you have a bolted route. And trad climbing is even more complex than that.
This one-upmanship is very much part of climbing culture. From the top down, the hierarchy is: free soloists (climbing at height without any protection), trad climbers (carry the rope up with them, set their own protection), sports lead climbers (carry the rope up with them, bolts are set into the wall), top rope climbers (rope already there dangling from the top). Right down the bottom, you have aid climbing, when you use equipment to haul yourself up on a rope.
And then there's indoors vs. outdoors, with some dedicated outdoor climbers regarding anything done in a climbing gym as "not real climbing".
Most people don't take this literally, and it's generally considered to be a standing joke in climbing. Sadly, though, some people take it very seriously.
Aid climbing isn't respected on smaller or well-established cliffs, but most understand that it has a place. All the great routes started as aid routes, only being climbed "free" years later. Aid is also a very useful skill in the rain or rescue situations when friction disappears.
Such differences are actually at the heart of climbing. Is taking a helicopter up to the top climbing? Of course not. So from day one it is not about getting to the top but about following invented rules governing how you get to the top. How about a bolted-on ladder? Or pre-placed protection (bolts)? Clean trad, leaving the rock as you found it, is generally seen as the highest form.
(Climbing rock without ropes may be more "pure" but is so dangerous that it should never be idolized.)
What could be interesting is if you could compare your attempts to the avatar climbing and receiving feedback afterwards. This would effectively be a step up from simply recording your send attempts.
Of the full distribution of possible video qualities one can take on a modern phone camera, the vast majority of video qualities will be fine for the AI to understand fine details. Obviously, if you somehow or for some reason, take a video with really bad quality, it will not give you what you want.
Same explanation goes for the walls. If you take a video of just a really dark wall with really bad holds, it is probably won't give you what you want either.
I don't see the shape of the holds being a big problem. With some help from indoor companies and hold makers, figuring out which hold model is on the wall should be possible.
As for the usefullness of the software, I'm sceptical too as it don't really solve a problem. But maybe I'm not seeing it and it could be good for beginners :) A good improvement would be adding a comparison between you and the model in term of body position and fluidity of movements.
The idea of incorporating actual hold data and "recognizing" specific holds is interesting, but I'm not sure it completely solves the problem.
The "Boss" from Pusher is arguably the most famous climbing hold ever made. For a decade or more, every gym had one, but they were all unique. Lots of them had micro chips that became critical to usage of the hold. Some had decent texture and some were glassy smooth from years and years and years of use. A lot of the accidental variation in new holds has gone away as the industry has standardized around a handful of industrial fabricators like Aragon, but even over the course of a single indoor boulder problem's life, the accumulation of chalk, sweat, and shoe rubber can have a significant impact on how a hold climbs.
I guess the real question is, do these changes just make routes harder or do they make them fundamentally different? Do they actually change the set of moves that constitutes the easiest way to the top? To be honest, I'm not entirely sure. But it's something interesting to think about.
Exactly, holds will evolve as they get used and more polished, even indoors. Climbing a Moonboard with a new set of holds is quite different than climbing on one with older more polished holds, even if it's the exact same problem and the same holds.
It's an interesting project and it could be fun to watch, but it's completely useless.
To me, the customer here would be climbing gyms, offering a service to climbers.
1. Set up camera on routes
2. Record all climbs
3. Reason through hold details
4. Generate potential movements
5. Show climbs vs ghost movements
6. Feedback to tune model
3 being accomplished by reasoning "if a movement should be possible using the identified hold, but no one successfully does it, the hold must be misidentified or have different properties."
The point in this thread seemed to be "real world holds have different properties, and that defines possible approaches to holds."
To which I pointed out that, with enough data, you could reason backwards to figure out their properties.
Assuming that's solved, if the question is "What is the point?" then I'd answer the same point as golf swing analysis -- structured comparison feedback for continual improvement.
"Have you thought about trying X move at Y point?" or "You're trying X move at Y point, but here's how you differ from someone successfully doing it" both seem useful feedback.
And essentially what's manually generated now, from someone watching and then providing feedback.
With regards to strength, hell, if you wanted to get fancy you could also deduce a specific user's strength, comparing their moves against others' moves on the same features.
> I don't see the shape of the holds being a big problem. With some help from indoor companies and hold makers, figuring out which hold model is on the wall should be possible.
Even if you know the exact hold model and it’s in pristine condition, it’s basically impossible to tell how it’s gonna work from a single angle on video at a distance. Even tiny variations in angle of the wall and rotation of the hold on the wall can completely change how you use it.
This is really cool from a technology point of view but the main attraction of climbing for me is figuring out how to "solve" the climb. Trying to solve a crux with a buddy is half of the fun for me.
Right, but for people in their first couple of years of climbing, it can be really helpful to try to solve it, then see what the AI coach says, examine the diff, and thereby improve your solving and climbing skills.
I have definitely discovered in my life that I learn probably 10x faster in situations where I can try something, and then immediately compare it with an expert solution. Rather than just trying and trying and trying on my own.
And once you're experienced in climbing (like you probably are), then of course you don't need to compare anymore. The joy is in the solving. But expert solutions are really helpful to get to the point where you can consider yourself experienced, so much more quickly.
With bouldering, the set of moves is fairly limited and what actually prevents someone from pulling harder comes down to flexibility, core strength, and contact strength.
Well if we're talking about "wall climbing", the set of moves don't expand that much as one progresses. Your technique needs to be more efficient and your transitions and balance on point, but it's not like there's a esoteric 13a move that a 10d climber hasn't seen. Sure the first time someone dyno'd on a roped climb was a sea change.
Bouldering arguably has a larger set of moves than "wall climbing", but again the physicality of this sport is the crux for most people. One can study and maybe drill all the "advanced" moves, but without the flexibility and strength to execute on microholds, that knowledge is useless.
A few thoughts as I've done academic research & built products in this area:
- if you're using SMPL body parameters this will have to stay research / open-source
- is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing.
Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.
This is an end-end system that just takes in video frames. Camera parameters are one of the things that is predicted. It gives promising results for a wide variety of environments (cliffs, diff types of bouldering walls, diff outdoor walls, etc.), though not always accurate. Path planning is also part of the end-end system. Will share more details in the paper.
commercial license - the research group formed a corporate entity that licenses the body model and all derived work (SMPL-X, etc.): https://meshcapade.com/SMPL
Climbing route setting (when done at a high level) is constantly in conversation with itself. Many climbers don't realize that there are trends or fads in movement styles that sweep across the industry. This has only picked up steam in an era of social media.
Just like movie dorks will happily spend hours explaining how an individual shot in a movie is actually an insider reference to another movie, and as a result a statement of intent for the movie as a whole, professional route setters will talk your ear off about the way one of their problems embraces or rejects specific kinds of movement trends of the last 6 months.
That intentional rejection is interesting. Many route setters, especially for competitions, are in constant search of novelty. One kind of perfect problem is something that looks confusing and impossible, up until you see it done, at which point it seems almost obvious. It's the feeling of solving a sudoku. But critically, they want climbers to be initially confused.
I wonder if AI might actually be better than humans at sequencing these kinds of problems. Humans bring so much context and experience and expectation to the process that we are easily tricked. AI just looks through a few terabytes of video and says "What about this?".
> I wonder if AI might actually be better than humans at sequencing these kinds of problems
Probably a deeply unpopular take here, but without knowing anything about climbing routes, I'm gonna say no. I'm not saying that they won't have excellent quality output that might even solve problems that human output can't, but the process of creating something is meaningful, even commercially. Surely this will be useful in some respects, but I just don't buy the idea that humanity is destined to passively consume automated algorithm-generated utility products-- especially creative ones-- no matter how smooth, cheap, and clever they might be.
Well, the tricky bit here is that the route setter, a human, is the one actually solving the problem. So the problem as set is (and must be) a human creation first. This is especially true in outdoor climbing, where the first ascent process might involve the installation of anchor fixtures, or the removal of poorly-secured features for safety. You'd need some pretty wild sensor suites to correctly differentiate between a really good hold, and a dangerous flake that will peel off the wall if the slightest force is applied to it. The AI just generates potential solutions to the problem once the holds are found/placed. Certainly, there's some interesting conversations about how satisfying it is to solve a rubick's cube using somebody's algorithm vs. just figuring it out, but its not like the computer is inventing a rubick's cube.
Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall. I expect that for a long time, AI generated climbing routes would rarely be good, but would consistently be physiologically impossible, feature uninteresting movement, or be too easy.
Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
> Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall.
This would follow the exact path image GenAI evolved through.
Step 1: Teach a model to recognize objects from noisy data.
Step 2: Reverse-feed that model random noise and force it to hallucinate that noise back into likely objects.
As there's probably physics simulation at some point in this particular scenario, there'd probably also be step 3 of simulating a climb through the generated path to validate feasibility / specific qualities.
Spotting a refugee from rockclimbing.com on hacker news was not on my bingo card for the day. But I guess if I'm here (writing novels about route setting) then I shouldn't be surprised other people are too.
It just goes to show how impactful online communities are capable of being. rockclimbing.com was in it's heyday right as I was discovering climbing. I was a bored kid constructing my entire identity around climbing and there was no other place to do that outside of the gym. No mountain project. No youtube. No social media. I spent a lot of hours lurking those forums. There are only a handful of users I could still name, but I bet I would recognize a lot of them.
> You'd need some pretty wild sensor suites to correctly differentiate between a really good hold
Ah, I see.
> The AI just generates potential solutions to the problem once the holds are found/placed
Yeah and I think that's really going to be the sweet spot for generative tools for the forseeable future.
> Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
Right right. I have a feeling that making a more convincing substitute is primarily a matter of having less access to data than say, paintings and photography which are certainly not less nuanced than this creative task. But as I said, a lot of people care about how something was made, too. I'll bet that's going to be a much bigger factor, at least in marketing, than many realize in the near future.
> the process of creating something is meaningful, even commercially.
That's true, but why does it mean that the answer to the more or less objective question "will AI actually be better than humans at sequencing these kinds of problems?" (As stated it's not really objective, but one could easily come up with metrics like, say, total time to a correct solution, or time spent observing the route or other climbers, or ….) One can imagine other, related questions that are less objective (like "will it be a good idea to integrate this AI assistance into climbing competitions?"), but, to me, the answer to the (implicit) original question has nothing to do with whether or not the activity is meaningful, or with humans' destiny one way or the other.
Sure, they may well be quantitatively better. If you were to create metrics to measure the number of mistakes or weird spots or overly annoying things, it's quite possible that the output from the algorithm could score better than human output, and the throughput would obviously be incomparable. But, whether or not something is qualitatively better is far more subjective-- it's influenced by our culture, our experiences, and everything else that creates the lens through which we see the world. Something's origin absolutely affects the way people experience it, be it a physical object, story, experience, etc. Don't get me wrong-- I realize there's real value in affordable quantity with with mediocre quality-- how many restaurants in the world are McDonald's? But then again, how many restaurants in the world aren't McDonald's? If McDonald's could sell you a ribeye comparable to Capital Grille, I'd be astonished if it put a dent in Capital Grille's bottom line. Applebees, however...
I think you probably need to re-read what I wrote. I'm not against AI at all-- I use it all the time. I just don't think it's going to be qualitatively better than human creative output because how something was made matters to people. I also think the tech world thinks they're a lot better at judging creative output than they are.
Right, those are important considerations, but I don't think that they're the same consideration. If you ask whether a celebrity chef is better at cooking a certain dish than I am, then the answer (no matter the dish, for I'm not much of a chef) is almost certainly "yes." If I answered instead no, that the provenance of a dish matters, that the celebrity-chef culture is bad for fine dining, and that my wife would rather have the dish made by me, then I think I'd be regarded as missing the point of the question, even if my objections were true on their own.
Well, I think that celebrity vs husband is not a good analog for human vs generated. I also think that in this case, origin has broader connotations than provenance, in that it includes context. What if the celebrity chef was your wife's brother? What if you had a degenerative neurological disease? What if it was the dish you cooked for her on your second date? Anything can seem very cut and dried if you impose artificial restrictions and examine it in a vacuum. And, because you can't easily define and quantify things like context, the engineering and tech worlds tend disregard them because they don't really factor into the engineering equation... and that's why designers exist.
Very nice. Please do open it up as much as possible. I'm curious about the data pipeline. I love bouldering and have thought about different ML ideas in that domain. I'm also an avid beta breaker. Finding alternative solutions somehow tickles my hacker-brain. So my first idea was intended beta vs. alternatives but I was only thinking in 2d and images :)
Another idea I've played around with is more objective grading but labeling is kind of hard especially with gyms rotating routes so I thought about using Kilterboatd data and transfer learning it or maybe unsupervised approaches.
Congrats on a cool idea and most importantly the execution.
I'm genuinely curious how the model classifies holds. I could (hypothetically) see this doing something like assigning more utility to certain holds (e.g. a steep sloper that is impossible to use as a handhold but is can be used as a smearing foothold) and thereby giving bad/implausible beta. That being said, it seems like an obvious shortcoming and I'm very interested in learning more!
Very cool idea but I wonder how well this can possibly work in practice. So many bouldering problems hinge on small things, such as how a little crimp hold is oriented or how deep the groove in it is. It can be hard to determine this even in person until you get up close or try to grab it.
Additionally, this model seems to take into account the body type but not other relevant factors, such as climber's strength and flexibility/mobility. Even in the demo, that suggested move of putting the right foot high and pushing off the jug up and to the right could not be done by a lot of new climbers. This parallels what I've personally seen when watching others climb. For example, some climbers are exceptionally strong and can basically campus their way up a route. Watching their beta is irrelevant to me. I don't have the same strength to pull those moves.
Could this be used to grade routes? It would be cool if a photo or short video of a route could be used to better assist in grading climbing routes. Grading is a massively subjective thing, to some extent, so this could be super tricky.
Also, how will the AI know when it is appropriate to inflate the grade to make the climbing seem harder, and when it is appropriate to sandbag so I can pretend a route is easier for me than it actually is?
Very interesting, do wish we could play with it since it is a show hn, maybe repost when it is open sourced/usable. Looking forward to reading about how it works. How does it know which holds are on? How does it know what the top is? Are those inputs or inferred or is it just end to end completing likely body positions one after the other?
Does the AI create the 3D model from a static camera?
Can I choose the color of the avatar? (It's not an important feature, but people will love it.)
Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)
- it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks.
- don't have that functionality but its really easy to implement.
- ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.
Yeah it performs best with indoor bouldering videos with spread out holds and great lighting. I will post examples of it climbing natural walls in the paper. Will send soon.
I'm sceptical about the usefulness of this for a number of reasons, including the points raised by others about hold recognition etc.
It's also difficult to see how the system would work out the 3D arrangement of holds. It can clearly try to infer 3D relationships from video of an existing climb, but it would be hard for it to work out the 3D position or orientation of any holds not used by the climber in the video.
These two put together make solving the climbing problem even more difficult, because appropriate body positions and moves are often very sensitive to even tiny differences in shape, relative position, and orientation of holds; and occluding volumes, arêtes, cracks and so can make the problem even harder.
But as a simplified demo, this is cool, and I salute it.
I have a side project in a related space. For mine I'm syncing climbing between different videos automatically. I find this pretty useful because it highlights difference.
I have had a similar idea for the past few years: less an AI model that takes in a photo and generates a video, more a movement planning tool given a 3D model of the wall.
Echoing some of the other comments, I think it would be more interesting to not just see a single avatar climbing the problem, but to see many possible approaches to the same problem, even for climbers with the same body type. This way, even skilled climbers that are stuck on a problem can consider alternate beta/techniques that they may find easier to execute.
Very cool!! I am unfortunately very far from an AI expert. Can you share how you built this? Did you have to train your own model, or is it an open source model that you finetuned for climbing?
This has always been a dream, because there is a "perfect" way for your body to climb a route but it would be insanely complex to calculate, especially when factoring in every tiny physiological aspect like ankle strength and hamstring flexibility.
This is a really cool step though.
Maybe a more useful variant with the current tech would be isolating a single move and seeing every variation the AI can think up.
One of the most fun part of climbing is the problem-solving aspect of it. Figuring your way to get up the wall/boulder. Why would you want to get rid of that?
Do you enjoy climbing with others? Personally, I enjoy seeing the different beta on a climb, especially between different body types. It is interesting seeing unique approaches to the same problem.
What about when you try to solve the problem and you get stuck and can't? Or solve it but in a bad way that requires way too much exertion?
The point is to try to solve it yourself, then compare with an expert solution, and therefore learn how to improve.
If you're just blindly trying to problem-solve through trial and error without ever comparing against expert feedback, you're going to learn climbling extremely slowly...
When faced with a challenge, you want to figure out as much as you reasonably can, and then learn what you were missing.
Struggling at the same problem for several sessons, or god forbid years, sounds like misery to me. I'd rather use that time productively learning, rather than struggling for the sake of it, because I refuse to learn from others.
Cool idea! My friends and I have chatted about building something similar, though with a coaching focus. Specifically for use with spray walls/kilterboards/etc.
It could be interesting to combine something like this with an analysis tool. Analyse a climbers attempts or successes and compare it to the beta figured out by the model. Then offer tips on body positioning or technique based on your weight/height/strength.
I guess it's just how my mind works but I thought this will make a great part of some stories I'm working on, only adapted for cat burglars to figure out the optimal route on a building beforehand.
Maybe also good thing for the next Mission Impossible movie if they ever make one.
> The basic physics of strength/weight is that strength is reliant on the cross sectional area of muscles (number of fibers) and therefore increases quadratically, whereas weight is a function of volume and increases cubically. So the amount of strength you can gain as your mass increases will always lag behind the amount of mass gained. This is why insects are so strong compared to their bodyweight, and why the creatures with the highest strength/weight ratio's are all small.
> If we analzye male pro climbers we'll see that the heaviest of them max out at around 170-180 lbs (and there are VERY few of those, the majority being between 130 and 150), and that's not a fluke. In order to be a great rock climber you need strong forearms, and strong lats, and that's about it. Muscle mass, or fat mass basically anywhere else is mostly just pulling you off the wall (a strong core helps too but it doesn't need to weigh a lot).
Looks cool! It definitely looks technically challenging, especially for expert climbers who have a variety of creative moves at their disposal. But I can definitely see this being useful for novices.
I will be open sourcing the codebase and model. Due to compute and training dataset size constraints, the model is not exactly "production ready". I have (and am) cherry picking rare examples of it where it is decent. You can play around with it when I open source it but truly good results require multiple millions of dollars (but it really is a linear scaling of compute and dataset).
Ha! That was unintended.
"You can't connect the dots looking forward; you can only connect them looking backwards." - someone somewhere (likely Steve Jobs)
cool technical project, but as a climber, part of the fun is solving the problem with the help of my own brain and the brains of the other climbers around me.
Considering the notoriety of AI models in making confident but blatant mistakes or simply lying about things they don't have real answers to I don't know how smart it is to trust one with something in which bad advice can kill you very quickly and immediately.
Great job! A lot of folks from my CS PhD program are really into climbing (including me), but none of us has conducted research that explored climbing. Rather climbing is our respite from research.
Anyway, I think this could work really well as an app. Good luck.
Thanks! Still unsure if we will commercially pursue this specific idea or not. This was moreso meant to be a public progress marker from our broader work pursuing virtual avatar simulations. If enough interest is there, we will think about it. But, for now, this (will be) a paper and some open sourced code.
This is cool and all but I can't help but wonder if it'd be even cooler to instead show the beta for climbing out of the negativity found in the comments here. /s
But really though, awesome project and congrats on getting it to this point! I'm excited for the day when something like this can be used to critique form
I do BJJ and wonder how it could AI models could help. I find watching tape to help, not only myself but everyone who I know has recorded their matches has found it helpful.
Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!)
I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app.
If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it.
Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).
I assume, at minimum, there's:
Worst case able to build this, scenario, author builds something else entirely.
I think OSS is best for this category right now.
Figuring a route for the first time is by far the most rewarding part of climbing (although I love all of it). Same with kids - if they grok something on their own, what a reaction and reward compared to being told how to do it.
If you never feel like you need help from watching someone else climb, then I think you are not trying hard enough problems.
I believe this is consistent with most elite (or elite-aspiring) athletes from many sports.
While I personally enjoy the problem-solving aspect of climbing (when I rarely do get out), I absolutely see the value in this project (and other climbing apps that thrive on beta sharing)
Edit: Unsurprised to see the demo on a gym route instead of actual rock.
- Like all of these things, your training data matters and the internet is awash with videos of people climbing badly. A lot of people specifically post "I can't climb this, what am I doing wrong?" videos. World cup climbers are, by the nature of the competition, extremely talented and technically proficient climbers. Even when they fail, they fail in smart interesting ways.
- There's lots of high quality video footage out there. Heck, the problems are even set with visual clarity in mind which would help when parsing that footage. There's potentially enough video to train instances on individual climbers. You could run side by sides like "How would Tamoa climb this and how would Janja climb this?".
- World cup problems are stylistically distinct. They involve lots of moves "typical" climbers will never ever encounter. Many climbers will look at a typical gym problem and think "I have an idea of how to climb this" but will look at a world cup problem and just think "????????". An app that told you how a problem like that should be climbed might be useful.
There are drawbacks too.
- World cup climbers are outliers, whose physical ability (strength, flexibility, etc.) give them access to kinds of movement that other climbers just don't have. No amount of "knowing the sequence" will get me up a climb that requires a full bat hang (look it up) because I just don't have the ankle strength to do the movement.
- World cup "style" is only commonly used at high level comps and in very large commercial gyms. It's probably not extremely relevant to a typical climbing session.
- World cup problems are very hard. Mostly v10 and up? It would be hilarious to watch a model trained on genetic monsters crushing the world's hardest boulder problems try to tell a doughy office worker (me) how to climb v2.
It is ok that it is a solution looking for a problem. There is obviously no 'business' or 'product' here. It isn't like there is a payment link on the page or anything.
What I'd like to see the comments focus on is that we should just be happy that someone is making the effort to learn more about AI and building tooling around it. Experimentation is king.
They've put their work out into the open (soon to be open sourced even!), not to be criticized over whether it is useful or not, but just that they created something that could spawn other interesting things that solve real world use cases.
Huge kudos for doing this work.
The route setters would effectively do what you're saying, while they were setting the route. They knew exactly what would work and what wouldn't. The holds/wall are really what dictate things and if you're not able to climb it... it isn't really their problem.
As far as I can tell, while this project is cool, it really has nothing to do with climbing or route setting. It is an AI project where the developer just kind of made up some task and had AI follow it.
Also, this claims that the wall type or video quality doesn't matter, but I have a hard time understanding how the model would be able to understand that a small crimp could possibly be dual textured and therefore has only a few specific ways of approaching it.
So it seems that this is more for visualizing a climb (which is a skill most climbers should develop) and not really for dialing in some sort of microbeta for a problem.
So, in terms of solving complicated beta faster, I see real utility to this.
It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
> It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
I would think this is actually a Bad Thing. It's very easy to get stuck trying to make a sequence fit your style of climbing. The better approach (especially for long term skill acquisition) is a willingness to learn new styles. That's to say that every sequence is only solvable via one particular style, but I think long term development is hindered if you approach every crux with the one thing you are good at.
> So, in terms of solving complicated beta faster, I see real utility to this.
I can agree with this. But, to the point that others have made, I do wonder what this and the availability of beta videos for many, many routes and blocs does to climbing skill overall. Perhaps I'm just a grumpy old man, but, particularly when bouldering, sorting out the beta should be part of the journey toward eventually sending. Last fall, I visited Hueco Tanks after a six year absence. I suppose I was a bit disappointed to see so many people watching YouTube beta videos of nearly every problem they tried.
That's a fair concern. That said, there are certain sequences that I'm physically incapable of doing without a dedicated multi-month program of stretching beforehand. Turns out falling on a thumbs-down jam is close enough to a shoulder dislocation that I maybe should've done some PT about it. The extreme is, obviously, the Edlinger vs. American example, but I think the middle ground actually addresses people's peculiar body geometry and/or range of motion. Alex Honnold's exact hip or elbow position might not be as meaningful for Ashima, even if they're on the exact same route.
Such nuances have made a difference in some cases (the one that springs to mind is Todd Skinner's observation that Steve Petro's hips sagged just a little mid-crux on Fiddler on the Roof, which, until corrected, had prevented Petro from nabbing the first ascent of one of the hardest cracks ever climbed to that point). Probably a net-bad for folks projecting Midnight Lightning or similar, but definitely useful for somebody looking to repeat Silence or whatever.
And then there's indoors vs. outdoors, with some dedicated outdoor climbers regarding anything done in a climbing gym as "not real climbing".
Most people don't take this literally, and it's generally considered to be a standing joke in climbing. Sadly, though, some people take it very seriously.
(Climbing rock without ropes may be more "pure" but is so dangerous that it should never be idolized.)
Of the full distribution of possible video qualities one can take on a modern phone camera, the vast majority of video qualities will be fine for the AI to understand fine details. Obviously, if you somehow or for some reason, take a video with really bad quality, it will not give you what you want.
Same explanation goes for the walls. If you take a video of just a really dark wall with really bad holds, it is probably won't give you what you want either.
As for the usefullness of the software, I'm sceptical too as it don't really solve a problem. But maybe I'm not seeing it and it could be good for beginners :) A good improvement would be adding a comparison between you and the model in term of body position and fluidity of movements.
The "Boss" from Pusher is arguably the most famous climbing hold ever made. For a decade or more, every gym had one, but they were all unique. Lots of them had micro chips that became critical to usage of the hold. Some had decent texture and some were glassy smooth from years and years and years of use. A lot of the accidental variation in new holds has gone away as the industry has standardized around a handful of industrial fabricators like Aragon, but even over the course of a single indoor boulder problem's life, the accumulation of chalk, sweat, and shoe rubber can have a significant impact on how a hold climbs.
I guess the real question is, do these changes just make routes harder or do they make them fundamentally different? Do they actually change the set of moves that constitutes the easiest way to the top? To be honest, I'm not entirely sure. But it's something interesting to think about.
It's an interesting project and it could be fun to watch, but it's completely useless.
To me, the customer here would be climbing gyms, offering a service to climbers.
3 being accomplished by reasoning "if a movement should be possible using the identified hold, but no one successfully does it, the hold must be misidentified or have different properties."To which I pointed out that, with enough data, you could reason backwards to figure out their properties.
Assuming that's solved, if the question is "What is the point?" then I'd answer the same point as golf swing analysis -- structured comparison feedback for continual improvement.
"Have you thought about trying X move at Y point?" or "You're trying X move at Y point, but here's how you differ from someone successfully doing it" both seem useful feedback.
And essentially what's manually generated now, from someone watching and then providing feedback.
With regards to strength, hell, if you wanted to get fancy you could also deduce a specific user's strength, comparing their moves against others' moves on the same features.
Even if you know the exact hold model and it’s in pristine condition, it’s basically impossible to tell how it’s gonna work from a single angle on video at a distance. Even tiny variations in angle of the wall and rotation of the hold on the wall can completely change how you use it.
I have definitely discovered in my life that I learn probably 10x faster in situations where I can try something, and then immediately compare it with an expert solution. Rather than just trying and trying and trying on my own.
And once you're experienced in climbing (like you probably are), then of course you don't need to compare anymore. The joy is in the solving. But expert solutions are really helpful to get to the point where you can consider yourself experienced, so much more quickly.
This is about wall climbing where there's lots of choice for moves.
Bouldering arguably has a larger set of moves than "wall climbing", but again the physicality of this sport is the crux for most people. One can study and maybe drill all the "advanced" moves, but without the flexibility and strength to execute on microholds, that knowledge is useless.
- if you're using SMPL body parameters this will have to stay research / open-source - is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing.
Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.
This is an end-end system that just takes in video frames. Camera parameters are one of the things that is predicted. It gives promising results for a wide variety of environments (cliffs, diff types of bouldering walls, diff outdoor walls, etc.), though not always accurate. Path planning is also part of the end-end system. Will share more details in the paper.
I'm assuming this is evolutionary / brute force of some nature, given OP's comments about it being expensive to run.
Just like movie dorks will happily spend hours explaining how an individual shot in a movie is actually an insider reference to another movie, and as a result a statement of intent for the movie as a whole, professional route setters will talk your ear off about the way one of their problems embraces or rejects specific kinds of movement trends of the last 6 months.
That intentional rejection is interesting. Many route setters, especially for competitions, are in constant search of novelty. One kind of perfect problem is something that looks confusing and impossible, up until you see it done, at which point it seems almost obvious. It's the feeling of solving a sudoku. But critically, they want climbers to be initially confused.
I wonder if AI might actually be better than humans at sequencing these kinds of problems. Humans bring so much context and experience and expectation to the process that we are easily tricked. AI just looks through a few terabytes of video and says "What about this?".
Probably a deeply unpopular take here, but without knowing anything about climbing routes, I'm gonna say no. I'm not saying that they won't have excellent quality output that might even solve problems that human output can't, but the process of creating something is meaningful, even commercially. Surely this will be useful in some respects, but I just don't buy the idea that humanity is destined to passively consume automated algorithm-generated utility products-- especially creative ones-- no matter how smooth, cheap, and clever they might be.
Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall. I expect that for a long time, AI generated climbing routes would rarely be good, but would consistently be physiologically impossible, feature uninteresting movement, or be too easy.
Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
This would follow the exact path image GenAI evolved through.
Step 1: Teach a model to recognize objects from noisy data.
Step 2: Reverse-feed that model random noise and force it to hallucinate that noise back into likely objects.
As there's probably physics simulation at some point in this particular scenario, there'd probably also be step 3 of simulating a climb through the generated path to validate feasibility / specific qualities.
It doesn't sound impossible.
What's really wild to me is how somebody would recognize a mid-tier poster from a website I thought effectively defunct for nearly 10 years now.
Ah, I see.
> The AI just generates potential solutions to the problem once the holds are found/placed
Yeah and I think that's really going to be the sweet spot for generative tools for the forseeable future.
> Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
Right right. I have a feeling that making a more convincing substitute is primarily a matter of having less access to data than say, paintings and photography which are certainly not less nuanced than this creative task. But as I said, a lot of people care about how something was made, too. I'll bet that's going to be a much bigger factor, at least in marketing, than many realize in the near future.
That's true, but why does it mean that the answer to the more or less objective question "will AI actually be better than humans at sequencing these kinds of problems?" (As stated it's not really objective, but one could easily come up with metrics like, say, total time to a correct solution, or time spent observing the route or other climbers, or ….) One can imagine other, related questions that are less objective (like "will it be a good idea to integrate this AI assistance into climbing competitions?"), but, to me, the answer to the (implicit) original question has nothing to do with whether or not the activity is meaningful, or with humans' destiny one way or the other.
1. You don’t like what that would mean about the destiny for humanity.
2. A human making it makes it inherently better.
3. If it’s lower quality, it’s lower quality.
I get why this would lead to strong beliefs. But these arguments aren’t very convincing.
Congrats on a cool idea and most importantly the execution.
Additionally, this model seems to take into account the body type but not other relevant factors, such as climber's strength and flexibility/mobility. Even in the demo, that suggested move of putting the right foot high and pushing off the jug up and to the right could not be done by a lot of new climbers. This parallels what I've personally seen when watching others climb. For example, some climbers are exceptionally strong and can basically campus their way up a route. Watching their beta is irrelevant to me. I don't have the same strength to pull those moves.
Edit: apprently I wasnt the first to make this joke. I emphasise that this is Really a cool project. Great job!
One also learns to just recalibrate according to regions. E.g. climbing 11d at one gym, versus 12a at another.
Can I choose the color of the avatar? (It's not an important feature, but people will love it.)
Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)
- don't have that functionality but its really easy to implement.
- ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.
(Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))
It's also difficult to see how the system would work out the 3D arrangement of holds. It can clearly try to infer 3D relationships from video of an existing climb, but it would be hard for it to work out the 3D position or orientation of any holds not used by the climber in the video.
These two put together make solving the climbing problem even more difficult, because appropriate body positions and moves are often very sensitive to even tiny differences in shape, relative position, and orientation of holds; and occluding volumes, arêtes, cracks and so can make the problem even harder.
But as a simplified demo, this is cool, and I salute it.
Some examples:
https://www.instagram.com/p/Cvg3bJkp30a/
https://www.instagram.com/p/Cpuv2zvptdW/
It's actually syncing in 3D so is fairly independent of camera angle (as the second video shows)
Echoing some of the other comments, I think it would be more interesting to not just see a single avatar climbing the problem, but to see many possible approaches to the same problem, even for climbers with the same body type. This way, even skilled climbers that are stuck on a problem can consider alternate beta/techniques that they may find easier to execute.
Does it just tell you how it would climb things you can already climb?
This is a really cool step though.
Maybe a more useful variant with the current tech would be isolating a single move and seeing every variation the AI can think up.
The point is to try to solve it yourself, then compare with an expert solution, and therefore learn how to improve.
If you're just blindly trying to problem-solve through trial and error without ever comparing against expert feedback, you're going to learn climbling extremely slowly...
It's an essential part of the sport - the satisfaction of using one's own body and mind to overcome the seemingly impossible.
If you never get stuck, how could you possibly experience it?
When faced with a challenge, you want to figure out as much as you reasonably can, and then learn what you were missing.
Struggling at the same problem for several sessons, or god forbid years, sounds like misery to me. I'd rather use that time productively learning, rather than struggling for the sake of it, because I refuse to learn from others.
It could be interesting to combine something like this with an analysis tool. Analyse a climbers attempts or successes and compare it to the beta figured out by the model. Then offer tips on body positioning or technique based on your weight/height/strength.
Maybe also good thing for the next Mission Impossible movie if they ever make one.
Even if you have a great BMI and muscular build, but if you are heavy, then at some point physics starts working against you.
https://www.reddit.com/r/bouldering/comments/595a2m/how_much...
> The basic physics of strength/weight is that strength is reliant on the cross sectional area of muscles (number of fibers) and therefore increases quadratically, whereas weight is a function of volume and increases cubically. So the amount of strength you can gain as your mass increases will always lag behind the amount of mass gained. This is why insects are so strong compared to their bodyweight, and why the creatures with the highest strength/weight ratio's are all small.
> If we analzye male pro climbers we'll see that the heaviest of them max out at around 170-180 lbs (and there are VERY few of those, the majority being between 130 and 150), and that's not a fluke. In order to be a great rock climber you need strong forearms, and strong lats, and that's about it. Muscle mass, or fat mass basically anywhere else is mostly just pulling you off the wall (a strong core helps too but it doesn't need to weigh a lot).
> (...)
...then this would be very valuable data.
Eventually you could train AI to generate routes, and then you could fire all of your master setters!
Climb setters should unionize and copyright their art before it's too late (like for painters and software developers and musicians)
Anyway, I think this could work really well as an app. Good luck.
But really though, awesome project and congrats on getting it to this point! I'm excited for the day when something like this can be used to critique form
I do BJJ and wonder how it could AI models could help. I find watching tape to help, not only myself but everyone who I know has recorded their matches has found it helpful.