Designing UI for VR Hand Tracking Applications

(circuitstream.com)

58 points | by barbelldan 1395 days ago

5 comments

  • monkeydust 1394 days ago
    Been working on a VR dataviz application at work that can run on the Quest. Started in Jan. Got a few weird looks from top management. Covid-19 happened, WFH a thing, top management now interested in new methods of collaboration and wanta demo.
    • granshaw 1394 days ago
      I think you just gave Scott Adams a full Dilbert episode on a silver platter! :)
    • zmmmmm 1394 days ago
      Have been thinking of doing something similar. The sad reality is that social distancing and WFH also impose huge barriers because nobody has a headset and sharing them is now both inconvenient and unsanitary. Hoping that post-COVID it doesn't form a permanent setback for headset market growth.
      • monkeydust 1394 days ago
        Lack of access to headsets is a massive issue for the industry. I do think if people could 'try' XR easily somewhere it would help with adoption but I struggled to find anywhere around London months back where you could do that.

        What is worth noting though is the Qualcomm release* last week on their XR plans. It is interesting because they are tying it to 5G. Mobile network operators who have spent billions on the infrastructure and now need to find compelling use-cases for customers are partnering up with them. This could mean you get a XR headset with your next mobile phone plan.

        * https://www.qualcomm.com/news/releases/2020/05/26/qualcomm-c...

  • keenmaster 1394 days ago
    VR hand tracking will be better as an input method when the user has little or no expectation of physical resistance. Imagine being in a wizard game where you have hundreds of spells memorized - water spells, fire, electricity, teleportation, telekinetic manipulation, etc.... all of which are activated by esoteric hand gestures. Make it an RPG with UE5-demo level graphics and it will be one of the most mind-blowing games ever.
    • airstrike 1394 days ago
      Would love it even more if the game had hilarious consequences for not making the right gestures
      • novok 1394 days ago
        I get that feeling already when I tried half life alyx's gesture to gravity grab objects in the air.
    • GuiA 1394 days ago
      Step 1: Make an amazing game with gestures

      Step 2: ???

      Step 3: Profit!

      Joke aside, such game mechanics, albeit with cursor gesturing rather than hand gesturing, have been tried many times before. See Arx Fatalis, Black & White, etc. Jonathan Blow has a pre-Braid prototype about that as well that he shows in some detail in a talk he gave.

      The reality is that sure, it sounds cool when you describe it like that, but there are many design issues to contend with:

      - how does the player remember all these obscure symbols ?

      - how do you deal with the frustration when a symbol isn't recognized? When it's confused by the system with another symbol?

      Etc. On top of that, there is the fact that drawing squiggles to cast spells is not a game mechanic that is inherently extremely fun. Why are people going to the bother of doing that vs pressing a button?

      It's all a lot of work to make it make sense, and there is little reason to believe that gesturing with your hands instead of your mouse would solve some of the more fundamental problems.

      • keenmaster 1394 days ago
        The limitations you describe are rooted in a pre-VR world. VR is much more conducive to gestures. In Half Life Alyx, Valve could have technically allowed you to lob a grenade by just pressing a button "because it's easier." But they didn't, and opted for a gesture-based input. They could have allowed you to turn a wheel by mashing a button, but they didn't, and you actually have to turn the wheel in circles. These decisions would have been insane with a keyboard and mouse as inputs. They're natural for a VR game.

        Without a keyboard, it's cumbersome to switch between a bunch of spells. Gestures are both more immersive and functionally better. Maybe most people wouldn't memorize hundreds of spells, but a couple dozen are certainly feasible. There could be a spell "language," where the initial part of the gesture could correspond to the category of spell. Fireball, fire wave, etc...can start with the same basic gesture. All wave spells can end with the same gesture. Hundreds of spells can be constructed from permutations of 20-30 foundational gestures, kind of like how a core group of trilateral roots in Arabic and Hebrew can construct a much larger set of words (sometimes without previous knowledge of a specific word/permutation) https://en.wikipedia.org/wiki/Semitic_root

        • GuiA 1394 days ago
          The examples you mention work well because there is a real world affordance (the wheel, the act of throwing) that the gesture maps to. I agree that rotating a wheel gesturally makes more sense as an interaction in VR than the same action does with a mouse pointer.

          The grenade example is a good one, because throwing/flicking doesn't require much feedback, unlike say rotation or fine grained manipulation. Of course the interaction would feel nicer if you could feel the weight of what you're throwing - the physics of throwing objects always feels kind of off in VR games - but it still works well enough. Note that many games did this with the Wiimote, and there is nothing about that interaction that is singular to VR.

          However, the moment you lose that affordance based in a real world object, and ask your user/player to make gestures in mid air based on recollection, things really do start to fall apart. There is a host of HCI literature/projects around this - e.g. check out papers/demos from the Kinect era. An RPG game like the one you describe could be made with Kinect, and I vaguely recall a few in development - but Kinect demonstrated well that the only kind of gestural interaction that works in long term sessions is dancing.

          Btw, this is why if you go to e.g. the VR center in Tokyo, almost every booth has physical props for you to hold/grip/sit on while you're fishing/golfing/shooting a gun/riding a bike in VR.

          • keenmaster 1394 days ago
            I suspect that gesture input adoption, categorically, has emergent properties. Many different things have to "go right" for the input category to take off in a way that is hard to grok.

            The use of gestures has to appeal to hardcore gamers (in this case). The motion tracking needs to be sufficiently advanced. There can't be a swifter, dopamine enhancing alternative (like mashing buttons on a keyboard). A critical mass of developers need to make games with gestures as a core mechanic. Etc...

            I offer Pavlov VR as an example. Pavlov is like Counterstrike, except many of the mechanics are gesture based.

            - You grab your gun off your back

            - You actually need to look through the scope for zoom, physically bringing your head and hands into alignment

            - You have to reach over your left shoulder to activate the walkie-talkie for remote communication

            - Reloading is a multi-step process and varies from gun to gun

            You might say that I'm proving your point about gestures mapping to real-life, but that's not why I bring up Pavlov. I'm using it to illustrate the aforementioned emergent properties, because all of those gestures add up to something unexpected: even with subpar graphics and a low budget, it is fun. Not only that, but it is fun in an unexpected, novel way for FPS games. Suddenly, your kill count, headshots, etc...don't matter as much. The immersion as well as the social element of the game are huge. You are so zoomed in, so in the moment, when you are in a firefight, poking out your head out to snipe the sniper before rushing back below the barrier, in a way that you wouldn't be in a 2D game.

            Microsoft could technically have made a Pavlov Kinect, but who would play it? Almost no one. People would complain that traditional controls make much more sense, and they would have a point. It's simply awkward to squirm around in front of a 2D screen. There's a disconnect between the input modality (3D gestures) and the display modality (2D screen) when using the Kinect. In VR, both the input and the display are in 3D harmony.

          • Miraste 1394 days ago
            The only gesture input the Kinect could recognize reliably was dancing. I don't think that says much about the appeal of gesture controls in a system that can see finer movements. As a counterexample, iOS and MacOS have a lot of gestures which aren't based on real world motions, and people use those all the time.
            • GuiA 1394 days ago
              Kinect offered good full body skeletal tracking. Saying that it could “only recognize dancing” is objectively wrong. And even if that was the issue, look at eg Magic Leap, which had excellent finger tracking. What compelling things were built for it?

              The issue is that “gestures” are not a binary thing that you do or not; they live on a continuum. What is waving vs raising your hand to swat a fly? Categorizing these “gestures” in a meaningful way and layering effective gameplay on top of it is the challenge, that pretty much no one solved despite the huge incentives.

              Gestures on two-dimensional multitouch screens are not what we’re talking about here (and even then, you can’t expect people to remember much more beyond the 5/6 core ones).

      • Swizec 1394 days ago
        > drawing squiggles to cast spells is not a game mechanic that is inherently extremely fun

        Drawing squiggles isn't inherently fun. But dancing your spells now that's hecking fun.

        Human culture already uses lots of weird hand symbols to cast all sorts of spells. You can say "hello" with a hand gesture. You can cast teh "I'm sorry spell". There's a great "I need your attention" spell, another for "I can't understand/hear" ... and let's not forget the wonderful repertoire of spells for "You're an idiot"

        Hand and body gestures are a fantastic user interface. Computer recognition might not be quite sophisticated enough yet.

      • nico_h 1394 days ago
        I think a starting point would be Avatar , the cartoon. The spells were based on mostly full body martial arts, but each element had a distinct flavor.
        • GuiA 1394 days ago
          That’s actually a very neat line of thinking! If you could model it in terms of gait (slow and heavy vs light and limber) rather than specific gestures, it could make for a compelling system.
      • nico_h 1394 days ago
        I raise Beat Saber as a counter example: it would be less tiring to press buttons than to flail your arms and wrists, but somehow it would not be as fun.
        • GuiA 1394 days ago
          Beat Saber is a great example - it asks you to do something tiring, but it’s super fun and you have immediate feedback so the effort is worth it.

          Games can ask players to do almost anything as long as they’re having fun in return - but that means that the more outlandish things you ask them to do, the more fun they must have.

          • keenmaster 1394 days ago
            >as long as they’re having fun in return

            This is true, but not in a a way that maps directly to the gesture. For example, reloading in Pavlov is not fun in isolation. It’s a relatively tedious process that mirrors reloading in real life. It is rewarding to successfully reload while you’re taking fire and then proceed to kill your enemy. As a result, reloading gesture > reloading button, even though you need to memorize a different reloading action for each gun (it becomes muscle memory).

  • warp 1394 days ago
    Using _just_ hand tracking seems like a bad way to interact with anything.

    I have an Oculus Quest, and the hand tracking is like the Kinect all over again -- without having physical buttons every interaction is slower because the software waits for you to keep a particular gesture/pose long enough at a particular location to register as a "click".

    Obviously the tracking in Quest is more fine-grained than the Kinect was, so perhaps we'll see some interesting uses for the technology in the future. But right now I'm not sold on the tech.

    • zmmmmm 1394 days ago
      Yes, it's disappointinly hard to use the whole UI with hands compared to the controller. I guess it's a mix of improving the tracking and getting the UI right. But even from where we are now, projecting forward to perfect tech I am still not sure that I want to use hand tracking over controllers for most things. A few types of games and other applications could clearly benefit, but mostly controllers >> hands.
    • ramblerman 1394 days ago
      I wonder if that is just people lazily mapping old paradigms onto this new space. Holding a gesture to simulate a button is a perfect example of that.

      While in the demo of the cat, you see all kinds of sliders being manipulated as well as rotations and pinches that seem instanteneous.I think they got it right by making the buttons virtual entities.

    • melling 1394 days ago
      Considering that we’re going through a pandemic, I can imagine a lot of things I’d like to interact with by only using hand tracking, outside of VR.

      Perhaps this necessity with be the mother of invention.

    • keenmaster 1394 days ago
      For sure, but gestures and controllers are not mutually exclusive. Specific controllers can be made for wizard games. Wands come to mind. Even without a wand, the Oculus controller can be combined with gestures just fine.
  • Qahlel 1395 days ago
    VR will never take off (at least in our lifetimes):

    1. battery issues. 2. weight issues. 3. quality issues. 4. VR is a niche market.

    • dividido 1394 days ago
      It's already taking(taken) off. I don't think any of the 3 points up top are typically valid and linked together. I use an "old" oculus rift setup that is tethered to my desktop with a 1070ti. The only batteries to speak of are in my hand controls and they last months. The head piece is not heavy, so you might be referring to a wearable pack seen in vr installations mixing vr in a closed set/space?

      Quality wise, visuals are pretty amazing. Good enough on mainstream games like Half-Life Alyx, GTA5, Pavlov and Lone Echo to make me think and feel like I'm in the environment. As a private pilot let me tell you that flying in vr in X-Plane 11 is just amazing. 360 views and interactions with knobs/levers in the cockpit. Even more, utilizing more advanced setups in unreal engine and megascans/quixel goes to another level.

      It's still not entirely accessible to everyone in terms of cost as it's not cheap. Especially if you want the "best" graphics which requires the headset and a computer with decent gpu. So I do agree with point 4 and being a niche market.

      I wonder what "taking off" to most people is? It's definitely not Lawnmower Man/the matrix/ready player one yet. It's cumbersome when using a tethered setup and room scale is amazing but I don't exactly have a dedicated play space to wander within or a specialized endless treadmill.

      It does however provide an interesting escape with unique experiences. Something as simple as google maps is mind blowing when traveling around the world. I was recently working on a house project and needed to figure out the remodel. I modeled the house in Maya, imported it into UE, and was able to put on my oculus and "walk" around the space.

      For me it's hard to go back to "flat gaming" after vr. I really enjoy it and look forward to seeing how it continues to evolve.

      • outworlder 1394 days ago
        > It's definitely not Lawnmower Man/the matrix/ready player one yet

        I think we have surpassed Lawnmower man already. We are only missing the extra hardware, like the full body suits and moving beds/leonardo devices. And even then, not really: Lawnmower man happened at a well funded laboratory, the scientist just had some expensive company hardware at home. We have better quality headsets at home today.

        Ready player one was... inconsistent. They handwaved the movement problems.

        > if you want the "best" graphics

        This will always be true, VR headset or not. If you want the 'best' graphics, you always have to for over a lot of money for specialized GPU hardware. The thing is, it's perfectly fine to use an untethered Quest to play something like Superhot or Beat Saber.

        > For me it's hard to go back to "flat gaming" after vr.

        This is why I think the market is going to explode. Every single person I've shown it has walked away impressed. Young, old, doesn't matter. Even non-gamers.

        Oculus Quest and similar devices are on the right track.

        > I modeled the house in Maya, imported it into UE, and was able to put on my oculus and "walk" around the space.

        I'll definitely try this!

        I wonder when 3d Modeling will be primarily done with a headset. Like, would it be useful to do the initial modelling in Maya itself?

        • dividido 1394 days ago
          > I wonder when 3d Modeling will be primarily done with a headset. Like, would it be useful to do the initial modelling in Maya itself?

          I think the desired goal would help dictate if you use VR. For example, for the Lion King remake the world was entirely cg. To do the rough and final layouts of the environment multiple people (director/art director/modelers/set dressers/dp) all went into vr (sometimes together) and used tools in UE to place/scale/rotate objects. This made a lot of sense b/c they could more accurately place and model the environment within it.

          It could be quite useful when modeling but I wonder at what point make sense. I've done a fair bit of modeling which involves a lot of manipulation of points and faces and also involves lots of object tumbling, which might prove tiring in vr. But I think it ultimately makes sense b/c it allows for a 3dimensional view of what you're doing.

          • outworlder 1394 days ago
            > To do the rough and final layouts of the environment multiple people (director/art director/modelers/set dressers/dp) all went into vr (sometimes together) and used tools in UE to place/scale/rotate objects.

            I didn't know about that. This is amazing.

            > I've done a fair bit of modeling which involves a lot of manipulation of points and faces and also involves lots of object tumbling

            I've done some hobbist-level CAD. Very often I had to turn the object slightly in a couple of directions to get a better sense of perspective. That comes for 'free' with VR, and you can also move your head, or yourself, without turning.

            Mind you, I'm picturing working while sitting down so as not to be too tiring. Fine vertex manipulation may still be tiring, but I'm thinking organic modelling would benefit. Also specially for CAD, when you want to visualize and "explode" your model. Normally it involves lots of camera moving around.

      • ShamelessC 1394 days ago
        > GTA5

        Curious, are you using Vorpx to play GTA V in VR? I need to give that a try.

        My biggest issue with non-vr games in VR is that you don't get to use your hands to aim. Would be amazing if a mod could get that to work.

    • outworlder 1394 days ago
      This comment will not age well.

      1- Battery? I use a Rift, it is fine, it takes power from my PC. For the portable Quest, you get 2 to 3 hours of constant usage. That's pretty good

      2– Weight issues: headsets are not very lightweight today. They are not ridiculously heavy either. Have you even tried one? My mom has, she's a senior. It's fine. They should be lighter if we are expected to throw away our monitors and use them exclusively on a day job. For normal usage? They are fine. If anything, the heat (and associated sweat, specially with physical activity) is much more of a problem. Again, I doubt you have tried one. For me, the main problem is using with glasses, it is not completely straightforward to put them on.

      3– Quality issues, what quality issues? Do you mean QA or resolution? If it is resolution, it's already pretty good unless you are doing lots of reading (and even then YMMV, there are people coding with them already). There are many other things that need to be improved before resolution becomes the main factor. The fresnel lens used in some headsets have some strange aberrations, specially with primarily dark scenes.

      4– This objection is straight from the Millennium Bug era. It's mostly(not only) used by a subset of the gaming market, but that in itself is huge. There are entire industries in other fields which are smaller than the current VR market.

      Yes, improvements are needed. They are coming, if headsets for professional applications are any indication. VR could be said to be 'niche' when headsets cost multiple thousand dollars. Now? You can get decent ones for the price of a good monitor – and the thing actually has two monitors running at a much higher refresh rate, plus a whole lot of extra hardware.

    • ipsum2 1395 days ago
      People have bought $100 million worth of VR content on the Quest https://techcrunch.com/2020/05/18/oculus-surpasses-100-milli....
    • shafyy 1394 days ago
      Unless your life ends in the next 5 years: you couldn't be more wrong. Not going to go into details, the others comments have laid out the arguments pretty well =)
    • AurelioB 1394 days ago
      Mobile computers on our hands!? they will never take off (at least in our lifetimes):

      1. battery issues. 2. weight issues. 3. quality issues. 4. mobile computing is a niche market.

      • machinehermit 1394 days ago
        To take a counter view.. I have literally been waiting to get into VR since 1993 and the Lawnmower man.

        Why do I not have a system?

        It is not cost. It is not building the system.

        I would say the main reason is I don't like video games.

        There is just something so lacking with VR if you don't want to play games.

        I remember the old MIT media lab mobile computing devices and videos in the 90s. They were nothing like a smart phone. More like the failure of google glass.

        I do wonder if the headset is simply a wrong path.

        • outworlder 1394 days ago
          > There is just something so lacking with VR if you don't want to play games.

          Yes, it's lacking more apps. Which is rather a Catch-22. There has to be demand for them. Games are just an 'easy' (relatively speaking) medium to get into.

          There are a few practical concerns of course. In a work environment, your manager can no longer easily interrupt you. They can't easily keep tabs on what you are doing either. This is enough to kill it at a workplace – maybe an exception for some engineering/medical settings.

          If you want to experiment with something that's not a game, take a look at this: https://github.com/SimulaVR/Simula

    • bigyikes 1395 days ago
      4 seems like a consequence of the first 3 points, and I see no reason why we shouldn’t expect to see further progress with those.
    • gfodor 1394 days ago
      I upvoted you so as to not prevent your "Less space than a Nomad. Lame." post from settling into its rightful place in history :)
    • yboris 1395 days ago
      "never" seems like a strong word when talking about technological developments over a 30+ year horizon.
    • iphone_elegance 1395 days ago
      That's fine but it doesn't have to 'take off' everywhere to be useful and interesting
    • cjsawyer 1394 days ago
      Why are you on HN if you’re going to just be dismissive of exciting technology?