Interaction in 3D

Almost everything you do in a spatial interface reduces to four core actions, and naming them makes the design tractable. Selecting is picking one thing out of the scene to act on - a wall, a piece of furniture, a menu item. Manipulating is changing that thing's position or orientation - grabbing it, moving it, rotating it. Navigating is moving yourself through the space - walking, teleporting or flying to a new viewpoint. Scaling is changing the size of the model or the world - shrinking a whole building to a table-top model you can see at once, or growing it to true 1:1 so you stand inside it. Every richer task is built from these four, so getting each one to feel right is most of the work.

Each verb has natural techniques and natural failure points. Selection can be by direct touch when the thing is within reach, or by pointing a ray at it when it is further away, or by looking at it (gaze) - but small or distant or overlapping targets are genuinely hard to hit, far harder than clicking a precise pixel with a mouse, so selection in 3D needs generous target sizes and forgiving aim. Manipulation by hand is the medium's showpiece, direct and intuitive, but it is imprecise for fine work and tiring to sustain. Navigation is where comfort is won or lost, because moving the viewpoint is the main trigger for motion sickness. Scaling is a genuine superpower with no clean equivalent on paper - the ability to fluidly move between doll's-house overview and true-scale immersion is one of the things spatial computing does that drawings simply cannot.

Designing interaction, then, is largely about choosing the right technique for each verb given the task, the distance and the user, and keeping the vocabulary small and consistent so the person is not learning a dozen gestures. A first-time client should be able to do the whole review with two or three simple, well-chosen actions. The complexity you can pile onto a mouse-and-keyboard expert does not transfer - in the air, every extra gesture is one more thing to remember, mis-perform and tire the arm doing. And whatever the interaction lets someone move or measure on screen, the binding dimensions still come from the verified model and drawings, not from where a hand happened to drop an object.

Direct manipulation - reaching out and handling digital objects as if they were real - is what people mean when they say interaction in 3D feels natural, and the feeling is not an illusion. It works because it borrows the body's lifelong, pre-trained skill at handling physical objects: you already know how to reach, grasp, turn and place things, so a well-tracked hand lets you transfer that skill directly onto the model with almost no learning. For design tasks this is genuinely powerful - arranging furniture by picking it up and putting it down, rotating a component to inspect it, pulling a model open - because it collapses the translation between intent and action that a mouse and a set of menus always impose.

The cost is equally real and worth stating plainly. First, there is no haptic feedback: your hand passes through the virtual object and closes on nothing, so without a deliberate substitute - a highlight, a snap, a sound, a subtle vibration from a controller - the person genuinely cannot tell whether they have grabbed the thing. Good 3D interaction therefore leans heavily on visual and audio feedback to stand in for the missing sense of touch. Second, there is fatigue: holding the arms up in mid-air to manipulate floating content produces gorilla-arm tiredness within minutes, so interactions must be short, hands allowed to rest low, and long sustained mid-air work avoided. Third, there is imprecision: hands in the air jitter and drift, so direct manipulation is superb for coarse, spatial, approximate moves and poor for fine, exact adjustment - the very opposite of a mouse.

This is why controllers have not simply been replaced by bare hands, and why the honest designer treats hand tracking as one tool among several rather than the obvious best. A tracked controller adds a physical button that clicks and a body that can buzz, restoring some of the feedback and precision that bare hands lack, at the cost of the natural, empty-handed feel. Gaze and voice can select and command without lifting an arm at all, saving fatigue for people who will wear the device a while. The craft is matching technique to task: hands for the expressive, spatial, brief and approximate; a controller or precise input for the exact; gaze and voice to rest the arms - and always feedback to replace the touch that is not there. None of these change where the binding truth lives, which remains the verified model and drawings.

It is worth being precise about why the hard-won conventions of 2D interface design do not simply carry over, because assuming they do is a common and expensive mistake. On a screen, interaction happens on a single flat plane through one pointer with infinite precision and no fatigue: the cursor goes exactly where you put it, targets can be a few pixels wide, everything is the same comfortable distance away, and you can click all day. Decades of refinement - menus, buttons, drag handles, tooltips, right-click - all assume that flat, precise, tireless, bounded surface. Almost none of those assumptions hold in three dimensions.

In 3D, interaction happens across a volume with six degrees of freedom (three of position, three of rotation), through hands, gaze and body that are imprecise and tire, aimed at targets that sit at different depths and distances, some too far to reach. A menu that works beautifully on a screen becomes a floating slab you must physically reach or point at; a small close button becomes a target you fumble; a precise drag becomes an approximate shove. The extra dimension is not just more room - it is a fundamentally harder aiming and manipulation problem, because depth is genuinely difficult to judge and act in, and because the input devices are your own imperfect limbs rather than a precise mouse.

The practical consequences are consistent. Targets must be larger and more forgiving. The interaction vocabulary must be smaller and more consistent, because there is no muscle memory bank to draw on and every gesture must be learned. Feedback must be stronger, standing in for the touch and the precise cursor position that are missing. And a great deal of interaction is better handled by not doing it in mid-air at all - snapping objects to sensible positions, offering constrained rather than free movement, and pushing precise or dense work to a controller or a companion screen. The lesson is not that 3D interaction is worse than 2D - for spatial, expressive, approximate tasks it is far better - but that it is *different*, with its own strengths (directness, scale, spatial understanding) and its own weaknesses (precision, fatigue, reach, target acquisition), and that designing it well means respecting that difference instead of porting screen habits into the air.

Pulling the honest threads together gives a working map of where interaction in 3D delights and where it grates - the map a designer needs to decide what to build. It feels natural when it matches how the body already handles the physical world: grabbing and moving a model at arm's length, walking a few real steps to change your view, rotating an object in your hands, scaling the world between overview and true size. These borrow deep bodily skill, they exploit the medium's real strengths, and they are exactly the tasks - spatial, approximate, expressive - that a mouse handles poorly. This is where you should lean in, because it is where 3D interaction is genuinely better than a screen and where it earns the headset.

It feels awkward when it fights the body or demands precision the hands cannot give. Typing anything more than a few characters in mid-air is miserable, so text entry belongs on a real keyboard or a companion device. Fine, exact adjustment - nudging a dimension by a millimetre, precise alignment - is frustrating with jittery hands and better done with a controller, a snap or a screen. Reaching things across the room fails because arms are short, so distant selection needs a ray or teleport, not a stretch. Long sustained mid-air manipulation exhausts the arms. And smooth artificial locomotion - gliding or flying the viewpoint while your body stands still - is the classic route to motion sickness, which is why teleporting is so widely preferred. Each of these is a place to design *around* the hand rather than doubling down on it.

The mature stance mirrors the whole course: interaction in 3D is a powerful tool for the tasks it suits and a poor one for the tasks it does not, so make each interaction earn its place rather than doing everything by hand in the air because the first grab felt magical. Keep the vocabulary small, forgiving and consistent; give strong feedback for the missing touch; let arms rest; choose teleport over glide for comfort; and route precise, textual and dense work to inputs that handle it well. Do that, and the natural magic of reaching into space is preserved for the moments it truly serves - while the awkward edges are quietly designed out, and binding measurements and decisions stay, as always, with the verified model, drawings and the specialists rather than with wherever a hand left an object floating.