The Limits of AR & MR

The previous lesson opened up the first two limits in depth, so name them briefly here as part of the full set, then add the one most people notice first. Registration error and drift mean the overlay is never exactly where it should be and does not stay put: at any instant the model sits a little off its true place (centimetres at room and building scale, worse far from the anchor), and that offset changes as you move and as time passes, occasionally jumping when the tracking loses lock. This is structural, not a passing hardware flaw, and it is why AR and MR are for judgement and never for setting out.

The third limit is the one that gives AR away instantly: broken occlusion. In the real world, nearer things hide the things behind them - a real column blocks your view of a sofa behind it. For an AR overlay to look right, the device must know the real geometry well enough to hide the parts of the virtual model that are behind real objects, and reveal the parts in front. This is genuinely hard, because the device only has its rough, live map of the real surfaces to work from. So the common failure is that virtual objects float incorrectly in front of real ones that should hide them: the virtual sofa appears pasted over the real column instead of tucked behind it, the virtual wall shows through a real person walking past, the design hovers on top of reality rather than sitting inside it. Good mixed-reality devices are improving at occlusion - sensing real depth and hiding virtual content behind real surfaces - but it remains imperfect, especially for thin, moving or complex objects and at the edges where virtual meets real.

Why does this matter beyond looking unconvincing? Because occlusion is a huge part of how we judge spatial relationships. If the virtual island is drawn on top of the real column rather than correctly behind it, you cannot properly read how they sit together in space - the exact thing you brought the headset to judge. Broken occlusion does not just spoil the illusion; it degrades the spatial understanding the overlay is supposed to give. It is a limit to design around: place and interpret virtual content knowing the device may not correctly resolve what is in front of what, and treat depth relationships near real objects with caution.

The fourth limit is field of view, and it is one of the most under-reported gaps between the demo and the reality. Your natural vision spans a very wide arc, close to 180 degrees side to side, filled continuously with the world. Many AR and MR displays, especially see-through ones, can only place virtual content within a much narrower central window - a letterbox floating in the middle of your vision. Outside that window the digital content simply is not drawn, so a full-size virtual wall is cut off at the edges, and to see a large model you must swing your head around as if peering through a small frame held at arm's length. The real world remains fully visible (that is the point of AR), but the added design is confined to a central patch.

For design this narrow window quietly undermines the very thing AR and MR are best at - understanding space at true scale - because a huge part of spatial understanding is peripheral: the sense of enclosure, of a wall looming beside you, of the whole room around you, comes from your wide field of vision. Squeeze the virtual content into a central letterbox and you lose much of that felt envelopment; you are inspecting the design through a window rather than being surrounded by it. It is improving with each hardware generation, but it remains a real constraint, and one worth setting client expectations about before a review.

The fifth limit bites hardest exactly where MR on site is most exciting: washout in bright light and outdoors. See-through AR and MR work by adding light to the scene, and added light can only compete with the ambient light already there. Indoors in controlled lighting the virtual content looks solid and bright; step into direct sun - abundant across India for much of the year - and the ambient light overwhelms the display, so the virtual model fades to faint, washed-out ghosts, or becomes almost invisible. The outdoor site walk, one of the medium's most compelling promises, is precisely where the display struggles most. There are partial answers (dimming visors, brighter displays), but bright-light washout is a genuine current limit that makes many outdoor AR uses fragile, and it is honest to plan around it - shade the review, pick the time of day, or accept that outdoors the overlay may be too faint to rely on even for judgement.

The sixth limit is subtler than the others but matters for design decisions: depth and lighting realism. Even when registration, occlusion, field of view and brightness all behave, virtual objects often still do not sit fully convincingly in the real scene, and the reasons are worth understanding because they affect what you can trust the overlay to tell you.

Depth cues can conflict. Your eyes judge distance from many signals - stereo disparity, focus, how objects occlude each other, how they are lit. AR displays get some of these right and others wrong or inconsistent, so a virtual object can feel like it is at a slightly different distance than it should, or sit in an uneasy no-man's-land relative to the real things around it. When occlusion is also broken, the confusion compounds. Lighting does not match. In reality every object is lit by the actual light in the room - the direction of the sun, the colour of the walls, the soft fill from the sky - and casts real shadows onto real surfaces. A virtual object, unless the system works hard to match the real lighting, is lit by its own computed light that does not agree with the room: its shadows fall the wrong way or are missing, its brightness and colour temperature do not match the real daylight, so it reads as pasted on rather than present. Advanced systems estimate the real lighting and add plausible virtual shadows, which helps a lot, but a perfect match is rare.

For design this has a specific consequence: be careful judging material, colour, light and fine spatial fit from an AR or MR overlay. The medium is excellent for the big spatial questions - does this layout work, is this corridor tight, does this massing suit the plot - where a little unrealism does not matter. It is unreliable for the delicate ones - exactly how this marble reads in this room's light, whether this shadow gap looks right, the precise perceived depth of a reveal - because the depth and lighting are approximations that can mislead. This is the same discipline as everywhere in the course: know which questions the tool answers honestly and which it does not, and take the delicate material-and-light judgements to real samples, real mock-ups and the real space, not to the headset's approximation.

Gather the five limits and a clear map appears of when to put the headset down. This is not a grudging admission; it is the positive skill the whole course teaches - making immersion earn its place, and recognising the many moments it does not.

When you need a precise, measured, legal record, use a drawing. The overlay drifts and is approximate; a drawing is exact, unambiguous, shareable to everyone without a device, and the accountable document the whole project is built and signed off from. For dimensions, details, specifications and the record, the humble drawing wins outright. When you need certified position on site, use a total station and a surveyor. Boundaries, levels, footprints, setting-out and structural clearances demand millimetre, accountable accuracy that an estimated, drifting overlay cannot give - the previous lesson made this non-negotiable. When bright sun or an open outdoor site will wash out the display, a drawing or a screen is more reliable than a ghostly overlay you cannot see. When the judgement is about fine material, colour and light, use real samples and mock-ups in the real space, not the headset's approximate lighting. When the work is fast solo editing, detailing or documentation, a screen is quicker than reaching for a headset with its preparation, weight and narrow window. And when a headset would isolate or exclude people - a review where not everyone can be equipped, or where someone gets motion-sick or cannot comfortably wear one - a shared drawing or screen keeps everyone in the conversation.

What is left is where AR and MR genuinely shine, and it is a real, valuable territory: the true-scale, in-place, shared understanding of a design - the tabletop model a group gathers around, the empty room walked at 1:1, the design set onto the real site to judge feel, clash, sightline and context. Used there, within its limits, deferring every measured and binding result to the drawings and the survey, AR and MR earn their place clearly. Used beyond there - as a measuring tape, a record, a material sample, or an all-day tool in the sun - they disappoint. The competent designer holds both truths at once, and that balanced judgement, not any device, is the durable skill.