Lesson 4.1Lesson 4.1 · Neural Capture: NeRF & Gaussian Splatting
The Neural Turn in Capture
For a decade capture meant measuring geometry directly; now machine learning reconstructs whole scenes and invents new viewpoints from a handful of photographs, and the results are breathtaking, useful and quietly treacherous for measurement
A machine that has seen thirty photos of a room can now show you that room from a viewpoint no camera ever occupied — and make it look utterly real. That is the neural turn, and it changes what capture can mean.
For the first three modules of this course, capture meant one honest idea: to know where a surface is, you measure the distance to it, directly, with a laser pulse or with the geometry of overlapping photographs. The output was explicit — a point cloud, a mesh, coordinates you could dimension. You could trust a number because a number had been measured.
The neural turn breaks that assumption in the most interesting way. Instead of measuring geometry and storing it, you hand a machine-learning model a set of ordinary photographs and let it *learn the scene* — learn how light leaves every surface in every direction well enough to paint the scene convincingly from viewpoints no photo ever captured. The results, from Neural Radiance Fields (NeRF) in 2020 to 3D Gaussian splatting in 2023, are so photorealistic they can be mistaken for video. They are genuinely useful. And they carry a caveat you must hold from the first sentence: looking real and being metrically true are not the same thing, and this family is strong on the former while unreliable on the latter.
The neural turn: photos in, a learned scene out, new views invented. Beautiful to look at. Verify before you measure.
What actually changed: from measuring to learning
To see what is new, hold the old methods clearly in mind. Laser scanning measures distance directly: fire a pulse, time its return, record a point. Photogrammetry measures indirectly but still geometrically: find the same feature in many overlapping photos and triangulate where it must be in space. Both end in the same currency — explicit 3D geometry, points and surfaces with coordinates you can measure against. The computer stores *where things are*.
Neural capture stores something different: it stores *how the scene looks*. You feed a machine-learning model many photographs of a subject, each tagged with the camera position it was taken from, and the model trains until it can reproduce those photos. In doing so it learns an internal representation of the scene's colour and structure rich enough to render the scene from a brand-new camera position it was never shown. This is novel view synthesis — inventing a plausible photograph from a viewpoint that never existed. The two landmark techniques are NeRF (2020), which stores the scene inside the weights of a small neural network, and Gaussian splatting (2023), which stores it as millions of little coloured 3D blobs; Modules 4.2 and 4.3 take each in turn.
The shift is profound. Classical capture answers "where is this surface?" with a measured coordinate. Neural capture answers "what would a camera see from here?" with a rendered image. The first is a ruler; the second is a painter with an extraordinary memory. Both are built from the same raw ingredient — photographs or scans of the real world — and both belong to the broad field of computer vision and 3D reconstruction. But their outputs are not interchangeable. A point cloud is a measurement you can audit; a neural render is an appearance you must take on trust until you verify it. Understanding that one-sentence difference is the whole foundation of this module, and the reason a beautiful neural capture is never, by itself, a survey.
Old capture stores WHERE things are. Neural capture stores HOW the scene LOOKS. One is a ruler; the other is a painter with perfect memory.
Why it is exciting: photorealism and new viewpoints
The excitement is real and worth naming precisely, because neural capture is not hype for its own sake — it does things the older methods genuinely struggle with. The headline is photorealism. A laser scan of a carved temple column gives you exact geometry but a rather clinical, sometimes patchy appearance; the colour is sampled coarsely and the subtle way light plays across the stone is lost. A neural capture of the same column can look like a film of it — every highlight, every soft shadow, the sheen of polished surfaces, all reproduced so faithfully that a viewer forgets they are looking at a reconstruction.
The second gift is view-dependent appearance, the thing that makes surfaces look alive. Real materials do not look the same from every angle: a glazed tile, a mirror, a lacquered door, water, a glass facade all change as you move. Classical geometry-plus-texture handles this badly, because a single colour is baked onto each point. A NeRF learns colour *as a function of viewing direction*, so reflections and glints move correctly as you orbit the scene — something point clouds and meshes simply cannot do. It also means neural methods cope gracefully with the soft, fuzzy, semi-transparent things that wreck laser scans and photogrammetry: foliage, hair, lace screens, fine grilles, thin edges.
Third, it is accessible. The raw input is just photographs or ordinary video — no lakhs-rupee scanner, often just a phone. For visualisation, presentation, virtual and augmented reality, marketing walkthroughs, and capturing the *visual richness* of a space that a point cloud flattens, neural capture is extraordinary: you can hand a client an immersive, fly-anywhere, photoreal model of their building captured in an afternoon. That is a capability the profession did not have five years ago, and it is why the field has moved so fast. Hold the excitement — and keep reading, because the same properties that make it dazzling are exactly what make it unreliable for measurement.
Why it needs care: appearance is not measurement
Here is the discipline this whole module exists to instil. The very mechanism that makes neural capture beautiful — optimising a model until it *reproduces the photographs* — optimises for appearance, not for metric truth. The model is rewarded for rendering images that match the input photos. It is not rewarded for getting the real-world distance between two walls correct. So a NeRF or a splat can look flawless from every trained angle while its underlying geometry is soft, locally distorted, or simply invented in places the cameras did not see well.
Three specific cautions follow. First, scale is not inherent. Classical capture ties into real units through measured distances or survey control; a raw neural capture from phone photos has no idea whether the room is three metres wide or thirty, unless scale is imposed externally. Second, geometry can be confidently wrong. Neural methods produce "floaters" (wisps of density hanging in empty air), smear thin objects, and guess at hidden or poorly-photographed regions in ways that look plausible but are not measured. A flat wall may render perfectly yet not be flat in the model. Third, there is no honest error figure you can read off. A survey-grade laser scan comes with a stated accuracy; a neural capture comes with a subjective impression of realism, which is not the same thing and can be dangerously persuasive.
The professional consequence is simple and non-negotiable, and it echoes the whole course: treat a neural capture as a visualisation, not a survey. If a dimension off a neural model will drive a cut, a fabrication, a boundary, a setting-out or a structural decision, you do not trust it — you verify that dimension against a laser scan, a photogrammetric model built with control, a tape, or a licensed surveyor. Neural capture is a superb way to *see* a place and a poor way to *measure* it, and the rest of this module (4.2, 4.3, 4.4) unpacks exactly why, method by method, and where it does belong.
The model is trained to MATCH THE PHOTOS, not to get distances right. Looks perfect, measures unreliable. Verify every load-bearing number.
Where the neural turn sits in capture — and in this module
It helps to place neural capture on the same map as the methods you already know, rather than treating it as a rival that wins or loses outright. Think of two independent axes. One is photorealism and visual richness — how convincingly the result reproduces the look of the real place. The other is metric reliability — how much you can trust a dimension taken from it. Laser scanning sits high on metric reliability and modest on photorealism: superb for measurement, visually plain. Photogrammetry sits in the useful middle: measurable when done with control, with good texture. Neural capture sits high on photorealism and, for now, low on guaranteed metric reliability: dazzling to look at, not to be dimensioned blindly. No method is "best"; each occupies a different corner, and mature practice chooses by what the job needs — or combines them, as Module 4.4 will show.
There is one more property of this field you must internalise: it moves extraordinarily fast. NeRF is from 2020; Gaussian splatting, which largely eclipsed it for real-time use, is from 2023; the literature churns monthly. Techniques that impose real-world scale, extract clean meshes, remove floaters, and tie neural captures to survey control are active, rapidly improving research. So every specific limitation stated in this module is a *snapshot as of writing*, not a permanent verdict. The honest position is to learn the principles — why appearance and measurement diverge, what each representation is and is not — so that when the tools improve, you can judge the new claims for yourself rather than believing a vendor demo.
Across the next three lessons we go deeper: 4.2 builds an intuition for NeRF — a scene as a continuous learned field of colour and density. 4.3 does the same for Gaussian splatting — the faster, real-time representation that took the field by storm. 4.4 draws the honest conclusion: where neural capture genuinely belongs (visualisation, presentation, VR/AR, view synthesis, hard-to-scan visual richness), where it does not yet belong (anything metric or binding), and how it complements rather than replaces measured capture. Keep the course's through-line in view throughout: capture is measurement, you must always know and check the accuracy, and binding results go to a licensed surveyor.
Appearance vs accuracy
What a neural capture optimises for
Training matches photographs, not distances; realism is not a stated accuracy. Treat output as visualisation. Principles here; binding accuracy follows verified specs and a surveyor.
Scale & georeferencing
Real-world units and position
Raw neural captures have no inherent scale or coordinate frame; imposing and checking them is a survey task. Defer georeferencing to control and a licensed surveyor (Module 1.3-1.4).
Verification of dimensions
Any number that drives a decision
Cross-check load-bearing dimensions against laser scanning / controlled photogrammetry / a tape, never off the neural view alone. Module 9 on quality and accuracy.
Fast-moving field
Currency of any accuracy claim
NeRF (2020) and Gaussian splatting (2023) evolve monthly; re-verify current capabilities and vendor claims rather than treating this snapshot as permanent.
Workshop — sort a set of capture tasks by what each one actually needs
The core judgement of this whole module is knowing when photoreal appearance is enough and when measured geometry is mandatory. In this workshop you build that instinct by triaging realistic tasks, before you touch any neural tool.
Just a notebook and your own project experience. No software or capture hardware — this lesson builds judgement, not a pipeline.
Goal: reliably separate 'needs measurement' from 'needs a good picture' Inputs: the task list below (or your own projects) + this lesson + a notebook Time: ~35 minutes
- 1List 8-10 realistic capture tasks across your work: e.g. a client walkthrough of a heritage haveli, a fit-out survey for new joinery, a marketing video of a completed villa, setting-out for a steel extension, documenting a carved facade, a VR tour for a competition, checking a floor is level, an insurance record of a space.
- 2For each, write the single decisive question: 'what will someone DO with this output?' and decide whether it must carry trustworthy dimensions or only needs to look real.
- 3Tag each task 'metric' (needs measured geometry), 'visual' (needs photoreal appearance), or 'both', and note which capture method you would use for the metric part (laser / photogrammetry) and which for the visual part (NeRF / splatting).
- 4Pick the one task most likely to be captured neurally by mistake, and write two sentences on what could go wrong if someone dimensioned from that neural model.
- 5Write a one-paragraph rule for yourself: a plain-language test you will apply in future to decide whether a neural capture is acceptable for a given job, and where you would still call a licensed surveyor.
You’ll walk away with
A one-page triage table: each task tagged metric / visual / both, the method chosen for each part, the one dangerous case flagged with its failure mode, and your personal rule for when neural capture is and is not acceptable. Keep it; it is the decision framework Module 4.4 formalises.
Three altitudes on the same idea
Read the band that fits you — or all three.
Neural capture gives you a new, cheap, photoreal way to record and present a site or building — but it does not give you measurable geometry, and confusing the two is where it bites. Use it for immersive client walkthroughs, design-review context, heritage visual documentation and VR/AR, captured from a phone or camera in hours. Do not let a dimension from a NeRF or a splat drive coordination, setting-out, fabrication or a structural decision: for the metric basis of the project you still commission laser scanning or controlled photogrammetry, and you still scan-to-BIM from that. The skilled move is to run the two together — a metric scan for the model, a neural capture for the experience — and to know, every time, which one a given number came from. Defer anything binding, georeferenced or legally consequential to a licensed surveyor.
For interiors, neural capture is a beautiful way to show a space and a risky way to measure one. A phone-shot NeRF or Gaussian splat of a room can give a client a stunning, fly-through record of an existing interior, its materials, its light, its character — richer than any point cloud for conveying atmosphere, and invaluable for mood, presentation and before/after storytelling. But when you need the real dimensions to design joinery, fit-out or furniture that actually fits — ceiling heights, out-of-square walls, the true opening for a wardrobe — do not trust numbers pulled off the neural view. Verify against a measured capture (even a careful tape or phone-LiDAR scan with checks) and flag anything binding for professional survey. Capture the look neurally; capture the measurements properly.
This is the newest, fastest-moving frontier of reality capture, and understanding it clearly — especially its limits — marks you out. Learn the one idea that matters: classical capture measures and stores geometry, while neural capture learns to reproduce appearance and synthesise new views, which makes it photorealistic but not reliably metric. Be able to explain why training a model to match photographs optimises for looking right, not measuring right, and why scale, floaters and invented geometry are the result. You are not expected to publish NeRF research; you are expected to reason about when a photoreal capture is trustworthy for dimensions (rarely, not yet) versus for visualisation (superbly), and to know that a beautiful render is never, by itself, a survey.
“NeRF and Gaussian splatting are just a better, newer kind of 3D scanning — they produce photorealistic models from cheap photos, so they make laser scanning and photogrammetry obsolete and you can measure off them exactly as you would a point cloud.”
Do it yourself
No tools needed — reason it through from the lesson.
- 1In one sentence each, contrast what laser scanning, photogrammetry and neural capture store about a scene.
- 2What is novel view synthesis, and why is it something classical point clouds cannot do?
- 3Explain why training a model to reproduce photographs optimises for appearance rather than for metric accuracy.
- 4Name three specific reasons a photorealistic neural capture can be metrically unreliable (e.g. scale, floaters, invented geometry).
- 5Give one task where a NeRF or splat is the right tool and one where it would be a dangerous mistake, and say why.
The one line to carry out
Peer-reviewed journals & authoritative standards
- 013D reconstruction — Wikipedia — 3D reconstruction, 2026.
- 02Novel view synthesis — Wikipedia — Novel view synthesis, 2026.
- 03Computer vision — Wikipedia — Computer vision, 2026.
- 04Machine learning — Wikipedia — Machine learning, 2026.
- 05Photogrammetry — Wikipedia — Photogrammetry, 2026.
We have framed the neural turn and its promise and peril in the abstract. Next we open the first technique and build a real intuition for how it works: a Neural Radiance Field that stores an entire scene as a continuous, learned field of colour and density.
The author
Amogh N P
Architect, interior designer, and creative polymath. Studio Matrx began in his notebooks — his vision of design made honest, useful, and open to everyone. Its Academy is written and taught in his memory, and free, forever.
More about Amogh →