Lesson 8.3Lesson 8.3 · Beyond Images
Text-to-3D & Gaussian Splats
Inventing objects from a prompt, and capturing real space into a navigable scene - the early state, honestly
One AI dreams a chair. Another captures your whole site from a phone video.
Two very different breakthroughs both live under '3D'. Text-to-3D invents a rough object from a sentence, the way Module 0's diffusion invents an image. NeRF and 3D Gaussian splatting do the opposite - they take photos of a real space and reconstruct it into a photoreal scene you can fly through from angles you never shot. Both are thrilling; one is far more useful to you today; and both stop short of a model you can actually build from.
Is the AI inventing or recording? That question decides which tool you need.
Two families under one word
The first thing to get straight is that '3D AI' names two opposite operations, and confusing them causes most of the disappointment people report.
Generation starts from nothing but a prompt. Text-to-3D - the DreamFusion line of work is the landmark - asks a 2D image diffusion model (Module 0) to hallucinate an object consistently from every angle, and distils those views into a rough 3D form. You type 'a mid-century lounge chair' and get a blobby, dreamlike 3D object. It is the same magic as text-to-image, one dimension up, and at a much earlier, rougher stage.
Capture starts from reality. You photograph or video a space that exists - a room, a facade, a physical study model - and the AI reconstructs it into a 3D scene you can revisit from new viewpoints. NeRF (Neural Radiance Fields) was the 2020 breakthrough that made this photoreal; 3D Gaussian splatting was the 2023 leap that made it fast enough to fly through in real time.
Hold the distinction with a simple test: is the AI inventing, or is it recording? Generation invents (great for props and quick blocks, weak on fidelity and control); capture records (astonishing fidelity of real places, but limited to what exists in front of the lens). For architecture and interiors today, capture is the far more valuable of the two - because our work so often begins with a real site, a real building to renovate, or a real physical model, and turning that into a navigable digital scene is immediately, practically useful in a way that dreaming a wobbly chair is not yet.
Generation invents. Capture records. Ask which one you actually need.
NeRF, splats and meshes - three ways to hold a scene
Because capture is the useful half, it's worth understanding the three ways a scene can be represented - they behave very differently.
A NeRF stores the scene implicitly, as a small neural network you query: 'at this point in space, looking this direction, what colour and how solid?' Ask it enough times along each ray and you render an image. NeRFs are gorgeous - they handle glass, reflections and soft haze that defeat older methods - but training and rendering are slow, and the scene is locked inside a network you can't easily edit.
3D Gaussian splatting stores the scene explicitly, as literally millions of tiny fuzzy 3D blobs (Gaussians), each with a position, colour, size and transparency. To render, the machine 'splats' these blobs onto your view - and because that is cheap, you get photoreal quality and real-time navigation, which is why splatting largely displaced NeRF for practical capture almost overnight. The cost is heavy files and a representation that is a cloud of blobs, not clean geometry.
A polygon mesh - triangles and faces - is what CAD, BIM and game engines actually want: editable, measurable, snappable, the native language of your design tools. Here is the rub that governs everything: a splat or a NeRF is not a mesh. Converting one into a clean, usable mesh is possible but lossy and often messy, and this capture-to-mesh gap is the single biggest practical limit of the whole field right now. You can fly through a stunning splat of a heritage facade; getting that facade into Revit as tidy geometry you can dimension and design against is a different, harder, still-maturing step.
NeRF hides it in a network. Splat is a cloud of blobs. A mesh is what you can edit.
What this is genuinely good for now
Strip away the hype and a clear set of real, today uses remains - almost all of them on the capture side.
Site and existing-condition capture. Walk a site or an interior with your phone, shoot a careful video, and get a navigable 3D record you can revisit, measure roughly, and share with a remote team - a live, spatial alternative to a folder of disconnected photos. For renovation and adaptive reuse, capturing what is there is often the hardest, most valuable step, and splatting makes it near-effortless.
Heritage and documentation. Photoreal capture of a facade, a courtyard, a carved detail, preserved as a scene future work can reference. Physical-model digitisation. Built a chipboard massing model? Capture it and share a fly-through with a client across the country. Immersive presentation. A splat scene navigated live is a more visceral way to walk a client through a real place than any slideshow - feeling matters, and splats deliver feeling.
On the generation side, be soberer. Text-to-3D today is useful mainly for rough concept props and quick massing blocks - a placeholder sofa to test a layout, a loose form to think with - not for anything requiring fidelity, dimension or control. The objects are blobby, the topology is unusable for real modelling, and you cannot yet dictate precise proportions. It is genuinely at the stage text-to-image was a few years ago: astonishing that it works at all, not yet a production tool. Match your expectation to that reality and you will be delighted rather than let down. (Studio Matrx's DesignAI focuses on the reliable, controllable end of this spectrum for exactly this reason.)
It helps to say plainly why capture beats generation for our field, because it is not a temporary accident. Architecture is overwhelmingly a discipline of the specific - this site, this existing wall, this client's actual room - and capture is built to reproduce the specific with fidelity. Generation, by contrast, produces the generic: a plausible chair, an approximate room, an object no one measured. When your work begins from a real thing, a faithful record of that real thing is worth more than any number of invented approximations - and that ordering is likely to hold even as both halves improve, because it reflects what the two operations are for, not merely how good they are this year.
The honest ceiling, and how to use it well
This field is moving faster than any other in the course, so state the limits in a way that will age well rather than quoting this month's numbers.
Capture reproduces appearance, not understanding. A splat of a room is a beautiful photographic shell; it does not know where the wall plane truly is, what is structure versus finish, or any dimension to survey accuracy without care. It is superb for seeing and rough measuring, not yet a substitute for a measured survey. The mesh gap persists. Until splat-to-clean-mesh is solved, captured scenes live slightly to the side of your BIM workflow rather than inside it - a reference you design against, not geometry you design with. Generation lacks control and fidelity. Text-to-3D can't yet give you a specific object at a specific size with clean topology, which is most of what design modelling needs. And capture needs real inputs - good coverage, decent light, patience; a hurried phone sweep yields a holey, smeary scene.
The way to use all this well is the same discipline the whole module has drilled: match the tool to the job and stay honest about which half you are in. Reach for capture when you have a real space to record and want to see or present it - and treat the result as a rich reference, not a survey. Reach for generation only for rough, throwaway props where fidelity doesn't matter. And keep watching, because of everything in this course this is the corner most likely to leap forward between now and when you next open the tools. The skill that will still be true then is not knowing today's best app; it is knowing the difference between recording and inventing, and between a photoreal shell and a buildable model - the judgement, once again, is the durable part.
A photoreal shell is not a measured survey. Design against it, not with it - yet.
3D Gaussian splatting
Explicit scene capture as millions of fuzzy 3D blobs; photoreal and real-time
The practical capture method today - fast to render and view; heavy files, and not a clean mesh you can edit.
NeRF (Neural Radiance Fields)
Implicit scene capture inside a small neural network
Gorgeous with glass and haze; slow to train and render and hard to edit - largely superseded by splatting for practical use.
Text-to-3D (DreamFusion line)
Generating a rough 3D object from a text prompt via a 2D diffusion prior
Early-stage: good for concept props and massing blocks, weak on fidelity, control and usable topology.
The capture-to-mesh gap
Converting a splat or NeRF into clean, editable, measurable geometry
The field's biggest practical limit for architects: lossy and immature, so captures stay reference, not BIM geometry.
Workshop - capture a real space and meet the mesh gap
You'll do the useful half - capture - and run straight into the honest limit, so both the promise and the ceiling become concrete. A phone and a free Gaussian-splatting app are enough.
A smartphone and a free Gaussian-splatting or NeRF capture app; optionally any text-to-3D tool for the comparison. No paid software required.
Goal: feel why capture beats generation today, and where the mesh gap bites Inputs: a phone + a free splatting/NeRF capture app + a small real object or room Time: ~45 minutes
- 1Capture well: pick a small physical model or a corner of a room, and shoot a slow, overlapping video from many heights and angles in even light. Process it into a splat or NeRF.
- 2Fly through: navigate the result to viewpoints you never actually shot. Note where it is convincing (surfaces, light) and where it breaks (holes, smears, thin edges).
- 3Test the mesh gap: try to export or convert the capture to a mesh, or drop it toward a CAD/BIM tool. Record honestly how clean or messy the geometry is.
- 4Contrast with generation: in any text-to-3D tool, prompt the same object (e.g.
a wooden stool). Compare fidelity, control and usable-ness against your capture. - 5Write the verdict: one paragraph on which method you'd reach for on a real project and why, and one line on what would have to improve before a capture goes straight into BIM.
You’ll walk away with
A short write-up with your captured fly-through (screen recording or stills), an honest note on the mesh-export result, a side-by-side against a text-to-3D object, and your verdict on when to use capture versus generation.
Three altitudes on the same idea
Read the band that fits you — or all three.
Capture is the half that pays off now: a careful phone video of a site or an existing building becomes a navigable 3D record for feasibility, renovation and remote review - immediately useful, near-effortless with Gaussian splatting. Treat it as a rich reference, not a measured survey, and remember the mesh gap: getting a splat into BIM as clean, dimensionable geometry is still lossy and maturing, so you design against the capture, not yet with it. Keep text-to-3D for rough concept props only, and watch this space - it is the fastest-moving corner of the field.
For interiors, splat capture is a quietly brilliant site-record tool: walk a client's existing room, capture it, and revisit or present the real space from anywhere without a second visit. It reads materials, light and proportion far better than a handful of photos. Text-to-3D can spin up a rough prop - a placeholder chair or lamp to test a layout - but the objects are blobby and unmeasured, so never treat one as a real product model. Capture the real; sketch with the generated; buy and specify from neither.
This is the most exciting corner to experiment in, and you can start free with a phone. Capture a physical study model or a corner of your studio with a splatting app and fly through it; you will learn more about how these representations behave in an afternoon than from any explainer. Understand why a splat isn't a mesh, why capture beats generation for real work today, and how NeRF and splatting differ - that conceptual clarity, not fluency with one app, is what will still be valuable when the tools have moved on twice.
“Text-to-3D and Gaussian splatting can generate finished, buildable 3D models I can drop straight into BIM or CAD and design with.”
Do it yourself
No tool needed - reason it through.
- 1What is the core difference between 3D generation and 3D capture?
- 2Why did 3D Gaussian splatting largely replace NeRF for practical capture?
- 3What is a mesh, and why does the 'capture-to-mesh gap' matter to an architect?
- 4For a renovation of an existing building, which method helps most today, and why?
- 5Why should you treat a photoreal splat as a reference rather than a measured survey?
The one line to carry out
Peer-reviewed journals & authoritative standards
- 01Kerbl, B., Kopanas, G., Leimkuehler, T., & Drettakis, G. - 3D Gaussian Splatting for Real-Time Radiance Field Rendering — ACM Transactions on Graphics (SIGGRAPH), 2023.
- 02Mildenhall, B., Srinivasan, P. P., Tancik, M., et al. - NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis — European Conference on Computer Vision (ECCV), 2020.
- 03Poole, B., Jain, A., Barron, J. T., & Mildenhall, B. - DreamFusion: Text-to-3D using 2D Diffusion — International Conference on Learning Representations (ICLR), 2022.
- 04Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. - High-Resolution Image Synthesis with Latent Diffusion Models — IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Capture and generation gave you space in 3D. The last frontier of the module adds time: AI video and walkthroughs that promise motion through a scene. The same honesty applies - where they help now, and where coherence still breaks.
The author
Amogh N P
Architect, interior designer, and creative polymath. Studio Matrx began in his notebooks — his vision of design made honest, useful, and open to everyone. Its Academy is written and taught in his memory, and free, forever.
More about Amogh →