From Images to 3D (SfM & MVS)

Everything downstream is built on one humble operation: finding features and matching them across photos. A feature (or keypoint) is a small, distinctive, repeatable patch of an image that the software can reliably recognise again from a different angle or distance - a corner of a window, a knot in timber, a speck of lichen, the junction of two mortar lines. The software scans each photo and detects thousands of these, then computes for each a compact numerical descriptor: a little fingerprint of the patch's local appearance, designed to stay roughly the same even when the patch is seen larger, smaller or rotated.

With every photo reduced to a set of fingerprinted features, the software plays a giant matching game: for each pair of overlapping photos, it looks for features whose descriptors are nearly identical and declares them correspondences - the same real point seen in two images. These matches are the raw evidence from which all the geometry is later solved. Because honest mistakes are inevitable (two different windows can look alike), the software filters matches with geometric consistency tests, throwing out pairs that cannot be explained by any sensible camera arrangement, keeping the reliable ones.

This single step explains, mechanically, the capture rules from Lessons 2.1 and 2.2. Features need texture and detail, which is exactly why blank, smooth surfaces fail - there are no distinctive patches to detect. Matching needs the same point to *look the same* from different photos, which is why consistent, diffuse light helps and why shiny, reflective and transparent surfaces break it - their appearance changes with viewpoint, so descriptors do not match. Matching needs features to be sharp, which is why blur is so destructive - a smeared patch has no reliable fingerprint. And matching needs photos to overlap, because correspondences only exist between images that see the same thing. When people say a capture 'had nothing to work with', this is the layer they mean: too few good, matchable features. Everything the pipeline later achieves is only as good as the correspondences it starts from, so rich, sharp, overlapping, evenly-lit photos are not an aesthetic preference - they are the fuel this first stage runs on.

Now the detective work. Structure-from-Motion takes those correspondences and solves, simultaneously, for two unknowns: the structure (the 3D positions of the matched points) and the motion (where each camera was and how it was oriented - its *pose*). The insight that makes this possible is the same triangulation from Lesson 2.1, run in reverse and at scale: if the same set of points is seen across many photos, there is essentially only one arrangement of cameras and points in space that is consistent with all those 2D observations at once. The software finds it, typically by starting from a couple of well-matched images, triangulating an initial set of points, then incrementally adding more photos - each new photo's pose fixed by the points it shares with those already solved, and each new photo in turn adding fresh points. Step by step, a cloud of cameras and a sparse point cloud of tie points grow together out of nothing but the photos.

This sparse cloud is not the final model - it is a skeleton, a few thousand to a few hundred thousand confident points that prove the geometry hangs together and, above all, pin down exactly where every camera was. That camera solution is the real prize of SfM, because the dense reconstruction to come depends entirely on knowing the poses precisely.

Errors accumulate as photos are chained, though, so the pipeline periodically runs bundle adjustment: a big simultaneous optimisation that nudges *all* the camera poses and *all* the 3D points together to best fit *all* the observed correspondences at once, minimising the total reprojection error (how far each 3D point, projected back into each photo, lands from where it was actually seen). Think of it as tightening every joint of the whole structure at the same time so stresses even out rather than piling up at the end. Bundle adjustment is what keeps a large capture from drifting and bending, and it is also where weak geometry shows its teeth: if your photos were a single thin line with no cross-links, or covered a surface from only one direction, the solution is poorly constrained and can warp - the mathematical echo of the capture advice to shoot from many well-spread angles. Good overlap and varied viewpoints do not just help matching; they give bundle adjustment the redundancy it needs to find a stable, accurate answer.

With the cameras pinned down, the pipeline switches from 'where were the cameras?' to 'what exactly is the surface?' - and that is Multi-View Stereo. MVS takes the known camera poses and goes back to the full-resolution images to reconstruct a dense surface: for effectively every pixel (not just the sparse features), it searches the overlapping photos for the matching point and triangulates its depth, exploiting the fact that once you know where two cameras are, a point's position along one camera's ray can be pinned by finding its match in the other. Done across all the images, this produces a dense point cloud of millions - sometimes billions - of coloured points, a genuinely complete description of the visible surface rather than a skeleton. This is the heavy, slow stage: MVS is where the hours and the powerful hardware go, because it is solving a depth for a vast number of pixels.

From the dense cloud the pipeline builds the deliverables. Meshing skins a continuous surface - a polygon mesh of triangles - over the cloud, turning a set of discrete points into solid geometry you can section, measure and model against; this step also makes choices (smoothing, filling small gaps, deciding what is surface and what is noise) that you should understand, because an over-smoothed or over-filled mesh can quietly invent geometry that was never measured. Finally, texturing projects the original photographs back onto the mesh, so the model wears the real colours and surface detail of the subject - the photorealism that makes photogrammetry so compelling for heritage and visualisation. The same original images thus serve three times: as features, as dense-matching data, and as texture.

Be honest about what all this does and does not guarantee. The output can look stunning and still be metrically imperfect: gaps where surfaces were occluded or featureless, noise on difficult materials, subtle warping where geometry was weak, and invented detail where the mesh filled holes - and, as ever, scale that is only as good as the references you provided. Automation has genuinely improved, and the software increasingly hides these steps behind a single button, but the steps are still there and still fail in the same old ways. Knowing the pipeline is what lets you read a result critically: ask where the cloud is dense and trustworthy versus sparse and guessed, whether the geometry was well-constrained, and whether the scale was properly set and verified - and defer any survey-grade or binding accuracy to a licensed surveyor rather than the software's quiet confidence.

Understanding the chain turns you from a button-pusher into someone who can reason about a result. Most software reports diagnostics along the way, and learning to read them is the skill: how many images aligned (and which were dropped, and why); the reprojection error after bundle adjustment (a rough health check on the geometric fit); the density and coverage of the dense cloud; and where confidence is low. A capture where half the photos failed to align, or where whole regions are sparse, is telling you plainly that matching or coverage broke down there - and that those parts of the model are guesses, not measurements.

This is also where you keep the method honest about automation. Modern tools, and increasingly AI-assisted ones, compress the whole pipeline into an almost hands-free experience, which is wonderful for speed and accessibility - but it does not repeal the underlying physics and geometry. The software will cheerfully produce *a* model from poor input; it simply produces a bad one, and an automated, confident-looking bad model is more dangerous than an obviously broken one, because it invites misplaced trust. The pipeline also quietly makes interpretive choices - which matches to keep, how much to smooth, how to fill gaps - that affect the geometry, so two tools, or two settings, can give measurably different results from the same photos.

Hold on to the honest limits that run through the whole course. Photogrammetry output is a measured approximation, not ground truth: it carries real and variable error, it is incomplete wherever surfaces were occluded or unmatchable, its accuracy depends on the whole chain from capture to processing, and its scale is only as trustworthy as the references and verification you supplied. None of that makes it unreliable - it makes it an instrument to be understood, specified and checked. Read the diagnostics, verify a few real dimensions, be sceptical of suspiciously smooth or suspiciously complete surfaces, and know the professional boundary: survey-grade accuracy, georeferencing and any legally or structurally binding deliverable belong to a licensed surveyor and verified equipment specifications, under the governing standards - not to the green tick at the end of a processing run. The reward for opening the black box is exactly this: the confidence to use photogrammetry's results well, and the wisdom to distrust them where they deserve it.