Diffusion on the Viewport

A pure text-to-image model - type a sentence, get a picture - is wonderful for mood and hopeless for a specific building. Ask it for "a three-storey brick house with a corner window and a flat roof" ten times and you get ten different houses, none of them the one on your screen. There is no mechanism for it to know your massing, your window rhythm, your proportions. For ideation that freedom is a feature; for representing a design you have already made, it is a dealbreaker.

The fix is conditioning: giving the diffusion model a structural input alongside the text, so it generates within a shape you supply. The most useful structural input a designer already has is the viewport itself - and specifically the passes you can export from it. A depth pass encodes how far each pixel is from the camera; a line or edge pass encodes the building's contours; a segmentation pass labels regions (wall, glass, sky, floor). Any of these can steer the image so it keeps your geometry while the prompt supplies the look.

This is exactly what enhancement tools do internally - Veras and the rest are conditioned diffusion under a friendly UI. Doing it yourself trades that convenience for control: you choose the conditioning type, its strength, the model, the prompt and the seed. When faithfulness to your design is non-negotiable - a competition board, a planning image, a set that must all read as the same building - that control is worth the extra steps.

ControlNet is the technique (and family of models) that lets a structural pass steer Stable Diffusion. You give it a conditioning image derived from your viewport, and it constrains the diffusion process to respect that structure while the text prompt fills in materials, light and atmosphere. Different conditioning types suit different jobs, and choosing well is half the skill.

Depth is the workhorse for exteriors and interiors alike - it holds massing, spatial depth and rough form, and tolerates messy geometry. Canny traces every edge, giving tight control over detail but also locking in noise you may not want. MLSD detects straight line segments and is superb for buildings, where the important structure is orthogonal - clean facades, mullions, floor lines. Segmentation assigns each region a material class, which is powerful for interiors where you want "this is oak floor, that is plaster wall" respected. You can even stack two ControlNets - say depth plus MLSD - for both form and crisp edges.

Most 3D tools can produce these passes directly: Rhino, Blender, SketchUp and Revit can output a depth or clay render, and preprocessors inside ComfyUI or Automatic1111 will derive canny, MLSD or depth from a plain screenshot if needed. The conditioning strength works like the last lesson's slider in reverse: high strength keeps the AI tightly on your geometry, low lets it wander. For design fidelity you generally want it high, then relax it only if the image looks stiff. The prompt then does the rest - and because the shape is locked, you can iterate on materials and mood freely without the building morphing underneath you.

A practical viewport-to-diffusion pass has a clear shape. You will most likely run it in ComfyUI (node-based, powerful, the current standard for controlled work), Automatic1111/Forge (simpler UI), or Krita AI (paint-and-generate on a canvas, great for interiors). The steps are the same regardless of tool.

1. Frame the view in your 3D tool; lock the camera.
2. Export a conditioning pass: depth or clay render, or a line pass
   (or just a clean screenshot for a preprocessor to convert).
3. Load it into ControlNet; pick the type (depth / mlsd / canny / seg).
4. Set conditioning strength high (~0.7-1.0) to hold geometry.
5. Write the prompt: materials, time of day, mood, camera, quality.
6. Fix the seed once you like a result, so you can iterate cleanly.
7. Generate, judge against your model, adjust prompt or strength, repeat.

A strong prompt names materials and light precisely and leaves geometry to ControlNet:

"warm afternoon light, exposed concrete and teak, floor-to-ceiling
 glazing, polished terrazzo floor, soft contact shadows, shallow
 depth of field, architectural photography, high detail"

Notice you are not describing the building's shape - the depth pass already carries it. That division of labour is the whole point: geometry from your model, look from your words. Keep the loop tight - generate, compare to the viewport, refine - exactly the human-in-the-loop from Module 0. The setup cost is front-loaded; once your graph is built, re-shooting a view or restyling a scheme is fast.

Controlled diffusion is powerful but fiddly, and most of the skill after the basics is diagnosing why a frame went wrong. Learn to read the two common failures. If the AI is ignoring your geometry - inventing windows, drifting massing - your conditioning strength is too low or the wrong pass is fighting the prompt; raise strength, or switch to a pass that carries what matters (MLSD for straight lines, depth for form). If the image looks stiff, flat or plasticky - technically obedient but lifeless - conditioning is too high or the prompt is too thin; relax strength a little and enrich the prompt with light, materials and camera language. Nearly every problem is somewhere on that fidelity-versus-life axis, and tuning it is the loop.

A short menu of levers, roughly in order of impact:

geometry drifting?   -> raise ControlNet strength / better pass
stiff, lifeless?     -> lower strength a touch / richer prompt
wrong materials?     -> prompt is doing shape work; simplify to look
set won't agree?     -> fix seed, share prompt, add reference image
noisy / low detail?  -> higher-res base, then upscale (lesson 5.3)

The harder judgement is when to reach for controlled diffusion at all. It is more setup than an in-engine enhancer, so spend that cost only where it pays: images that must be provably faithful, a set that must stay consistent, or a look you cannot get from presets. For a quick mood test or an internal option study, the enhancers of the last lesson are faster and good enough. There is also a real cost dimension - local generation ties up a GPU, cloud generation costs per image, and iterating a large set adds up - so batch your work and settle the prompt and seed on one frame before generating the rest. Finally, build once and reuse: a well-made ComfyUI graph or a saved Automatic1111 configuration turns a fiddly setup into a repeatable studio tool, which is what makes controlled diffusion practical rather than a party trick. The mechanism is the same every time; only your model, prompt and seed change.

One variable worth naming explicitly is the base model itself. The checkpoint you diffuse with - a general model, a photoreal one, or an architecture-tuned community checkpoint - shapes the look as much as your prompt does, so try a couple and keep the ones that suit your practice. A lighter cousin of the full ControlNet route is plain img2img at low denoise, which treats your rendered frame directly as the starting image and nudges it toward realism; it holds geometry loosely and is quick for a soft polish, but for anything that must be faithful, an explicit conditioning pass beats hoping a low denoise keeps your lines. Choose the heaviness of the setup to match the fidelity the image actually demands.

A single beautiful frame is easy; a set that reads as the same building, at the same time of day, in the same material world, is the real professional test - and where naive diffusion falls apart. Change the view and, left alone, the AI will re-imagine the brick, shift the sky, and invent new trees. Several levers pull the set back together.

Fix the seed across views so the model's random starting point is shared - this alone steadies palette and mood. Reuse the exact prompt, changing only what genuinely differs. Use reference conditioning - IP-Adapter or a style/reference image - to carry a consistent look from a hero frame into the others. Keep conditioning strength high so each view is anchored to its own geometry rather than drifting. And generate at modest creativity, since every extra degree of freedom is another chance for the set to diverge.

Even then, expect to curate: generate several candidates per view and choose the ones that agree with each other, rather than accepting the first of each. For interiors especially, small inconsistencies - a rug that changes weave, a wall that shifts hue - are glaring across a deck, so audit the set together, not frame by frame. Perfect view-to-view consistency remains one of diffusion's genuine weak spots in 2026; tools are improving fast, but a human still has to hold the set to a single truth. That curation is the design work here - the AI proposes many plausible buildings, and you enforce that they are all the one you actually designed.