Sketch-to-Image & ControlNet

In the last lesson the AI decided the form and you reacted. That is perfect for opening the solution space, but the opposite of what you want once you have committed to a composition. When you have a massing model, a plan, or a considered perspective sketch, you do not want the AI to reinvent it - you want it to render it. The problem is that a text prompt alone gives the model no idea of your geometry, so it will cheerfully produce a different, prettier building every time.

Image-to-image (img2img) solves half of this. Instead of starting from pure noise, the model starts from your image and partially denoises it toward your prompt. A single dial - variously called denoise strength or image weight - controls how far it drifts. Low strength (say 0.2-0.35) barely changes your input, just restyling surfaces; high strength (0.7+) keeps only a loose echo of your composition and invents the rest. Learning to feel that dial is most of img2img.

The honest limit: img2img respects colour and rough layout, but under a strong prompt it will still bend edges, move openings and hallucinate detail. For tight control - where a roofline or a column grid must survive exactly - you need something that reads structure, not just pixels. That is ControlNet.

It is worth being clear about why you would want constraint at all, because it feels like giving up the AI's magic. The answer is ownership. A text-to-image concept, however beautiful, is a suggestion the model made; a constrained render is your design, seen in a new light. As a project matures, the second is almost always what you need - to show a client the scheme you agreed, to test materials on the massing you resolved, to keep a competition entry recognisably one building across ten images. Constraint is not a limitation on creativity; it is what lets you apply the AI's speed and beauty to a decision you have already earned, instead of endlessly re-opening it.

ControlNet is an add-on to Stable Diffusion that conditions the image on a structural map extracted from your input, not just its colours. It comes in flavours, each reading a different kind of structure. Canny and Scribble follow edges and lines - ideal for a hand sketch or a line export from CAD. Depth reads near-versus-far - perfect for a grey massing model or a SketchUp view, so the AI understands the volumes. MLSD snaps to straight lines, which keeps architectural geometry crisp. Segmentation maps regions (wall, glass, sky, ground) so you can say what each area is.

The workflow is straightforward in ComfyUI or Automatic1111: load your sketch or massing render, pick a ControlNet type, and it produces a control map the model must honour while your text prompt supplies material, light and mood. Because the geometry is locked to your input, you can now run ten variations - warm timber at dusk, white render at noon, weathered concrete under cloud - and every one keeps your building.

Input:   grey SketchUp massing, eye-level view
Control: Depth (reads the volumes)
Prompt:  rammed-earth facade, deep window reveals, warm
         late-afternoon light, architectural photograph,
         no people, no text
Result:  your massing, now rendered - same form, new skin

The key mental shift: ControlNet makes the AI a renderer of your design, not a generator of its own. The composition is yours; the AI only dresses it.

A few practical notes save frustration. The quality of the control map matters: a clean line drawing or a clear grey massing with readable depth gives the model something firm to hold, while a muddy, cluttered input produces muddy, uncertain output. You can dial each ControlNet's strength or weight - how strictly the model must obey the map - so a lower weight lets the prompt breathe while a higher one clamps the geometry hard. And you can be selective about which flavour to trust: Depth is forgiving and great for volumes but vague about edges; Canny is precise about lines but ignores depth. Choosing the map that matches what you most need to protect - the silhouette, the volumes, the window grid - is half the craft of getting a faithful render out of your own sketch.

There is a spectrum from loose to tight, and the right point depends on how much of the form you have already decided. Pure text-to-image is loosest - the AI owns the form; use it to diverge when you have nothing committed. img2img at low strength keeps your rough layout and colour, good for quickly reskinning a concept. img2img at high strength hews close to your input. ControlNet is tightest, pinning edges or depth exactly - use it when the geometry is sacred.

A practical rule: the more of the design you have committed to, the tighter you should go. Early, go loose and let the model surprise you. Once you have a massing you believe in, go to ControlNet-Depth so exploration cannot quietly erode the decision you made. Sitting at the wrong point wastes effort - loose control on a committed design gives you pretty images of the wrong building; tight control too early kills the surprise you were hoping for.

You can also stack constraints. Combining a ControlNet map with a modest img2img strength, or two ControlNets (Depth for volume plus Canny for edges), gives fine-grained hold - though every added constraint narrows variety, so add them only as far as the design demands. As always, more scrutiny where the stakes are higher: a mood study can drift; a client-facing image of an agreed scheme should keep the scheme.

There is a natural progression across a project's life that maps onto this dial. In week one you are at the loose end, letting text-to-image throw forms at you. By schematic design you have a massing, so you slide toward img2img and light ControlNet to reskin it. By the time you are presenting a resolved scheme you are at the tight end, ControlNet-Depth locking the building while you explore only material and light. Reading where a project sits and picking the matching control level is a judgement in itself - and getting it wrong is one of the commonest ways designers waste hours with these tools, generating gorgeous images of a building nobody asked for.

This technique sits in the fertile middle of a project: past pure ideation, before real visualization. It shines at taking a rough SketchUp or Rhino massing and making it legible and atmospheric enough to discuss - with a client, a team, a jury - without the cost of a full render. It is also a fast way to test material and light options against a fixed form: same geometry, ten skins, in the time a single V-Ray render would take.

But keep the boundaries honest. ControlNet holds your composition, not your dimensions - it will not keep a window at exactly 1200mm or guarantee a buildable detail, and it still invents texture and small features. The output is a persuasive concept image, not a construction document and not a substitute for a measured drawing. Where accuracy is contractual - a planning submission, a heritage context - a diffusion image is evidence of intent, not fact.

And the authorship principle from the last lesson still holds, now with a stronger foundation: because the geometry started as yours, the result is far more clearly your design than a text-to-image guess ever could be. That is the quiet reason this workflow matters - it lets you use the speed and beauty of diffusion while the building on the screen remains the one you drew. Module 5 picks this up for finished visualization; here it stays a concept tool.

One workflow habit ties it all together: keep the source of truth in your CAD or BIM model and treat every diffusion image as a disposable view off that model. When the design changes, you change the model and re-export the control input, rather than trying to edit the AI image. That discipline stops the common trap of a beautiful render quietly becoming the design of record - a picture nobody can dimension, coordinate, or build from. Used this way, sketch-to-image is a fast, honest window onto a design that lives properly elsewhere, and that is exactly the role it should keep.