ControlNet Explained

A diffusion model is enormous and expensive to train - hundreds of GPU-days on a dataset you will never assemble - so you do not want to disturb it. ControlNet's clever move is to leave the original model completely frozen and bolt on a small, trainable copy of its encoder blocks that takes one extra input: your conditioning image. During its own training that copy learned the relationship between a condition - say, a Canny edge map - and the millions of images those edges came from. At generation time it feeds that learned structure back into the frozen model at every denoising step, biasing each step toward images whose edges match your map.

The copy connects to the base through zero-convolution layers - links that start at literally zero influence and grow only as training teaches them to help. That single design choice is why adding ControlNet can never break the base model: on step one it contributes nothing, so at worst you get the original model back; from there it can only guide. It also means one base model can host a whole rack of ControlNets - one for edges, one for depth, one for pose, one for scribble - each a separate, swappable adapter file of a few hundred megabytes. You keep every gram of the base model's quality and knowledge and simply clip on whichever steering wheel the job needs. Nothing is retrained, nothing is lost, and you can stack or remove controls at will.

ControlNet doesn't consume your raw photo or sketch directly - it consumes a condition map, a stripped-down abstraction of the one kind of structure you want kept. A preprocessor (sometimes called an annotator or detector) makes that map. Feed it a source image and it extracts a single property - the edges, the depth, the straight lines - and throws away everything else, handing ControlNet a clean, unambiguous target.

The point of the abstraction is leverage. By reducing your image to just its edges, or just its depth, you tell the model exactly which properties are non-negotiable and which it is free to reinvent. A photograph of a room carries a thousand entangled facts - colour, texture, light, furniture, geometry - and if you fed all of them the model would have no room to design. Strip it to a depth map and you have said, in effect, 'keep the arrangement of near and far, forget the rest'; strip it to Canny edges and you have said 'keep every outline, restyle every surface'. Each preprocessor also exposes an adjustable pre-step - Canny has low and high thresholds that decide how many edges survive, depth offers a choice of estimator (the older MiDaS or newer, cleaner models) - so you are tuning the condition before it ever reaches the model. Choosing and dialling a preprocessor is choosing your grip on the image, and that choice matters more than any single prompt word, because it decides what survives the generation and what the model gets to imagine.

Five earn their keep for buildings and interiors. Canny detects all edges - a strict tracing that holds composition tightly; ideal for locking a facade rhythm or a furniture layout where every outline matters. Its low/high thresholds let you keep only the strong edges (clean, forgiving) or every faint one (exact, but it will trace clutter too). Depth estimates how near or far each pixel sits - it keeps 3D massing and spatial arrangement while freeing surfaces, the natural choice for restyling a room without moving its walls or reskinning a massing model. MLSD finds only long straight lines - purpose-built for architecture's orthogonal world of walls, floors, mullions and openings, and blissfully blind to plants, people and mess. Scribble accepts loose, gestural lines - the softest grip, perfect for turning a napkin sketch into a render without over-constraining a drawing you haven't finished deciding. Segmentation paints the image into labelled zones by colour - this block is 'building', this is 'sky', this is 'road', this is 'vegetation' - so you control what goes where at the level of whole regions rather than lines.

Rule of thumb: the more precisely a preprocessor traces your source, the more faithfully the output obeys - and the less creative freedom the model keeps. Canny is a tight leash, MLSD a tidy one, depth a structural one, scribble a long one, segmentation a zonal one. Match the leash to how sure you are of the geometry. If the drawing is final, trace it hard; if it is still a guess, hold it loosely and let the model help you think.

At generation, the model denoises as always - but now, at every step, the ControlNet nudges it to keep matching your condition map. The prompt still does its full job on everything the map leaves open: materials, palette, light, weather, era, style. So MLSD lines of my elevation + red sandstone, monsoon sky, dusk gives you your facade, rendered in that material and mood.

Two dials govern the grip. Control weight - typically 0 to 2, with most work living between 0.8 and 1.3 - sets how strongly the condition is enforced. High means the output traces your map almost exactly; low lets the model drift toward a prettier but looser result. Guidance start and end (each a fraction of the run from 0 to 1) let the condition apply only during part of the denoising - a common trick is to hold structure hard for the first two-thirds of the steps, then release it (end around 0.7) so the final steps are free to bloom fine detail the rigid map would otherwise flatten. Learning to trade these against the prompt is the craft: too much control - weight at 1.5 across the whole run - and you get a flat colouring-in of your lines; too little and the geometry slips. The sweet spot keeps your drawing honest while letting the model do what it is brilliant at - surfaces, light and life.

This division of labour is what makes ControlNet feel like a real design tool rather than a lottery. The prompt and the condition are not rivals; they are a two-handed instrument. Your left hand (the condition) sets the structure that must not move; your right hand (the prompt) plays the material, the hour, the weather, the mood. Change the prompt and the same building re-materialises in sandstone at dusk, then in exposed concrete under monsoon cloud, then as a pencil study - each time with identical geometry, because the condition never let go. That is the promise of this whole module made concrete: not a different building every run, but the same building, endlessly re-dressed at your command.

Even with the right preprocessor, three failures recur - and each has a clean fix. The trace looks flat and lifeless. Your control weight is too high, or you held it across the whole run; drop it toward 1.0 and set the guidance end to about 0.7 so the last steps can add depth and texture. The geometry slips despite a strong weight. The condition map is weak - a faint pencil sketch gave Canny too few edges, or MLSD missed thin lines. Fix it upstream: darken and clean the source, raise the edge sensitivity, or switch to a stricter preprocessor before you touch the weight. The model ignores the prompt's material. Two tight controls (say Canny and depth both at 1.3) have left it no freedom; lower one, or shorten its guidance range, so the prompt can breathe.

A reliable starting recipe for a first architectural pass: one control - MLSD for orthogonal drawings, depth for massing, Canny for clean linework - at weight about 1.0, guidance 0 to 0.8, a base model suited to the look, and a prompt about surfaces and light only. Generate, judge against the source, then adjust one variable at a time - never three at once, or you will not know which move helped. Stacking a second ControlNet earns its keep when one property alone cannot hold the scheme - depth to keep volumes and MLSD to keep crisp edges - but give each a slightly lower weight so together they do not strangle the render. Control is a budget: spend it where the geometry must not move, and leave the rest for the model to design.