Lesson 3.1Lesson 3.1 · Control — Making AI Obey
Why Prompting Alone Fails Architecture
The moment words stop being enough - and control begins
You can write the perfect prompt and still never draw the same building twice.
By now you can make a diffusion model produce gorgeous, on-brief images. But try to use one on a real project and you hit a wall fast: you love the render, the client loves the render - now make the next view of the same building. Change the material but keep the massing. Match the plan you already drew. Pure prompting cannot do any of these, and no amount of prompt-craft will fix it, because the problem isn't your words - it's what words are. This lesson is the honest diagnosis, and it sets up the cure: control.
Words for the look, a picture for the layout. That split is the whole of control.
A prompt is a mood, not a measurement
A text prompt is a description of appearance - a bag of associations the model pulls the image toward. L-shaped courtyard house, west-facing, warm evening light names a feeling beautifully, but it does not specify a single wall position, a floor-plate, a viewpoint, or a dimension. Under the hood those words become a rough vector of meaning - a direction in the model's internal map - and the model wanders toward any image lying in that direction. There are millions of them, and it lands on a different one every run.
This is the core mismatch. Architecture is a discipline of exact geometry - this room is 3.6 metres wide, this sill sits 900 mm off the floor, this roof pitches at 22 degrees. Language is a discipline of approximate meaning. You cannot bridge that gap with more adjectives, because adjectives keep describing the mood, not the measurements. Say large window and you may get one 1.2 m wide or 3 m wide, centred or offset, square or arched - all equally 'large'. Say three bays and you may get three, or four that read as three, or a rhythm that only loosely counts. Words can steer the model into the right neighbourhood; they can never hand it an address. And this is not a limitation a bigger model will fix - it is the permanent gap between a fuzzy label and a precise coordinate, and no vocabulary ever closes it.
A prompt is a postcode, not a street address. It gets you the district, never the door.
The three things prompts can't pin
Play it out and the failure has a shape - three specific things words can never pin. Geometry: you can't hold a plan, an elevation or a massing steady - re-run the prompt and the building reshapes itself, because there is no stored plan to return to, only the prompt's rough direction. Viewpoint: you can't say 'the same house, now from the courtyard' - the model has no persistent 3D scene to move a camera around, so 'another angle' just generates an unrelated building that happens to match the words. It is not rotating an object; it is inventing a fresh image that also fits courtyard view. Consistency: you can't lock one thing while changing another - swap concrete for brick and the whole composition drifts, because you re-rolled the entire image from scratch rather than editing the one you had.
Each of these is fatal for real work. A design process is iterative and cumulative - you commit to a decision, then build the next one on top of it. You draw the plan, then the section that must match the plan, then the elevation that must match both. A tool that forgets everything the instant you change a word can decorate ideas, but it cannot develop them, because development means carrying a fixed decision forward while you vary what sits on top. That is the difference between a slot machine and a drawing board: the slot machine hands you a fresh, unconnected result every pull; the drawing board keeps yesterday's lines on the paper.
Why 'just prompt harder' is a trap
The tempting response is to fight drift with ever-longer prompts: pin the layout in words, over-specify every element, stack fifty descriptors. It feels like control. It isn't. Beyond a point, extra tokens dilute each other - the model averages a crowded prompt into mush, and the parts you care about most (the exact opening you drew) get no more weight than the parts you don't. Text encoders also have a hard budget - CLIP-based models stop reading at around 75 tokens - so past that budget the model simply ignores words, and you have no say over which it drops. The window you described in clause forty may never reach the image at all.
Worse still, the approach hides the real problem. The information you're trying to force through the narrow pipe of language - where things are - is inherently spatial. It wants to travel as an image, not a sentence. Describing a floor plan in prose is like emailing someone a song by typing the sheet music one note at a time: technically possible, absurd in practice, and lossy at every step. Even a flawless verbal description of a plan leaves the model free to interpret it a hundred ways, because it never learned to convert your words back into coordinates - it learned to convert them into a vibe. Adding words makes the vibe more specific; it never makes it geometric. The fix is not a better sentence. It is a second channel that speaks the model's other native language - pixels.
Describing a plan in words is humming a blueprint. Just show it the drawing.
The fix has a name: spatial conditioning
If where things go is spatial information, then give the model spatial information - as a picture. Alongside your text prompt you feed a second input, a conditioning image: an edge map, a depth map, a set of straight lines, a rough scribble, a segmentation of zones - or your own sketch or plan. The model is then constrained to honour that structure while the prompt handles everything the structure doesn't: materials, light, atmosphere, era. The two inputs divide the labour cleanly - one governs arrangement, the other governs appearance - and because they never compete for the same words, neither dilutes the other.
Picture the L-shaped courtyard house again. Feed a line drawing of its plan as the conditioning image and the courtyard stays exactly where you drew it in every generation; now red laterite, warm evening light changes only the dress, not the bones. Ask for a second view and you feed a second matching structure map - a section, or a depth pass of your massing - and get a genuine companion image of the same scheme. This is the whole premise of Module 3. Text tells the model what it looks like; a conditioning image tells it how it is arranged. Separate those two jobs and the slot machine becomes an instrument. Suddenly you can hold a layout dead still and change one material, or take a real second view by feeding a matching structure map - the exact tasks that were impossible with words alone. The rest of this module is the mechanism - ControlNet - and the craft of driving it from your own drawings. First, sit with the diagnosis: prompting didn't fail because you're bad at it. It failed because you asked language to carry geometry.
A worked example - one brief, four unrelated buildings
Make the failure concrete. Take a single, careful prompt: compact two-storey L-shaped courtyard house, west-facing entrance, deep verandah, sloping clay-tile roof, red laterite walls, warm evening light, architectural photograph. It is specific, well-ordered, and everything a prompt-craft guide would praise. Generate it four times.
You get four handsome houses - and no two share a plan. One puts the verandah on the wrong side; one loses the L and closes into a box; one pitches the roof the opposite way; one drops the courtyard entirely, because 'courtyard' merely tipped the odds, it did not command a void. Now try the three ordinary design moves. Hold the geometry: pick your favourite and reproduce its exact plan - even re-typing the words and fixing the seed will not rebuild that layout, because the seed pins the noise, not the arrangement. Change the viewpoint: add seen from the courtyard looking back at the entrance and you get a fifth, unrelated building. Hold one thing, change another: swap red laterite for grey basalt and the massing shifts along with the material.
Three moves, three failures - and each names precisely what words cannot carry: layout, viewpoint, and the ability to lock-and-vary. Keep this failure board, because in the coming lessons you will redo all three tasks with a conditioning image and win every one. The lesson to carry forward is diagnostic, not defeatist: you have not been prompting badly; you have been asking one channel to do two incompatible jobs.
Four handsome houses, four different plans. The words never named a building - only a genre.
Spatial conditioning
Feeding the model an image alongside the prompt to fix structure
The umbrella idea behind this whole module - a second, spatial channel that carries the geometry words cannot.
ControlNet
The mechanism that makes conditioning images steer a diffusion model
Introduced next lesson; the reason a sketch or plan can hold geometry while the prompt handles materials and light.
Seed
The random starting point of a generation
Fixing it reduces variation but still cannot pin layout - proof that reproducibility and spatial control are different problems.
Workshop - feel the wall for yourself
The fastest way to believe this lesson is to hit the limit with your own hands. Using any text-to-image tool you can access (Midjourney, a free Stable Diffusion space, Adobe Firefly, or Studio Matrx's DesignAI), try to do three ordinary design tasks with prompting alone and watch each one fail in a specific way.
Any text-to-image tool (Midjourney, a free Stable Diffusion web space, Adobe Firefly, or Studio Matrx DesignAI). No installation required for the diagnosis.
Goal: prove that prompting cannot hold geometry, viewpoint or consistency Inputs: one text-to-image tool + one building or room you can picture Time: ~30 minutes
- 1Write one clear prompt for a specific building, e.g.
a compact L-shaped courtyard house, west-facing, warm evening light, architectural photograph. Generate it four times and lay the results side by side. - 2Task 1 - HOLD GEOMETRY. Try to keep one of the four layouts and reproduce it by prompt alone. Note that even fixing the seed and re-typing the words won't reconstruct that exact plan.
- 3Task 2 - CHANGE VIEWPOINT. Add
seen from inside the courtyard looking backand generate. Observe that you get a different building, not another view of the same one - there is no persistent 3D scene. - 4Task 3 - HOLD ONE THING, CHANGE ANOTHER. Take your favourite result's prompt and swap only the material, e.g.
concretetored brick. Note how the whole composition drifts, not just the surface. - 5Write one sentence under each failure naming exactly what prompting could not pin. Keep this board - in the next lessons you'll redo all three tasks with ControlNet and win.
You’ll walk away with
A three-panel 'failure board' - hold-geometry, change-viewpoint, hold-one-thing - each annotated with the specific limit you hit, ready to be beaten with conditioning later in the module.
Three altitudes on the same idea
Read the band that fits you — or all three.
This is the lesson that turns AI from a toy into a tool for you. Ideation prompts are fine for a mood wall, but the instant a scheme is real - a plan committed, a massing agreed - you need renders that obey that geometry and stay consistent across views. Recognising that prompts structurally cannot do this is what stops you wasting hours re-rolling and pushes you to conditioning, where your drawing becomes the brief the model must respect. Make the diagnosis a working reflex: the moment a scheme is committed, stop re-rolling prompts and reach for a conditioning image, because no sentence will ever reconstruct the plan on your screen.
You feel this pain as soon as a client says 'same room, different sofa'. Prompt-only tools re-invent the whole space every time, so you can never hold a layout and swap one element. Understanding that words can't pin the arrangement tells you why - and points you at conditioning and inpainting, where you keep the room you have and change only what you choose. That is the difference between showing options of the same scheme and showing four unrelated rooms. Once you can name why - words pin appearance, not arrangement - you stop fighting the tool and start using the right one: condition to hold the room, inpaint to change the piece.
Naming this limit is a portfolio-level insight. Anyone can prompt; the skill employers notice is knowing when prompting stops working and reaching for control. If you can articulate why text alone can't hold geometry, viewpoint or consistency - and then demonstrate a conditioned generation that does - you've shown design judgement about the tool, not just fluency with it. Learn the diagnosis here; you'll earn the cure across the rest of this module. Practise saying it in one clean line at a review: prompting is for exploring, conditioning is for developing, and knowing which job you are doing is the judgement employers are actually testing for.
“If my renders keep changing, I just need to write a more detailed prompt.”
Do it yourself
No tool needed - reason it through.
- 1In one sentence, why can a prompt name a mood but not a plan?
- 2Name the three things pure prompting structurally cannot pin.
- 3Why does 'the same house from another angle' fail with prompting alone?
- 4Explain why a longer, more detailed prompt eventually makes control worse, not better.
- 5What kind of input carries the geometry that words cannot - and what is it called?
The one line to carry out
Peer-reviewed journals & authoritative standards
- 01Zhang, L., Rao, A., & Agrawala, M. - Adding Conditional Control to Text-to-Image Diffusion Models — IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
- 02Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. - High-Resolution Image Synthesis with Latent Diffusion Models — IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
- 03Ye, H., et al. - IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models — arXiv preprint, 2023.
So we need to hand the model a picture of the structure we want kept. The mechanism that lets a conditioning image steer a diffusion model - without retraining it - is ControlNet, and that is exactly where we go next.
The author
Amogh N P
Architect, interior designer, and creative polymath. Studio Matrx began in his notebooks — his vision of design made honest, useful, and open to everyone. Its Academy is written and taught in his memory, and free, forever.
More about Amogh →