Hallucination, Bias & Accuracy

A large language model is, at its core, a next-token predictor. Given the text so far, it produces the most probable continuation, learned from a vast corpus of human writing. It has no separate store of verified facts and no internal check for truth - it has a model of what language usually looks like. Most of the time, the most probable continuation is also correct, because true statements are common in its training data. But when the true answer is rare, absent, or contested, the model still produces its most probable-sounding guess, delivered with the same fluency. That guess is a hallucination: confident, coherent, and wrong.

This is why hallucinations cluster in predictable places. They spike around specific facts the model half-remembers - exact clause numbers, statistics, dates, citations - because the model reconstructs rather than retrieves, and a plausible-looking clause number is easy to fabricate. They spike at the edges of the training data: niche topics, recent events after the training cut-off, and anything the corpus covered thinly. And they spike when you push the model to answer something it does not know, because it is built to continue text, not to say 'I have no idea.' Knowing this, you can predict your own risk: the more specific, recent, or obscure the claim, the harder you check.

There is a deeper point worth absorbing: the model has no representation of its own uncertainty that reliably reaches you. A human expert signals doubt - they hedge, pause, say 'I think' or 'check this.' A model can be prompted to express confidence levels, but its stated confidence is itself generated text and correlates only loosely with whether it is right; it can be serenely certain about a fabrication and needlessly hedged about a fact. This is the single most disorienting thing about working with LLMs and the reason fluency is such a poor guide. You cannot read the model's certainty off its tone, so you supply the uncertainty externally - by knowing which kinds of question are risky and treating them accordingly, rather than waiting for the model to warn you, which it largely cannot.

Diffusion and other image models have their own version of the problem. They learn to generate images that match the statistical patterns of their training set, guided by your prompt - they are not building a physically correct model of the world. So they routinely produce output that is visually convincing but factually or physically nonsensical: staircases that lead nowhere, structurally impossible cantilevers, windows that do not align between floors, and - notoriously - garbled text on signage and drawings, because the model reproduces the look of letters without knowing language.

For designers this matters in two ways. First, an AI concept render is an impression, not a buildable proposition; treat it as mood and geometry to react to, never as a validated design. A beautiful AI image can quietly embed proportions or details that cannot be built, and if you trace over them uncritically you inherit the error. Second, because these models learn 'what an image of X usually looks like,' they smooth everything toward the average and the frequently-depicted - which is exactly the mechanism that produces bias, our next topic. The rule of thumb: use image AI for the qualities a picture can legitimately carry - atmosphere, palette, massing feel - and verify anything the image implies about function, structure, or fact against real design work.

The danger sharpens the more polished the image gets. A loose sketchy render invites scepticism; a photoreal one radiates false authority, and a client shown a gorgeous AI visual will reasonably assume it depicts a real, resolved proposal. That gap - between how finished the image looks and how unresolved the design actually is - is where expectations get set that the built project cannot meet. Guard against it by being explicit that early AI imagery is exploratory, and by never letting a render outrun the design it supposedly represents.

Algorithmic bias is not a moral failing added to the model; it is the training data's skew, faithfully reproduced and often amplified. If a corpus of architectural imagery over-represents glossy Western commercial towers, ask for 'a beautiful modern house' and you will disproportionately get one aesthetic - and under-see courtyard homes, vernacular forms, or the material palettes of most of the world. Ask for 'an architect' or 'a construction worker' and image models have well-documented tendencies to default to particular genders and ethnicities, because that is what dominated their training images. The model is a mirror of its data, and the data is not the world.

This has real design consequences. Bias narrows your creative range precisely when you are using AI to widen it - quietly steering ideation toward the mainstream and the already-photographed. It can produce imagery that misrepresents a community or a context, which is a professional and ethical problem when work is client-facing or public. And in text, the same skew shows up as advice weighted toward the contexts the model saw most - Western codes, assumptions, and norms - which may not fit your site at all. The mitigation is partly awareness (notice the default, and deliberately prompt against it - name the region, the vernacular, the specific people) and partly the discipline of not letting a statistical average stand in for real cultural and contextual judgement, which remains yours.

It helps to name the specific ways these tendencies bite in practice, because they are recognisable once you know them. Invented codes and standards: a model states a setback, a fire rating, or a clause with total confidence and it is simply wrong, or drawn from a different country's code, or from a superseded edition. Fabricated citations and products: references to papers, standards, or product models that do not exist. Plausible-wrong numbers: load figures, U-values, occupancy loads, or cost rates that look reasonable and are off. Biased or generic imagery: concept output that flattens to a single global aesthetic and mis-defaults on people and place. Confident recency errors: a discontinued product or a replaced standard described as current, because it was current when the model was trained.

None of these means the tools are useless - it means their errors are patterned, and patterns are things you can defend against. The next section covers the systematic defences, but the mindset is the takeaway: expect these specific failures, aim your checking at them, and never let a model's fluency substitute for the ground truth of a real code, a real datasheet, and a real understanding of the place you are designing for.

You cannot eliminate hallucination and bias, but you can sharply reduce their reach. The most powerful move is grounding: give the model the actual source to work from rather than relying on its memory. Paste the real code clause and ask it to interpret that; give it the real datasheet and ask it to summarise that. A model reasoning over a document you supplied is far more reliable than one recalling from training, and - crucially - its claims become checkable against the text in front of you.

At scale, this is what retrieval-augmented generation (RAG) does: it fetches relevant, trustworthy documents and has the model answer from them, ideally with citations, so answers are anchored to a source rather than invented. Custom assistants built on your own vetted reference library (Module 8) apply the same principle to a practice. Beyond grounding, the everyday defences from Lesson 9.1 still carry the load: ask for sources and open them; cross-verify across models or phrasings; sanity-check numbers; and against bias, prompt deliberately for the specific region, vernacular, and people you intend, and consciously widen beyond the model's default. None of this is a silver bullet - a grounded model can still misread, and RAG can retrieve the wrong passage - which is why the human-in-the-loop never leaves. The goal is not a model you can trust blindly; it is a workflow where the model's mistakes are contained, visible, and caught before they matter.