Building Data Architecture

Picture a temperature sensor in an air handling unit reporting 22.6 degrees C at 09:10. That number has to travel, and almost every building-data system moves it through the same four stages.

Ingest is the front door. The reading is collected from its source - a BMS controller over BACnet, an energy meter over Modbus, a wireless sensor publishing over MQTT - and pulled onto the platform. Ingestion has to cope with many protocols, unreliable networks and bursts of traffic, so it usually buffers data (often through a message broker or queue) so nothing is lost when a downstream system is slow.

Store is where the reading comes to rest. Building data is overwhelmingly time-series (a timestamp, a value, a tag), so the natural home is a time-series database, often beside a data lake that keeps raw, unstructured history cheaply for later. Storage has to be durable and cheap, because you will keep years of it.

Process is where raw numbers become useful. Readings are cleaned (drop the impossible -300 degree spike), aligned to common units and timestamps, enriched with context (which unit, which zone), aggregated (5-second raw rolled up to 5-minute averages) and run through rules or models. This is where a lone reading becomes AHU-1 supply air is drifting high.

Serve is delivery. Processed data is exposed through APIs, dashboards, alerts and the digital twin so a person or a piece of software can finally act. If nothing is served, the pipeline is a very expensive way to fill a disk. The art of a good architecture is making all four stages reliable, observable and cheap enough to run forever.

It helps to hold the whole journey in one image: our 22.6-degree reading is published by the sensor, buffered in a broker so a momentary outage loses nothing, written to a time-series store, averaged and checked against the unit it belongs to, and finally drawn on a chart an engineer glances at over coffee. That is a lot of machinery for one number - but the machinery is what lets you trust the number, and trust the ten thousand others arriving behind it. Notice, too, that value is added at every stage: ingestion adds reliability, storage adds memory, processing adds meaning, serving adds reach. Skip a stage and you do not just lose a feature, you lose everything that depended on it downstream.

The deepest design choice in building data is when processing happens. There are two modes, and mature buildings run both.

Batch processing collects data first and works on it later, in bulk - the classic overnight job. At 2am the platform sweeps up the day's readings and produces energy reports, benchmarks against last month, and re-trains a machine-learning model. Batch is simple, cheap, and perfect for anything where a few hours of delay costs nothing. Its weakness is latency: a fault detected in this morning's batch may already have wasted a night of energy.

Streaming (or real-time) processing handles each reading as it arrives. The moment a chiller trips or a CO2 level crosses a threshold, a streaming pipeline can raise an alarm, update a live dashboard, or feed a control loop - latency measured in milliseconds to seconds rather than hours. Streaming is more complex and more expensive to run, but it is the only way to serve now decisions.

The honest answer is almost never one or the other. A well-designed building runs a fast lane (streaming, for alarms, live views and control) alongside a slow lane (batch, for reports, benchmarking and model training) over the same stored data - an approach the industry sometimes calls a lambda or kappa architecture. When you evaluate a platform, ask which decisions it serves in real time and which it serves overnight; a vendor who streams a pretty dashboard but batches every actual alert has sold you a slower building than the demo suggested.

Data architecture is also a question of place. The same pipeline can run close to the equipment or far away in a data centre, and the trade-off shapes everything.

At the edge - on a controller, gateway or small computer inside the building - processing happens metres from the sensor. Edge processing cuts latency (a safety interlock cannot wait for a round-trip to the cloud), keeps working when the internet drops, and reduces how much raw data you ship. A common pattern is to filter and aggregate at the edge, sending summaries up rather than every raw sample. A gateway often sits between old field devices and the modern network, translating BACnet or Modbus into MQTT and pushing it onward.

In the cloud (or a central server), you get near-unlimited storage and compute, easy access from anywhere, and the horsepower for cross-portfolio analytics and heavy models. The cost is dependence on connectivity and a bigger attack surface - every connected building is an operational-technology system that must be secured, a theme Module 8 takes seriously.

Most real architectures are hybrid: time-critical logic and buffering at the edge, deep history and analytics in the cloud, a gateway stitching them together. A worked example - a campus streams live equipment status to a local edge server for instant alarms and control, while a nightly batch ships aggregated data to the cloud for portfolio energy benchmarking. Same readings, two paths, two very different jobs.

It is tempting to treat the pipeline as boring plumbing beneath the exciting twin. That is exactly backwards. A digital twin is only as good as the data it is fed, and every quality problem you will ever fight lives in the architecture.

Data quality is the first casualty of a weak pipeline. Sensors drift, drop out, report in the wrong units, or stamp readings with the wrong time. If ingestion and processing do not catch and flag these, the twin confidently reasons on garbage - and a twin that is wrong is worse than no twin, because people trust it. Good architectures build in validation, gap-filling rules and health monitoring of the data itself.

Governance matters just as much. Who can read this data? How long is it kept? Occupancy and access data can reveal where people are and when - genuinely private information - so retention and access rules are not optional niceties. Statutory and security sign-off belongs with qualified professionals, but the architecture is where those decisions are enforced.

Finally, openness. An architecture that traps data in one vendor's proprietary format is a liability the day you want a twin, a new analytics tool, or a different supplier. Favour standard protocols in, standard formats at rest, and open APIs out. The buildings that stay useful for decades are the ones whose data architecture was designed to be read by things not yet invented.

A last practical point: architecture is never finished, it is operated. Sensors fail, networks change, vendors update firmware, and a pipeline that worked on the day it was commissioned will silently rot without attention. The mature move is to monitor the pipeline itself - are readings still arriving, on time, in range, from every point? - with the same seriousness you monitor the building. Treat missing data as an incident, not an inconvenience. The teams who get lasting value from a twin are rarely the ones who bought the cleverest analytics; they are the ones who kept the unglamorous flow of good data running, year after year, so the clever analytics always had something honest to chew on.