There is a standard way to demo an IoT project. You hold up the hardware, explain what the sensors measure, and then swipe to a dashboard showing a map, a chart, or a live metric ticking upward. The audience sees the device and they see the data. What they don't see — what almost nobody thinks to show — is the roughly five layers of infrastructure sitting between those two things, doing work that is unglamorous, poorly documented, and absolutely load-bearing.
This is about that middle.
What the Middle Actually Is
When a sensor detects an event, it doesn't "go to the dashboard." It gets encoded into a compact binary format, transmitted over an encrypted connection to a broker, received by a server-side process, decoded from raw bytes into structured fields, written to a database, picked up by background workers, interpreted against domain logic, and — if all of that succeeds — eventually reflected in the UI.
That chain is the middle. And in most IoT systems, it's where the majority of actual engineering decisions live. The hardware side has clear references: datasheets, vendor SDKs, application notes. The application side has frameworks and decades of web convention. The middle has none of that. There is no standard shape for an IoT ingestion pipeline. The tradeoffs are yours to make, and wrong choices compound.
The Packet Is a Contract
The first decision in the middle layer is what the transmitted data looks like. This seems like a firmware question, but it belongs to both sides simultaneously — and the cost of getting it wrong is high.
Embedded devices typically transmit compact binary packets. Every field has an assigned position, an encoding convention, and a unit resolution sized to fit within a tight byte budget. The server has to know all of this. The database schema has to know it. Every downstream process has to agree on what each field means.
In practice, this means the packet format isn't just a representation — it's a contract. When you change it, everything downstream changes with it. Undisciplined packet design is one of the most common sources of silent corruption in IoT pipelines: the device is sending one thing, the decoder is interpreting it as another, and the numbers on the dashboard look almost right. Two practices that survive contact with real systems: include an explicit version byte in every packet, because you will change the format and you want to know which version you're looking at when you do; and encode physical units unambiguously at the packet level, because ambiguity about what a raw value represents gets expensive once it's propagated through four layers of infrastructure.
The Broker Has Opinions
Message brokers are usually treated as transparent infrastructure — a message goes in, a message comes out, the broker's job is routing. This is mostly true in development, on a local broker, with one client. In production, the broker has opinions.
Managed brokers often require specific configuration to route connections correctly across shared infrastructure — details that only surface when a connection fails silently during the handshake. Brokers enforce topic-level permissions that have to be designed before a single device ships, because the credentials a device carries determine what it can publish and subscribe to for its entire deployed lifetime. Changing that after deployment is painful in direct proportion to how many devices are already in the field.
The point isn't that brokers are complicated. It's that every component in the middle has its own behaviour, its own configuration surface, and its own failure modes that look, from any other layer, like nothing at all. Treating infrastructure as transparent is a reasonable simplification for a prototype. It stops being reasonable the moment a real device connects from a real network.
Separate Ingestion from Processing
The most important structural decision in any IoT pipeline — one that usually takes iteration to arrive at — is this: the component that receives data should do as little as possible, and everything else should happen separately.
The ingestion layer's only job is to receive a packet, decode it, and durably store the raw data. It should not evaluate alerts. It should not compute scores. It should not decide whether a meaningful event has occurred. It stores and gets out of the way.
Processing happens asynchronously, in separate workers, against the stored data. The reason this matters is operational: workers can be re-run. If you discover a bug in your event detection logic, you can fix it and replay against historical data. The raw record is the ground truth; everything derived from it is a function of whatever logic was running at the time. If you bake processing into ingestion, you lose that ability. Every historical record was processed by the old, wrong version, with no clean path back.
The cost of this separation is latency. For most IoT applications, that's acceptable. For time-sensitive paths — a safety alert that needs to fire within seconds — it isn't. The architecture needs to account for both: a fast path for urgent signals, and an async path for everything that can wait.
Normalization Is Where Meaning Is Created
Raw sensor data is not information. It becomes information when it's normalised against context.
The same accelerometer reading means different things depending on whether the vehicle was stationary or moving, in a turn or on a straight road, whether the reading arrived as part of a sequence or as an isolated spike. Without that context, you have a number. With it, you have a signal.
This is the work the middle layer actually does. Ingestion stores what the sensor measured. Processing assembles context — trip boundaries, preceding conditions, vehicle state, location — and interprets the measurement against it. The gap between a raw sensor reading and a meaningful event on a dashboard is filled entirely by that assembly work. Skipping a stage doesn't simplify the pipeline — it just means the missing context gets reconstructed incorrectly somewhere downstream, or not at all.
What Gets Skipped, and What It Costs
The unglamorous middle is also the part that gets deferred. Hardware gets built because the prototype needs to exist. The dashboard gets built because the demo needs to look like something. The pipeline gets built because, eventually, there's no way around it.
The cost of deferring it is proportional to how much data has accumulated in the meantime. Months of raw records with no processing layer means months of backfill work before the first meaningful analysis is possible. Processing logic designed retroactively, against a format that wasn't built with it in mind, often discovers that the fields it needs weren't captured — or were captured in a form that can't be recovered without guesswork.
The other casualty of deferred pipeline work is the audit trail. When something goes wrong — a false alert fires, an event is missed, a metric looks implausible — the first question is always: what did the raw data actually say? If the ingestion layer preserved it, you have an answer. If it processed and discarded the raw form, you're debugging in the dark.
What Good Looks Like
A well-designed middle layer is, ultimately, one you stop thinking about. Ingestion is boring. Processing workers are independently re-runnable. The data format is versioned and stable. Raw data is preserved, and everything derived from it is reproducible.
None of that is intellectually interesting. It doesn't make for good demo footage. But building MoveProtect made it clear, eventually, that this is precisely the point — the middle layer's job is to be invisible, reliable, and correct. Not to be clever.
The sensor is the beginning of the story. The dashboard is the end. Everything in between is where the story actually happens.
