I spent the last several months designing and building MoveProtect, a hardwired telematics device for peer-to-peer vehicle rentals. On paper, it's a small board that sits behind a dashboard: a microcontroller, an LTE modem, an IMU, a GNSS receiver, a CAN transceiver, and two MOSFETs wired into the vehicle's lock and ignition circuits. It reports where the car is, how it's being driven, and it can lock, unlock, or immobilise the vehicle remotely.
I came into this project with a reasonable software background and a decent grasp of embedded theory. What I didn't have — and what this project forced on me, one uncomfortable discovery at a time — was an understanding of how differently embedded systems behave once they leave the page. Here's what actually stuck.
Your tools have opinions, whether you asked for them or not
The first real lesson had nothing to do with my own code. STM32CubeMX, the peripheral configuration tool I used to generate the firmware skeleton, has a rule: anything you write outside its designated USER CODE markers gets silently deleted the next time you regenerate the project. I learned this the way you learn most embedded lessons — by losing work. Three separate build failures traced back to the same root cause: code I'd written by hand, sitting just outside the boundary the tool considered its own territory.
The fix wasn't clever. It was procedural: register every peripheral in the tool before writing anything that depends on it, and treat the USER CODE blocks as a hard boundary, not a suggestion. But the deeper lesson was about posture. My instinct, the first time this happened, was to treat it as a bug I needed to work around. It wasn't a bug. It was the tool's documented behaviour, and the fact that I hadn't internalised it was the actual problem.
The same lesson showed up again, more sharply, with the LTE modem. The AT command reference I'd been working from covers an entire family of modules, and for almost everything, that shared documentation is accurate. It wasn't accurate about this: my specific variant persists its unsolicited-output routing configuration in non-volatile memory, and defaults to redirecting all of it to USB whenever a USB cable happens to be physically connected at power-on — a detail the family-level manual doesn't mention, because it isn't true of every module in that family.
I'd trusted the shared reference because it had been right about everything else, and that's exactly the assumption that cost me a session's worth of downlink commands vanishing without a trace. Every MQTT command I sent during development with the debug cable plugged in was being silently rerouted away from the firmware entirely — not dropped, not corrupted, just delivered somewhere the code wasn't listening. The fix, once I found it, was three lines: check the stored setting on boot, correct it if wrong, restart the modem once. The harder fix was learning that "the docs for this family" and "the docs for this exact part" are not the same document, and the difference only shows up once it's already cost you something.
The failure and the cause live in different places
Not every lesson here came from my own code being wrong. One of the stranger debugging sessions I've had started with a stack overflow — a classic embedded failure mode I understood fine in theory, long before I understood it in practice.
The firmware runs several concurrent tasks, each with its own fixed block of memory for its call stack. One of the system's own internal health checks flagged a low-memory warning on the IMU task — its stack watermark had dropped into dangerous territory, exactly the kind of signal you're supposed to trust. So I went looking for a bug in the IMU task. I read through its code, its call depth, its local variables, hunting for whatever was eating stack space. There wasn't anything. The IMU task was fine.
The actual overflow was happening in a completely different task — the GPS task — which was quietly writing past the end of its own allocated stack and corrupting memory that happened to sit adjacent to it, which happened to be the IMU task's stack region. The IMU task wasn't failing. It was a bystander, reporting damage done to it by its neighbour.
This is the part of embedded debugging that nothing quite prepares you for if your instincts were trained on web applications or scripting languages, where a crash points, with rare exceptions, at the line that crashed. In a system with several tasks sharing a flat memory space and no hardware wall standing between them, an error report tells you where the damage was found, not where it was caused. Learning to treat a watermark warning or a crash log as evidence about a symptom — not a pointer to a culprit — was one of the more important shifts I made on this project, and it's not a shift you make by reading about it. You make it by spending an afternoon convinced an innocent task is guilty, and being wrong.
Physical consequences change what "correct" software means
There's a moment in this project I keep coming back to, because it's the clearest example I've ever personally built of what "cyber-physical" actually means, as opposed to just being a phrase you put on a slide.
The immobilisation feature cuts the engine's fuel injection remotely. That's a completely reasonable thing for a rental-security product to do — until you consider that fuel injection is also what's running the engine that's running the power steering pump and, on most vehicles, the vacuum-assisted brake booster. Immobilising a car that's stationary in a driveway is a security feature. Immobilising a car doing 80 km/h on a highway is closer to sabotage.
So the actual command path isn't "receive MQTT message, cut fuel." It's "receive MQTT message, check the vehicle's live GPS speed, and only proceed if that speed is at or below a walking pace — otherwise reject the command and tell the cloud why." That single conditional is, as far as I'm concerned, the most important line of logic in the entire codebase, and it looks completely unremarkable sitting in a function body next to a dozen other lines. Nothing about it is algorithmically interesting. What makes it important is that a bug there doesn't produce a wrong number on a dashboard — it produces a real hazard for a real person driving a real car.
Writing software that can hurt someone if you get the logic wrong changes how you review your own code. I stopped thinking of that speed check as a "business rule" and started thinking of it as a safety interlock, which is a different category of thing that deserves a different level of scrutiny, testing, and paranoia.
The bench is not the world
Every sensor threshold I tuned in a controlled setting turned out to be wrong the first time it met a real road — and it turned out to be wrong in two genuinely different ways.
The first way was noise. The IMU's job is to catch harsh braking and hard cornering from raw accelerometer data. On a bench, a clean deceleration spike looks exactly like what a textbook says a hard-brake event should look like. On an actual drive, a pothole produces a spike that's indistinguishable, in a single sample, from a genuine hard stop. Road vibration does too. A speed bump absolutely does. I didn't find this out by reasoning about it — I found it out by driving around with the device logging events and watching it flag things that were, quite obviously, not harsh braking. The fix was a debounce window: require two consecutive above-threshold readings, 200 milliseconds apart, before accepting the event as real. That number came from repeated test drives, not from a datasheet or a formula.
The second way was context, and it's a different problem than noise entirely. The device once flagged a hard-cornering event — a clean 0.95g of lateral acceleration, comfortably past threshold — from a vehicle that was, according to every other signal available, parked in a driveway with the engine idling. The IMU wasn't wrong. It genuinely measured 0.95g of lateral motion. What it didn't have was any way to know why the acceleration was happening, and a device mounted to a bracket on an idling engine rocks back and forth with enough force to look, from the accelerometer's point of view, indistinguishable from a car taking a corner hard at highway speed.
More samples wouldn't have fixed this — the signal wasn't noise, it was a real, sustained, correctly measured acceleration. The IMU simply had no idea the vehicle wasn't moving. The fix was to stop treating the accelerometer as a source of truth on its own, and instead gate every harsh-event classification on a second, independent signal: the vehicle's actual road speed, read over OBD-II. If road speed is zero, a 0.95g reading isn't a hard corner, no matter how clean the number looks. That's sensor fusion in the most literal sense — two individually untrustworthy signals, cross-checked against each other, producing one conclusion that neither could produce alone.
The same category of lesson showed up again with GNSS. A cold satellite fix without any assistance data can take sixty to a hundred and twenty seconds in the worst case, which is a fine number on a spec sheet and a genuinely bad experience if your actual use case is someone starting a short trip and being gone before the device has any idea where they are. The fix there was injecting satellite ephemeris data over the cellular connection to shortcut acquisition — a solution I only reached for once I'd personally felt how bad a ninety-second wait is when it's happening in a moving vehicle instead of a spec sheet.
The network is part of your system, not a pipe underneath it
I used to think of connectivity as infrastructure — something that either works or doesn't, and mostly not my problem once I've opened a socket. That framing didn't survive contact with a real carrier network.
The SIM connectivity I used routes through a partner network, and that network does things to the modem's configuration that nothing in the AT command reference warns you about — pushing auxiliary data contexts into the modem's connection table that have to be explicitly torn down before the real data connection can activate. DNS resolution for the broker's hostname failed intermittently in a way that had nothing to do with my code and everything to do with which resolver happened to answer first. The broker itself, being a multi-tenant service sharing one TLS endpoint across many customers, requires Server Name Indication to route the handshake to the right tenant — a detail that's invisible until your connection mysteriously terminates at the TLS layer for reasons that look, from the firmware's side, exactly like nothing at all.
None of this is exotic. It's just what happens when you stop treating "the network" as a black box that either has data or doesn't, and start treating it as a system with its own quirks, failure modes, and undocumented behaviours that are just as real as anything happening on your own microcontroller. Reconnection logic, backoff timing, and graceful handling of a connection that silently vanished mid-session aren't nice-to-haves for an embedded system that has to survive being installed in a vehicle for months at a time. They're core functionality, and I only came to believe that after enough sessions of watching a perfectly good piece of firmware sit there, blind, because of something happening three layers below anything I'd written.
Verification is not a phase — it's a posture
The single habit that ended up mattering most across this entire project was refusing to trust a description of what the system did, including my own descriptions, without checking it against the actual running code. More than once, I found that something I'd written down as fact — how often a task actually polled, what a particular module's real responsibility was, which parameters were actually being enforced versus just measured and logged — didn't match what the source code was actually doing. Not because anyone was being careless, but because firmware evolves faster than the sentences describing it, and the two drift apart quietly if nobody goes back and checks.
The instinct to treat documentation as provisional until verified against source — and to say "I don't actually know that" rather than restate something that sounds right — turned out to be worth more than almost any individual technical skill I picked up. Embedded systems punish confident-sounding guesses in a way that's hard to appreciate until a guess makes it into a system that's now bolted into a moving vehicle.
What I actually took away
Somewhere in the middle of testing, the system sent an alert email to a customer: Impact Detected. Their car hadn't been hit. It hadn't moved. It was sitting in their driveway with the engine running, exactly where they'd left it.
Nothing about that alert was a bug, in the narrow sense. The accelerometer reading was accurate. The threshold logic executed exactly as written. Every individual piece of the system did precisely what I'd told it to do, and the result was still a false alarm sent to a real person about a car that was never in danger. The code was correct, and the outcome was wrong, and closing that gap meant admitting that correctness at the level of logic isn't the same thing as correctness at the level of the world the logic is supposed to be describing.
That gap is, in the end, the whole difference between writing software and writing software for something physical. An accelerometer doesn't know it's bolted to a car. A stack overflow doesn't know it's about to make an innocent task look guilty. A modem doesn't know its factory default will silently swallow every command you send it. None of these systems are malicious, or even wrong, exactly — they're just narrower than the world they're embedded in, and the entire discipline turns out to be the work of noticing where that gap is before it ships, rather than after.
