← Back to Blog
For: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

Edge AI Anomaly Detection on IoT Fleets Learns to Go Silent

Verdict skew: a fleet hears about its false alarms in days and its missed failures in months, and that gap decides more than GRU versus Mamba ever will.

iot-edge-computingai-engineering
#edge-ai#iot#predictive-maintenance#anomaly-detection#mlops#concept-drift

On 24 July 1994 an explosion at the Texaco refinery in Milford Haven released about 20 tonnes of flammable hydrocarbons, and the fires burned until the evening of 26 July. The UK Health and Safety Executive's summary lists a chain of failures. A control valve was shut while the control system showed it open. Plant modifications had gone in without an assessment of their consequences. And the alarm system buried the people it served: "Excessive number of alarms in emergency situation reduced effectiveness of operator response." The alarms were firing. There were too many of them to act on. This article's simulation of a predictive maintenance fleet reaches the same kind of flood by week 36, after a quarter of near silence.

An edge AI anomaly detector on a fleet of Internet of Things (IoT) devices gets there through nothing more than its weekly tuning meeting on delayed labels. False alarms get labelled within days. Missed faults get labelled when the machine breaks, months later. So every week's evidence argues for fewer alerts, right up to the quarter when the misses arrive all at once. Tuning thresholds on resolved alerts is a natural way to fight alert fatigue. On a fleet with verdict skew, that habit is the failure.

My claim is that this delay structure decides more about an edge AI fleet than the model architecture does: verdict skew pushes any detector toward silence, and you cannot even measure what a better architecture bought until its misses have had time to surface. GRU against TCN against Mamba, INT8 against FP16: these choices are cheap to reverse compared with what follows, even though each one costs a firmware release. The ML team can shorten some verdicts: an explanation speeds an inspection, a teardown audit recovers censored repairs, a test rig fails on purpose. It cannot make a missed fault in the field fail sooner. The plant and the maintenance process set that, and its shape, which I name below, dictates what data must leave the device and how a new model can be judged after it ships. Nothing in the mechanism needs the model to sit on the device. The edge makes it worse, because a model running on the device decides which evidence survives.

Edge AI anomaly detection model choice matters less than label timing

Design reviews for these systems spend their time on a model table. Recurrent nets for the device, Transformers for the cloud, maybe a State Space Model (SSM) for long histories. Evidence from 2026 says that even the largest jump on that table, to a time-series foundation model, buys little on anomaly detection.

At ICLR 2026, Zhu, Carpentier and Verbeke tested five families of time-series foundation models (MOMENT, Chronos, TimesFM, Time-MoE and TSPulse) on anomaly detection. Across model sizes and context lengths, none did significantly better than two one-line baselines: moving-window variance and squared difference. Their explanation is worth keeping: "anomalies are not consistently harder to reconstruct or forecast," which breaks the assumption under every forecast-error detector. An ICML 2026 study on oil and gas drilling found the 120M-parameter Chronos-2 had the best F1 on both of its tasks, but a 5M-parameter CNN-FNO "reaches near-parity" on torque "with much lower inference time."

On the device itself, inference is already cheap. In the 2022 1.0 round of MLPerf Tiny, a benchmark described by Banbury et al. (2021), the fastest submission on the anomaly detection task ran in 5.6 ms on an Arm Cortex-M4 at 120 MHz (Plumerai's vendor-reported figure from 2022). Where exotic architectures do cost something is at deployment. BabyMamba-HAR, a 27K-parameter selective SSM for microcontrollers, keeps its recurrent state in float32 and only its projections in INT8; the fully INT8 baselines it was compared against lost dataset coverage on the Pico 2 and ESP32. FEMBA, a Mamba EEG model on a GAP9 microcontroller, reports that standard post-training quantization cost about 30% accuracy, and quantization-aware training was needed to recover it.

d2
grid-rows: 1
grid-gap: 48
classes: {
  argued: {style: {fill: "#95A5A6"; stroke: "#6F7E7F"; font-color: "#2C2C2A"; font-size: 18}}
  decides: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"; font-size: 18}}
}
review: "What the design review argues about" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 18
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-size: 20}
  a: "GRU vs TCN vs Mamba" {class: argued}
  b: "INT8 vs FP16 weights" {class: argued}
  c: "Which Transformer\nfor the cloud tier" {class: argued}
}
decides: "What decides whether the fleet works" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 18
  style: {fill: "#EAF2FC"; stroke: "#4A90E2"; font-size: 20}
  a: "When each kind of\nverdict arrives" {class: decides}
  b: "Which data leaves\nthe device" {class: decides}
  c: "How a model reaches the\nfleet and comes back" {class: decides}
}

Most of the meeting goes to the left column, and once the hardware is fixed, every item on it can be changed with a retrain and a release. An SSM choice that needs float32 state can decide the silicon itself, so that one item belongs partly in the right column. The right column is set outside the ML team, by the plant and by how maintenance and releases run, and once the fleet is installed it is very expensive to change. From here on this article is about the right column, starting with its top item.

Verification latency per confusion-matrix cell in predictive maintenance

Concept-drift research has a name for the delay between a prediction and the moment its true label is known. Marrs, Hickey and Black studied how this latency hurts learning under concept drift in 2010, and later work named it verification latency. That work studied the extreme case where labels never arrive at all (Souza et al., 2015) and how to evaluate a stream when labels come late (Grzenda, Gomes and Bifet, 2020). On an IoT fleet that does predictive maintenance, treating that delay as a single number hides the problem.

How long the delay runs depends on which cell of the confusion matrix a prediction lands in. A false alarm gets its verdict fast: a technician opens the housing, usually within days, and finds nothing. A missed fault usually gets its verdict when the machine fails in the field, weeks or months later, or when a scheduled inspection happens to find it. A correct alarm that was acted on often gets no clean verdict at all. The repair removed the failure that would have proved it right, and a replaced part is rarely torn down to check. NannyML calls this the censored confusion matrix in predictive maintenance. A correct silence is confirmed only by absence, over the life of the asset.

I call this verdict skew: verification latency that differs by cell of the confusion matrix, from days to months, with one cell that often never resolves. Its consequence is that each week's labels are dominated by the errors that are cheapest to confirm. Matured-verdict gating, later in this article, is its fix.

d2
grid-rows: 3
grid-columns: 3
grid-gap: 10
classes: {
  head: {style: {fill: "#95A5A6"; stroke: "#6F7E7F"; font-color: "#2C2C2A"; font-size: 17; bold: true}}
  fast: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"; font-size: 16}}
  slow: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; font-size: 16}}
  never: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"; font-size: 16}}
  absent: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"; font-size: 16}}
}
corner: "" {style: {opacity: 0}}
h1: "A fault was developing" {class: head}
h2: "No fault" {class: head}
r1: "Device alerted" {class: head}
tp: "Correct alarm\nverdict: often censored\n(the repair removes the proof)" {class: never}
fp: "False alarm\nverdict: hours to days\n(inspection finds nothing)" {class: fast}
r2: "Device stayed silent" {class: head}
fn: "Missed fault\nverdict: weeks to months\n(arrives as a field failure)" {class: slow}
tn: "Correct silence\nverdict: only by absence,\nover the asset's life" {class: absent}

Green is the only cell that is reliably fast, and it holds the error that argues for alerting less. Orange is fast when the repair records the fault and silent when it does not. Red holds the error that argues for alerting more, and its evidence arrives a season late. Base rates make the asymmetry worse. Backblaze's 2025 drive statistics report an annualized failure rate of 1.36% across 344,196 drives. That is about 90 failures a week, but only about 0.026% of drives in any given week, so true positives are a thin slice of each week's labels.

Backblaze's own analysis of SMART signals found that "23.3% of failed drives showed no warning from the SMART stats we record," while 4.2% of operational drives had at least one of the five tracked stats above zero. A flagged population full of healthy hardware is the normal case, and every one of those healthy units comes back quickly as a clean label.

The closest neighbouring idea is in fraud detection. Dal Pozzolo and colleagues model the alert-feedback interaction, where alerted transactions get investigator labels quickly and the rest arrive later as chargebacks, and they treat the bias that creates. Delayed-feedback models in display advertising (Chapelle, 2014) handle conversions that arrive later than clicks. The selective labels problem (Lakkaraju et al., 2017) covers outcomes you never observe because a decision intervened, which is the censored cell here. Hidden feedback loops (Sculley et al., 2015) and performative prediction (Perdomo et al., 2020) describe models that change the data they are later judged on. All of these share the core problem: when one kind of label arrives later, or never, recent data is biased.

Two older practices are close too. Reliability engineering calls the miss window the P-F interval, from potential failure to functional failure, and credit scoring has long judged models only on loan vintages old enough to have a full performance window, which is the matured-verdict idea. Verdict skew is not label skew in the class-imbalance sense; the skew is in when labels arrive. It differs from all of these in three ways. First, the delay is keyed to the cell of the confusion matrix rather than to the decision. In fraud, an alerted transaction gets an investigator's label either way. In the alerted row of a maintenance fleet, the false alarm is confirmed in days while the true alarm next to it is often censored by its own repair. Second, the intervention itself censors the true-positive cell. And the subject is a control loop that runs every week, where a quarter-long lag produces silence first and overshoot later.

Alert fatigue in predictive maintenance: how a detector learns to go silent, then floods

The simulation below puts ten thousand devices through 40 weeks of tuning. Each week 0.2% of them develop a fault, and the detector's scores separate faults from normal operation by two standard deviations. Each missed fault surfaces as a field failure 13 weeks later, and only half of the repaired true alarms record the fault they fixed. Every number is a stated assumption; none comes from a real fleet. At 0.2% a week, the fault rate is nearly 8 times Backblaze's drive rate, chosen so the effect shows inside 40 weeks. The tuning rule does what a weekly meeting does with whichever verdicts arrived that week. More than 12 field failures means lower the threshold, and a false-alarm share above 80% means raise it.

python
import randomDEVICES = 10_000FAULT_RATE = 0.002   # share of devices that develop a fault in a given weekMISS_DELAY = 13      # weeks until a missed fault surfaces as a field failureCONFIRM_RATE = 0.5   # share of repaired true alarms where the fault is recordeddef simulate_week(rng):    """One week of fleet data: (score, has_fault, fault_recorded) per device.    fault_recorded matters only if the device alerts and gets repaired."""    rows = []    for _ in range(DEVICES):        fault = rng.random() < FAULT_RATE        score = rng.gauss(2.0 if fault else 0.0, 1.0)        rows.append((score, fault, fault and rng.random() < CONFIRM_RATE))    return rowsdef false_alarm_share(rows, threshold):    alerts = [fault for score, fault, _ in rows if score > threshold]    return alerts.count(False) / max(len(alerts), 1), len(alerts)def surfaced_misses(history, week):    """Missed faults from MISS_DELAY weeks ago that fail in the field now."""    return sum(1 for w, t, rows in history if week - w == MISS_DELAY               for score, fault, _ in rows if fault and score <= t)def tune_on_arrived_verdicts(threshold, rows, misses_now):    """Wrong way: react to whichever verdicts arrived this week."""    if misses_now > 12:        return round(threshold - 0.1, 1)    share, _ = false_alarm_share(rows, threshold)    return round(threshold + 0.1, 1) if share > 0.8 else thresholddef recall_at(rows, threshold):    faults = [score for score, fault, _ in rows if fault]    return sum(1 for s in faults if s > threshold) / len(faults)rng = random.Random(7)threshold, history = 2.0, []print("week  threshold  alerts  misses_surfaced  true_recall")for week in range(1, 41):    rows = simulate_week(rng)    history.append((week, threshold, rows))    misses_now = surfaced_misses(history, week)    _, n_alerts = false_alarm_share(rows, threshold)    if week % 4 == 0 or week in (1, 13, 14):        recent = [r for w, t, rs in history[-4:] for r in rs]        print(f"{week:>4} {threshold:>10.1f} {n_alerts:>7} {misses_now:>16} "              f"{recall_at(recent, threshold):>12.2f}")    threshold = tune_on_arrived_verdicts(threshold, rows, misses_now)

misses_surfaced counts the field failures the team learns about that week. true_recall is the recall the current threshold gets on the last four weeks of devices, which only the simulation can see.

text
week  threshold  alerts  misses_surfaced  true_recall   1        2.0     230                0         0.57   4        2.3     110                0         0.40   8        2.7      53                0         0.21  12        3.0      22                0         0.23  13        3.1      13                0         0.21  14        3.1      12               10         0.19  16        3.0      14               16         0.16  20        2.6      61               21         0.25  24        2.2     149               19         0.42  28        1.8     385               19         0.52  32        1.4     803               18         0.79  36        1.2    1149               11         0.86  40        1.6     523                9         0.72

For thirteen weeks the loop raises the threshold, from 2.0 to a peak of 3.1. Alerts fall from 230 a week to 12 by week 14, and true recall falls from 0.57 to 0.16 by week 16. No field failure has surfaced yet, so every signal the team has says the detector is improving.

Field failures start to surface in week 14, and once they pass 12 a week the loop starts lowering. It keeps lowering for months, because the failures still arriving were caused by thresholds it abandoned a quarter ago. By week 36 the threshold is 1.2 and the fleet sends 1,149 alerts a week. The evidence was correct and late, and reacting to it swung the loop from silence to an alarm flood.

How far it swings depends on the rule. This one checks field failures before alert volume, and a rule with an alert budget would cap the flood. But no rule that waits for field failures can shorten the first quarter of tuning blind. A model of the delay distribution, as in Chapelle's work or a survival analysis of time to failure, can estimate recall sooner, with wider error bars.

Model quality decides the shape of the failure, not whether it happens. Rerun the same loop with fault scores centred at 3.0 instead of 2.0, a detector that separates faults much better: change 2.0 if fault to 3.0 if fault in simulate_week. Three rows of its output:

text
week  threshold  alerts  misses_surfaced  true_recall   1        2.0     237                0         0.87  13        2.8      36                0         0.59  40        2.7      47                9         0.67

The threshold climbs to 2.8 by week 13 and then drifts between 2.6 and 2.8. Field failures mostly stay below the rule's trigger of 12 a week, so the loop lowers only rarely. True recall falls from 0.87 to about 0.56 through weeks 14 to 28 and recovers only partly, to 0.67 by week 40. There is no flood, only a long, silent loss of a quarter to a third of the recall. Two simplifications in this simulation cut in opposite directions. The weekly meeting sees true labels for every alert, where a real one would count some censored repairs as false alarms and raise even sooner. And each fault gets one week to be caught, where a real developing fault keeps scoring high for weeks, so per-fault recall is higher and the swing smaller than shown.

d2
direction: down
classes: {
  main: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"; font-size: 17}}
  decision: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"; font-size: 17}}
  fast: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"; font-size: 17}}
  warn: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"; font-size: 17}}
  error: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; font-size: 17}}
}
t: "Alert threshold" {class: decision}
a: "This week's alerts" {class: main}
i: "Inspections" {class: main}
fp: "Verdicts back within days:\nmostly false alarms" {class: fast}
tune: "Raise the threshold" {class: warn}
miss: "Missed faults pile up" {class: error}
late: "Their verdicts:\nfield failures, 13 weeks later" {class: error}
t -> a: "1. alerts fire"
a -> i: "2. every alert inspected"
i -> fp: "3. fast labels"
fp -> tune: "4. weekly review"
tune -> t: "5. loop closes in one week"
t -> miss: "fewer alerts"
miss -> late: "outside the loop"

No single step in this loop is wrong. On the left, the loop closes every week, and every turn of it is justified by real labels. The chain on the right reaches the threshold a quarter after the decisions it should have informed, so the loop first optimizes the only error it can see and then overcorrects on stale evidence. A 2026 preprint on wind turbine Supervisory Control and Data Acquisition (SCADA) data shows the quiet end of that trade: a chronologically calibrated method tuned down to 4 false-positive events, from 17, and "only 5 of 18 anomaly events were detected."

How to tune anomaly detection thresholds with delayed labels

The fix is patience, enforced by a rule about which evidence may judge a change. I call the rule matured-verdict gating. Call a prediction matured once enough time has passed for its miss to have surfaced if it was one. A raise ships only after it has been replayed against matured predictions and its recall holds above a floor. The gate needs two things the wrong-way loop never used: the stored raw scores, so a proposed threshold can be replayed, and a measured record rate, so censored repairs do not distort the count. Append this block to the same file as the first one; it reuses its functions and constants.

python
def matured_recall(history, now, threshold, record_rate, min_faults=50):    """Recall at a proposed (higher) threshold, from weeks old enough for    every miss to have surfaced. Repairs that recorded no fault are censored,    so recorded catches are scaled up by the measured record rate."""    caught, missed = [], []    for w, t, rows in history:        if now - w < MISS_DELAY:            continue        for score, fault, recorded in rows:            if fault and score <= t:                missed.append(score)      # surfaced as a field failure            elif recorded:                caught.append(score)      # repair confirmed the fault    total = len(caught) / record_rate + len(missed)    if total < min_faults:        return None    hits = sum(1 for s in caught if s > threshold) / record_rate    hits += sum(1 for s in missed if s > threshold)  # after a past lowering    return hits / totaldef gated_tune(threshold, rows, history, now, record_rate, floor=0.40):    """Right way: a raise ships only with a matured recall verdict."""    share, _ = false_alarm_share(rows, threshold)    if share <= 0.8:        return threshold, "no change proposed"    proposed = round(threshold + 0.1, 1)    recall = matured_recall(history, now, proposed, record_rate)    if recall is None:        return threshold, "blocked: too few matured faults"    if recall < floor:        return threshold, f"blocked: matured recall {recall:.2f} < {floor}"    return proposed, f"accepted: matured recall {recall:.2f}"# record_rate comes from a teardown audit of replaced parts; here the# simulation's true rate stands in for that auditfor record_rate in (CONFIRM_RATE, 1.0):    rng = random.Random(7)    threshold, history = 2.0, []    print(f"record_rate assumed {record_rate}:")    for week in range(1, 41):        rows = simulate_week(rng)        history.append((week, threshold, rows))        threshold, verdict = gated_tune(threshold, rows, history, week,                                        record_rate)        if week == 13 or "accepted" in verdict or week == 40:            print(f"  week {week:>2}: threshold {threshold:.1f}  {verdict}")
text
record_rate assumed 0.5:  week 13: threshold 2.0  blocked: too few matured faults  week 15: threshold 2.1  accepted: matured recall 0.48  week 16: threshold 2.2  accepted: matured recall 0.41  week 40: threshold 2.2  blocked: matured recall 0.37 < 0.4record_rate assumed 1.0:  week 13: threshold 2.0  blocked: too few matured faults  week 40: threshold 2.0  blocked: matured recall 0.28 < 0.4

With the measured record rate, the same false-alarm pressure proposes a raise almost every week, and the gate refuses all but two. Nothing ships for fourteen weeks, because there are fewer than 50 matured faults to judge with. The wait scales as roughly MISS_DELAY + min_faults / (devices x weekly fault rate): 13 + 50 / 20 here, and about 32 weeks for the same fleet at Backblaze's failure rate. Two raises pass at weeks 15 and 16, and from then on the threshold holds at 2.2 for the rest of the run.

Treat the record rate as 1.0, as if every repair recorded its fault, and the censored repairs vanish from the count. Estimated recall drops to 0.28 and the gate freezes at 2.0 for good. Censoring makes a naive gate pessimistic, so the record rate has to be measured, for example by tearing down a sample of replaced parts.

As written, the gate only judges raises. A lowering costs alert volume, which has a fast verdict, so it can ship behind health gates that include an alert budget, and that is how a gated fleet answers recall lost to drift. Raises are harder under drift, because matured data describes the fleet as it was a quarter ago: restrict the replay to the most recent matured weeks, and block raises while an input-drift alarm is open.

The floor of 0.40 sits between the expected recall at 2.2 (about 0.42) and at 2.3 (about 0.38), so it decides where the threshold holds. With a floor of 0.30 the same run climbs to 2.6, one step past where it should stop, since the expected recall at 2.6 is about 0.27. Fifty matured faults are too few to resolve a 0.03 gap, which is why min_faults needs the sizing rule in the checklist below. On a real fleet the floor comes from what a missed failure costs, and min_faults should be large enough that the recall estimate's uncertainty is smaller than the distance to the floor.

d2
grid-rows: 1
grid-gap: 48
classes: {
  fast: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"; font-size: 17}}
  bad: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; font-size: 17}}
  gate: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"; font-size: 17}}
  good: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"; font-size: 17}}
}
wrong: "Tuned on arrived verdicts" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  style: {fill: "#FDEDEC"; stroke: "#E74C3C"; font-size: 20}
  a: "Threshold 2.0 up to 3.1,\nthen down to 1.2" {class: fast}
  b: "Alerts 230 down to 12,\nthen up to 1,149 a week" {class: fast}
  c: "True recall 0.57 to 0.16,\nthen 0.86" {class: fast}
  d: "Raises on fast labels,\nlowerings on stale ones" {class: bad}
}
right: "Gated on matured verdicts" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  style: {fill: "#EAFAF0"; stroke: "#6BCF7F"; font-size: 20}
  a: "First change at week 15" {class: gate}
  b: "Threshold 2.0 to 2.2,\nthen held" {class: gate}
  c: "Raise to 2.3 blocked\nat recall 0.37" {class: gate}
  d: "Works only with a\nmeasured record rate" {class: good}
}

In forty weeks the gated fleet changes its detector twice and the ungated one changes it almost every week, in both directions. That 14-week wait is the real speed limit, because no recall verdict from field failures exists on this fleet any sooner. Without a model of the delay distribution, a fleet that learns about misses from field failures cannot measure a recall change sooner than those failures arrive. Work that needs no recall verdict can still move weekly, and that includes drift checks, crash tracking, and changes aimed at alert volume or latency. Keep the two kinds of iteration on separate clocks.

Edge vs cloud data split for predictive maintenance labels

Bandwidth is the usual argument for edge AI: process locally and send only the events. That is correct, and the strongest result in this area supports a version of it. Vervlimmeren and colleagues, working on four years of 10-minute SCADA data from an offshore wind farm, found that a time-aware fleet-median filter "reduces the volume of anomaly scores by up to 65 % without sacrificing predictive accuracy." The filter works because the cloud tier sees what no single turbine can: whether the whole farm is behaving strangely at the same moment.

Verdict skew adds a second rule to the split. Events that carry their scores are enough to replay a raise. Alert bits alone are not, and then every threshold change is judged by the fast labels. Continuous scores are what let you replay any threshold below the one that was in force, and trace a missed fault's onset back from its failure. Without raw data from before the failure, the training set never gets the one example it most needs.

d2
direction: down
classes: {
  dev: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"; font-size: 17}}
  cloud: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"; font-size: 17}}
  ledger: {style: {fill: "#9B59B6"; stroke: "#7D3C98"; font-color: "#FFFFFF"; font-size: 17}}
  ticket: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"; font-size: 17}}
}
det: "Device: detector\nscores every window" {class: dev}
ticket: "Maintenance ticket\nor field failure" {class: ticket}
fleet: "Cloud: fleet-median filter" {class: cloud}
ring: "Device: ring buffer of\nrecent raw sensor data" {class: dev}
ledger: "Cloud: verdict ledger\nscore, alert, outcome, dates" {class: ledger}
train: "Cloud: training set" {class: cloud}
det -> fleet: "1. scores + events"
det -> ledger: "2. every prediction"
ticket -> ring: "3. request the window"
ring -> train: "4. raw window"
ledger -> train: "5. matured labels"

Scores and events go up continuously, and they are small. Raw windows go up only when a maintenance ticket or a failure asks for one, which is rare in practice. In the cloud, the verdict ledger is the asset the whole design protects. It keeps every prediction with its score and whether it alerted, and it records the date the outcome became known. For a missed fault, the window that matters is the onset, weeks before the failure, so the buffer has to cover the miss window, not the ticket response time. A device that failed in the field may also be dead, so anything it did not upload earlier is gone.

Over-the-air model updates for IoT fleets are firmware rollouts with two clocks

Nordic Semiconductor's Edge AI documentation states that the model is built into the application binary, so changing the model means shipping a full application update. On many microcontroller stacks, Nordic's included, the model ships inside the firmware, which puts every model change on the same path as firmware, with the same risk profile.

CrowdStrike's Channel File 291 is the public reminder of that risk. On 19 July 2024 a Rapid Response Content update, the content type CrowdStrike describes as "not code or a kernel driver," went to Windows hosts at 04:09 UTC and was reverted at 05:27 UTC. Microsoft estimated that it affected 8.5 million Windows devices. CrowdStrike's remediation committed to canary deployment with successive promotion to larger rings. Model files are the same kind of artifact: data that changes behavior, pushed to a heterogeneous fleet.

AWS and Azure both ship the standard defenses. AWS IoT Jobs can abort a rollout on a failure threshold, for example cancelling once a set percentage of executions have failed after a minimum number have run. Azure IoT Edge selects deployments by target conditions on device twin tags and resolves overlaps by priority, so a cohort per hardware revision is a few lines of configuration.

Chapter 13 of my book Right-Sized AI, "Fleet Operations and On-Device Lifecycle," works out the parts this section skips: how long a release takes to cross a shared uplink, why delta updates decide whether a rollback finishes in time to matter, and why canary cohorts should be chosen by hardware revision and climate instead of at random.

d2
grid-rows: 4
grid-columns: 2
grid-gap: 22
classes: {
  wave: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"; font-size: 17}}
  health: {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"; font-size: 16}}
  acc: {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"; font-size: 17}}
  accnote: {style: {fill: "#9B59B6"; stroke: "#7D3C98"; font-color: "#FFFFFF"; font-size: 16}}
}
w1: "Wave 1: small cohort,\nevery hardware revision" {class: wave}
g1: "Health gate, days:\ncrashes, input drift, alert rate" {class: health}
w2: "Wave 2: larger cohort" {class: wave}
g2: "Health gate, days:\nsame signals" {class: health}
w3: "Wave 3: rest of fleet" {class: wave}
g3: "Health gate, days:\nsame signals" {class: health}
a: "Accuracy gate" {class: acc}
an: "After the miss window, months:\nrecall on matured predictions" {class: accnote}
w1 -> w2
w2 -> w3
w3 -> a

Waves on the left run in days, and the right column shows what each step can actually check. Every signal a health gate watches has a fast verdict, so those gates can be strict and automatic. Recall does not. The accuracy verdict on a new model arrives months after the rollout finished, on predictions that matured under it. A staged rollout plan that has only health gates has decided, without saying so, that the new model's recall is whatever the old one's was.

Using an LLM to explain IoT anomaly alerts: keep it behind the detector

A 2026 conference paper by Gyawali, Singh Bhandari and Al-Absi built a version with the right shape: an Isolation Forest detects, Llama 3.2 3B explains, and ESP32 sensors report over Message Queuing Telemetry Transport (MQTT) to a laptop acting as the fog node. They report precision of 0.97 and recall of 0.60, from a proceedings venue and a small testbed. Detection stays with the cheap model. A cheap detector raises the event, and a small language model on a gateway or fog node turns it into an alert a technician can read, so the explanation layer touches only the alerted cells: it speeds the false-alarm verdict and may help a repair record its fault.

d2
direction: down
classes: {
  dev: {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"; font-size: 17}}
  slm: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"; font-size: 17}}
  ledger: {style: {fill: "#9B59B6"; stroke: "#7D3C98"; font-color: "#FFFFFF"; font-size: 17}}
  miss: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; font-size: 17}}
}
det: "Detector on the device" {class: dev}
gw: "Small language model\non the gateway: writes the alert" {class: slm}
tech: "Technician" {shape: person; style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
ledger: "Verdict ledger:\nalert, explanation, outcome" {class: ledger}
miss: "Missed fault:\nno event, so nothing to explain" {class: miss}
det -> gw: "1. event + context window"
gw -> tech: "2. alert with explanation"
tech -> ledger: "3. inspection outcome"
det -> miss: "silent"

Look at the silent branch. An explainer only ever sees events, so a recall of 0.60 means 40% of anomalies arrive with no explanation because they arrive with no alert. Explanations improve the green cell of the verdict-skew matrix, the fast one: a technician who reads why the alert fired finds the false alarm sooner. They do nothing for the red cell. An explanation is also a new claim, and it needs its own column in the verdict ledger so a wrong explanation is caught as quickly as a wrong alert.

Managed IoT and industrial ML services retire faster than the fleets they serve

Months-long miss windows mean a verdict ledger gets more valuable with every year it is kept, so it has to outlive the service it was collected through. Several managed services in this space have not lasted many years. Since 2023 the record is short and specific.

d2
grid-rows: 5
grid-columns: 2
grid-gap: 10
classes: {
  date: {style: {fill: "#95A5A6"; stroke: "#6F7E7F"; font-color: "#2C2C2A"; font-size: 17; bold: true}}
  ended: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; font-size: 17}}
  closing: {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"; font-size: 17}}
}
d1: "2023-08-16" {class: date}
e1: "Google Cloud IoT Core retired" {class: ended}
d2: "2024-07-07" {class: date}
e2: "Azure Time Series Insights retired" {class: ended}
d3: "2024-10-31" {class: date}
e3: "Amazon Monitron closed to new customers" {class: closing}
d4: "2025-12-15" {class: date}
e4: "AWS IoT Analytics end of support" {class: ended}
d5: "2026-10-07" {class: date}
e5: "Amazon Lookout for Equipment discontinued" {class: ended}

Two of those five, Monitron and Lookout for Equipment, were managed machine learning services built for industrial sensor data. AWS pointed Lookout for Equipment customers to IoT SiteWise before support ended, whatever the quality of their models, and Monitron customers now run on a service that takes no new sign-ups. Own the verdict ledger in storage you control and keep the model in a portable format such as Open Neural Network Exchange (ONNX). Treat the managed service as replaceable plumbing around both.

When delayed labels do not limit anomaly detector tuning

Solozobov's 2026 study of delayed ground truth is the strongest evidence against this argument, and it narrows the argument to iteration on accuracy. The study found that proxy signals detected covariate and mixed drift in every case tested, and missed concept drift where the inputs did not change. If your failures announce themselves through input drift, you can iterate faster than your verdicts arrive.

Maintenance events such as a bearing replacement are known change points, and here time-series foundation models have a real role: Uray and colleagues found zero-shot TimesFM unsuitable for multivariate anomaly detection but promising for change-point detection. Fleets with run-to-failure test rigs, or with seeded faults injected on purpose, can manufacture fast verdicts for the red cell. Visual inspection, where a person can label a sample in seconds, has much less skew. And where a fault really is invisible to small models, the architecture choice matters again. In the drilling study, Chronos-2 kept its lead on hookload even where the small model caught up on torque.

A verdict-skew checklist for an IoT ML fleet

  • Measure verification latency per confusion-matrix cell, and define when a prediction counts as matured. Write the miss window down as the p95 time from a missed fault's onset to its field failure. Get onsets by tracing stored scores back from each field failure to where they first crossed a lower reference threshold, or from run-to-failure rigs. A p95 window leaves about 5% of misses unmatured, so matured recall reads slightly high.
  • Keep a verdict ledger in storage you own. Each row is one prediction: its score, whether it alerted, any explanation, the outcome, and the date that verdict arrived.
  • Measure the record rate: tear down a sample of parts replaced after true alarms and count how many show the fault. Assume it is 1.0 and a matured-recall estimate is biased low.
  • Never judge a raise or a model change on resolved alerts alone. Replay it on matured predictions, and set min_faults so the recall estimate's 95% interval (a Wilson interval, or a bootstrap over the scaled counts) is narrower than its distance to the floor.
  • Send scores with every event, and continuous low-rate scores if you want to replay lowered thresholds or trace onsets. Size the raw-data buffer to the miss window, or keep a long low-rate buffer beside a short high-rate one, and upload summaries on a schedule so a device that dies in the field does not take its history with it.
  • Work out your own speed limit before building: weeks to the first recall verdict is roughly the miss window plus min_faults divided by matured faults per week.
  • Set the recall floor from the cost of a missed field failure against the cost of an inspection, and write the arithmetic down.
  • Let a lowering ship behind health gates with an alert budget, at most one step per miss window unless the budget has room: its cost shows fast, but its benefit has the same lag as any recall verdict.
  • Give each rollout two kinds of gate: health gates that run in days, and an accuracy gate that runs after the miss window.
  • Stage rollouts by hardware revision and site conditions, with an abort threshold set before the release.
  • In design review, cap the model-table discussion at a fixed share of the time, and spend the rest on the items above.

Measure verdict skew before choosing a model

Most IoT ML design effort goes to the part of the system that is easiest to change. The part that decides the outcome is the calendar of the labels: false alarms confirmed in days, misses confirmed in months, repairs that often confirm nothing. Tune on whatever arrived this week and a fleet will drift toward silence and then swing into a flood, as the simulation did between weeks 1 and 36. Every weekly decision on the way was justified by the evidence that had arrived. A better-separating model would have fallen less far and might never have swung back, but it would still have lost recall in silence. So measure verdict skew first, keep the ledger that records it, and gate every accuracy change on predictions old enough to have been proven wrong. The same trap catches systems that cannot tell when they are wrong in other domains.

Further reading: edge and fleet operations in Right-Sized AI

The fleet and edge material here is model-agnostic, and my book Right-Sized AI: Small Language Models in Production covers it for the language-model tier:

  • Chapter 10, "Composition: Routing, Cascades, and Speculative Decoding": when escalating from a small model to a larger one stops saving money, and why that escalation rate has to be monitored after launch.
  • Chapter 12, "Edge and Embedded Deployment": sustained against peak throughput on edge hardware, energy per inference, and planning for devices that run unattended.
  • Chapter 13, "Fleet Operations and On-Device Lifecycle": rollout duration, delta updates, staged rollout and the version tail.
  • Chapter 14, "End-to-End Case Studies": a related fleet of 1,200 factory gateways with no inbound network path, each summarizing alarm and log streams with a small language model, sized on sustained throughput in a sealed enclosure at the worst site temperature.

References

AI Engineering

More Articles

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

Books by Ranjan Kumar

Federated AI Agents cover

Federated AI Agents

Explore the Book
Right-Sized AI cover

Right-Sized AI

Explore the Book
Harness Engineering for Production AI Systems cover

Harness Engineering

Explore the Book
The 7 GenAI Architectures cover

The 7 GenAI Architectures

Explore the Book
Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

Explore the Book
The ChatML Handbook cover

The ChatML Handbook

Explore the Book
The Chat Templates Handbook cover

The Chat Templates Handbook

Explore the Book

Comments