A lower firmware risk estimate is not release permission

A firmware hotfix can make a risk estimate dramatically better and still leave a release unacceptable under the declared policy. For an engineering team deciding whether to update a meter fleet, those are different judgments. The improvement tells you something about the candidate. Permission depends on the remaining risk, the population being assessed and the evidence behind the estimate.

I built MeterGuard around keeping those judgments visible. It is a firmware rollout pre-flight demo using synthetic meter populations and synthetic firmware manifests. It estimates behavior from a changelog, models that behavior against fleet health and issues a local recommendation. It does not send firmware to meters or block an actual update. The useful question is what this separation lets a release owner inspect, and what it still cannot establish.

Improvement and acceptability answer different questions

Consider the synthetic population labelled Plano Water. An initial candidate produces a modeled mean failure rate of 74.22% of scored endpoints. A hotfix produces 2.85%. Both results use cached model-assisted behavioral profiles, rather than measured behavior from firmware binaries. Within those assumptions, the hotfix is a substantial improvement. That comparison is worth preserving even when the final recommendation remains NO-GO.

The release gate first checks the upper end of the 90% modeled interval. At or above 3.0% of scored endpoints, it returns NO-GO. For the hotfix, the mean is 2,449 modeled failures among 86,078 scored endpoints, with an interval from 1,536 to 3,531. The upper rate is 4.10%, so the hard-block rule applies. Another 1,922 endpoints lack sufficient telemetry and are excluded from that prediction, with manual review recommended.

A mean-only reading would miss why this is a hard block. It would not make the candidate eligible for GO under the rest of the policy, either: GO requires a mean at or below 0.5%, after clearing the upper-bound and confidence checks. Intermediate numerical risk leads to STAGED-CANARY. The distinction matters because “better,” “eligible for a limited evidence-gathering step” and “within the GO rule” should not collapse into one reassuring label.

I prefer to keep the improvement visible without allowing it to renegotiate the boundary. If a team responds to a disappointing recommendation by relaxing the threshold, it has changed its acceptance policy. That may be a defensible decision in a particular setting, but it is a separate decision requiring its own reasons. The evidence that one candidate is better than another does not supply those reasons by itself.

There is a cost to this position. A conservative boundary can delay a candidate that would have succeeded. The upper end of a modeled interval is not an observed field outcome, and calling the interval “90%” does not establish its coverage in a utility fleet. This demo cannot determine which threshold a real utility should adopt. What it can show is whether a recommendation follows the threshold that was declared, rather than a threshold quietly adjusted to fit the result.

Permission belongs to a candidate and a population

The same hotfix behavioral estimate gives a very different result against the generated population labelled Hill Country Electric Co-op. Its modeled mean failure rate is 0.02%, with an upper interval rate of 0.03%, and the gate returns GO for 118,222 scored endpoints. The population has a healthier battery distribution and less weak radio signal than the synthetic Plano population. Its 1,778 excluded endpoints remain outside that recommendation.

This is a comparison of estimated write behavior across generated health distributions. It says nothing about installing one manufacturer's image on another manufacturer's hardware. Compatibility and binary-derived validation are separate work that the demo does not perform.

The comparison changes how I want a release recommendation expressed. “This firmware is low risk” leaves the scope unstated. “This estimated behavior meets this policy on this scored population” preserves the conditions under which the result holds. Battery condition and radio recovery are inputs to the modeled outcome, so a favorable result cannot be detached from them and carried to another fleet.

For a release owner, this creates two different ways to respond to an unfavorable recommendation. One is to improve the candidate's behavior or the evidence used to estimate it. Another is to consider a narrower population whose conditions support a different assessment. They answer different problems. A narrower assessment may reduce modeled exposure, but it leaves the rest of the population unresolved. Better evidence about the candidate may improve the estimate, but cannot make missing fleet telemetry appear.

Those alternatives are prospective engineering choices, not operations this demo executes. Their value is that they direct work toward the source of the uncertainty. The verdict alone cannot tell a team whether it needs a better candidate, a better profile or better information about the intended recipients. The inputs and exclusions beside it can.

A code gate cannot validate what the model believes

MeterGuard keeps the policy in plain code. The model-assisted governance memo comes after the verdict and has no direct authority to override it. I want that separation because a fluent explanation should not quietly become a new release rule.

The model still has consequential influence earlier in the workflow. Its firmware profile estimates current draw, recovery after reset and flash-write behavior from synthetic changelog text. Those estimates feed the simulator. A different profile can change the modeled failures and therefore change the verdict, even if the gate code never changes. Inspectable policy establishes how inputs were judged; it does not establish that the inputs were correct.

The thin-changelog case makes the distinction visible. Against the same generated Co-op population, its modeled mean is only 0.05% and its upper rate is 0.06%. Those numbers clear the numerical boundaries. The profile nevertheless carries low confidence, so the gate recommends STAGED-CANARY instead of GO. Sparse firmware evidence is being treated as an independent reason to withhold the broader recommendation.

That is a useful guard, with a limit of its own. A model's confidence label is not calibrated empirical certainty. Requiring a non-low label can prevent one recognized evidence gap from being ignored; it cannot certify a “high” label as accurate. For production reliance, the profile would need validation against actual firmware behavior and outcomes. This remains work beyond the synthetic demonstration.

The safest sample leaves a harder question

The proposed staged route uses 500 endpoints from the lowest-modeled-risk cohort, with a 72-hour hold and re-evaluation using observed telemetry before widening. The demo recommends this plan; it has not executed a canary or collected its observations. The configured widening criterion is an observed failure rate below 0.1%.

I see a real tradeoff in choosing the safest cohort first. It reduces the exposure proposed for the initial step. But the same logic that made fleet condition matter also limits what that step could establish about endpoints in worse condition. In a hypothetical campaign, observing a successful update on healthy batteries and good radio connections would support a claim about that tested sample. It would leave open how the candidate behaves on aged batteries or weak radio connections.

There are at least two defensible responses to that gap. A team could keep widening restricted to populations sufficiently similar to the observed sample, accepting slower coverage and leaving degraded endpoints pending. Or it could seek evidence aimed at the degraded conditions, such as controlled validation of the relevant battery and recovery behavior, before considering those endpoints. The second path asks for more work; the first accepts a narrower conclusion. Neither makes a safe sample representative by declaration.

This is why I treat STAGED-CANARY as a request for specified evidence, not a softer synonym for GO. A staged plan needs to say what its observations will justify and where they stop. Without that scope, a cautious-looking process can still produce an overbroad conclusion.

Here is the founder walkthrough of these smart-meter release decisions in MeterGuard.

The MeterGuard explainer shows the pre-flight workflow and its decision records. My design position is to keep the estimated behavior, scored population, exclusions and declared rule beside the recommendation. For a release owner, the test is whether the next proposed step resolves the uncertainty that caused the hold. If it observes only the easiest conditions while the decision concerns harder ones, the boundary is still waiting for evidence.