A lower firmware risk estimate still needs a release boundary

An improved estimate can still fail the release policy

A release label describes intent. A pre-flight needs to examine the proposed behavior against the fleet that will receive it. In this demo, battery condition, radio recovery and flash wear influence modeled failures, so an average improvement alone cannot answer whether a candidate meets the declared risk boundary.

The worked case uses a generated fleet labelled Plano Water and synthetic firmware manifests. The initial battery-optimization estimate predicts 63,883 failures among 86,078 scored endpoints. An inrush-capped hotfix reduces that estimate to 2,449, but its upper modeled rate remains 4.10%. Both receive NO-GO under the same 3.0% upper-risk rule.

The distinction matters when inspecting pre-flight evidence: ask which population was scored, what the uncertainty includes and which rule permits the release. A favorable comparison with an earlier candidate answers only one of those questions.

Follow the hotfix from estimated improvement to release boundary

All fleets, manifests, version labels and records shown here are synthetic. These actual captures use cached model-assisted profiles, 200 Monte Carlo iterations and seed 1234. A modeled recommendation is not an executed firmware release or independent evidence of field safety.

Worked example: a much lower estimate, the same NO-GO

Start with the generated Plano Water population of 88,000 endpoints. The snapshot scores 86,078 and excludes 1,922 with insufficient telemetry. We compare an initial battery-optimization manifest with an inrush-capped hotfix against that same population and policy, so the improvement and the remaining release boundary can be inspected separately.

1. Inspect the behavior estimate before trusting the release label

The initial cached profile estimates 120 mA baseline modem current plus 100 mA extra current during a flash write. Its recovery probability after reset is 0.02. The hotfix profile changes these inputs to 100 mA baseline, 10 mA extra current and 0.96 recovery probability. These are changelog-derived estimates, not current measured on hardware or analysis of a firmware binary.

The generated population has median battery charge 69.0% and median age 4.4 years. In the assumed voltage-sag model, a flash write can cause a reset when terminal voltage drops below 3.30 V; failed radio recovery and flash corruption contribute to modeled failures. A high confidence token does not establish calibrated certainty about those inputs.

2. Compare the upper interval with the declared boundary

The initial candidate predicts 63,883 failures, or 74.22% of scored endpoints. The hotfix lowers the prediction to 2,449, or 2.85%. That is a large modeled improvement, but the gate checks the upper interval first: 3,531 divided by 86,078 is about 4.10%, still above the configured 3.0% hard block. It therefore returns NO-GO, even though the mean is below 3.0%.

The 90% modeled count interval runs from 1,536 to 3,531 for the hotfix. It describes the spread of simulated outcomes under these inputs, not a guaranteed field range. The practical review distinction is between “better than the earlier candidate” and “inside the declared release boundary”; this example satisfies only the first.

3. Keep the behavior profile, change the population

The same hotfix behavioral estimate produces GO on the generated Hill Country Electric Co-op population. Its median battery charge is 83.5%, median age is 2.8 years and weak radio signal accounts for 2.3%, compared with Plano's 69.0%, 4.4 years and 13.4%. The comparison shows why a firmware estimate cannot be separated from the condition of the population being scored.

GO does not cover the 1,778 excluded co-op endpoints, authorize an OTA job or demonstrate that the same image is compatible with different vendors' hardware. We are comparing estimated write behavior across generated health distributions, not deploying an image across manufacturers.

4. A small estimate cannot replace adequate firmware evidence

The thin-manifest case on the co-op population predicts only 54 failures, with a 39 to 75 count interval and a 0.06% upper rate. Its profile confidence is low, so the second policy branch prevents GO and recommends STAGED-CANARY. This is a different profile: recovery probability is 0.50 and write amplification is 0.008, rather than the hotfix's 0.96 and 0.004.

The non-GO recommendation proposes a 500-endpoint lowest-modeled-risk cohort, a 72-hour hold and fresh observed telemetry before widening; the configured observed failure-rate condition is below 0.1%. The app does not execute that canary. Nor does a healthy selected cohort establish that the degraded or excluded population is safe.

5. Preserve the exclusions and the basis of the decision

The initial-case HTML record below shows how the recommendation retains the synthetic snapshot, cohort breakdown, exclusions and advisory memo. The 1,922 missing-telemetry endpoints remain outside the prediction denominator and recommended for manual review. Exporting the record does not complete that review or silently count them as healthy.

The export also retains profile inputs, prediction and interval, policy thresholds, seed and timestamp. Its identifier is a truncated SHA-256 content hash; the operator signer is pending and the firmware hash is a synthetic placeholder. These fields make the generated decision inspectable, but do not make it a signed approval, compliance certificate or immutable audit archive.

Read the benchmark without hiding the control that needs review

The completed screen combines three measurements from a fixed 120-scenario synthetic evaluation with two current-profile regression controls. Each evaluation scenario uses 20,000 fully observed generated endpoints and 80 simulator iterations, with seed 2026. Generated truth comes from the same assumed mechanism with fresh noise, not an independent utility dataset.

Danger is labelled as a generated failure rate above 1.0%; both NO-GO and STAGED-CANARY count as blocked. The evaluation records 74 true positives, 13 false positives, 33 true negatives and zero false negatives within this fixed synthetic run. Four passing checks and one requiring review expose an input-sensitive mismatch; they do not establish perfect accuracy, independent validation or universal prevention.

Questions before a firmware rollout