Persistent Mass Upgrades

Demo video coming soon.

Turning it on is a single checkbox on the confirmation page, ticked by default. When it is enabled, a device that does not answer is not written off. The operation drops into a pending state and a background task keeps coming back to it, waiting longer between tries each time so a device that is down is not being poked every minute.

Every operation now carries three extra pieces of state: the persistent flag itself, a running count of how many attempts it has taken, and the time its next try is due. In the admin you can filter the operations list to the ones still pending and read the persistent flag and retry count for each.

If OpenWISP Monitoring is installed, the retry does not have to wait for the next scan at all: when monitoring sees a device recover, the waiting upgrade goes to a worker at once. Without monitoring there is a fallback: a periodic task wakes pending upgrades on a randomized exponential backoff, so they are spread out rather than all firing at once.

You can cancel a persistent upgrade from the admin or the REST API: a pending operation stops retrying at once, and an in-progress one can still be cancelled up until firmware flashing begins — below about 65% progress. Once the flash is underway it runs to completion.

Put together, an upgrade operation moves through the states below. The loop between in-progress and pending is what persistence adds; the four terminal states are the ones the upgrader already had.

%%{init: {"theme": "base", "themeVariables": {
  "lineColor": "#8b949e", "nodeBorder": "#8b949e",
  "stateLabelColor": "#1f2933", "transitionLabelColor": "#1f2933",
  "edgeLabelBackground": "#f1f3f5", "labelBackgroundColor": "#f1f3f5",
  "noteBkgColor": "#fff4e6", "noteBorderColor": "#ed7800", "noteTextColor": "#1f2933"
}}}%%
stateDiagram-v2
direction LR
state "in-progress" as IP
state "pending" as PE
state "success" as OK
state "failed" as FA
state "aborted" as AB
state "cancelled" as CA
[*] --> IP: operation created
IP --> OK: flash completes, device back online
IP --> FA: cannot reconnect after reflash, or unexpected error
IP --> AB: prerequisites not met, or device deactivated
IP --> CA: operator cancels before flashing starts
IP --> PE: device unreachable and operation is persistent
PE --> IP: backoff elapsed (Celery Beat) or device healthy again (monitoring)
PE --> CA: operator cancels
PE --> AB: device deactivated while pending
note right of PE
  retry_count + 1, next_retry_at set with
  exponential backoff: 10 min, 20 min, 40 min ...
  capped at 12 h, with 25% random jitter
end note
classDef active fill:#ed7800,stroke:#b35b00,color:#ffffff,font-weight:bold
classDef waiting fill:#fff1e0,stroke:#ed7800,color:#1f2933,font-weight:bold
classDef success fill:#2f855a,stroke:#276749,color:#ffffff,font-weight:bold
classDef failure fill:#c53030,stroke:#9b2c2c,color:#ffffff,font-weight:bold
classDef stopped fill:#4a5568,stroke:#2d3748,color:#ffffff,font-weight:bold
class IP active
class PE waiting
class OK success
class FA failure
class AB stopped
class CA stopped

And if one keeps failing for a reason that needs a person, OpenWISP raises a notification, then repeats a reminder on a configurable cadence (about every two months by default) while the upgrade is still unresolved, so a forgotten device does not sit pending forever. The reminders in the screenshot below are closer together because the demo uses a shortened interval.

Pull Requests:

• Persistent Mass Upgrades #436
• Documentation and screenshots #448

Scheduled Mass Upgrades

Scheduling adds one optional field to the confirmation page: a date and time. Leave it blank and nothing changes, the upgrade runs immediately. Set it and the rollout waits until then.

You enter the time in your browser's timezone; it is stored in UTC and shown back to you with the zone named, so there is no doubt about what "2 a.m." meant. It is checked to fall inside a sensible window — by default at least ten minutes out and no more than six months away, both bounds configurable — so a slipped finger on the date field cannot quietly queue a rollout for next year.

A scheduled batch sits in its own scheduled status, and while it is there you can still edit the time or cancel it. Instead of handing Celery an eta for a task that might be months out (which is not reliable that far ahead), a Celery Beat task scans once a minute for upgrades that are due and launches them. Everything is re-checked at that point — the matching devices, their firmware, and any conflicting rollout — because plenty can change between scheduling a rollout and running it.

End to end, from the confirmation page to the completion notification, the pipeline looks like this:

%%{init: {"theme": "base", "themeVariables": {
  "lineColor": "#8b949e", "textColor": "#1f2933", "nodeTextColor": "#1f2933",
  "edgeLabelBackground": "#f1f3f5"
}}}%%
graph TD
A(["Operator confirms a mass upgrade
(admin or REST API)"]) --> Q{"Scheduled time set?"}
Q -->|"no"| NOW["Runs immediately:
batch goes idle to in-progress"]
Q -->|"yes"| V{"Time inside the allowed window
and no overlapping upgrade?"}
V -->|"no"| REJ(["Rejected with a validation error"])
V -->|"yes"| S["Batch saved as scheduled
(time stored in UTC; the schedule can still
be edited or the batch cancelled)"]
S -->|"operator cancels"| CAN(["Batch cancelled"])
S --> BEAT["Celery Beat runs execute_scheduled_upgrades
every minute"]
BEAT -->|"scheduled_at reached"| CHK{"Re-check just before launch:
eligible devices left and no conflicting batch?"}
CHK -->|"nothing eligible, or conflict"| FAIL(["Batch failed, no device touched
'not started' notification"])
CHK -->|"yes"| RUN["Batch goes scheduled to in-progress
'started' notification"]
RUN -.->|"launch never completed (worker died):
back to scheduled on the next scan"| S
RUN --> OPS["One upgrade operation per device;
offline devices go pending and are retried when persistent"]
NOW --> OPS
OPS --> DONE(["Batch ends success or failed
'completed' notification"])
classDef default fill:#f6f7f9,stroke:#8b949e,color:#1f2933
classDef active fill:#ed7800,stroke:#b35b00,color:#ffffff,font-weight:bold
classDef waiting fill:#fff1e0,stroke:#ed7800,color:#1f2933,font-weight:bold
classDef failure fill:#c53030,stroke:#9b2c2c,color:#ffffff,font-weight:bold
classDef stopped fill:#4a5568,stroke:#2d3748,color:#ffffff,font-weight:bold
class NOW,RUN active
class S waiting
class FAIL,REJ failure
class CAN stopped
linkStyle default stroke:#8b949e,stroke-width:1.5px

Operators get a notification when a scheduled upgrade starts and when it is done. And since scheduling sits on top of the persistence work, a scheduled upgrade that starts and hits a few offline devices just keeps retrying them, with nothing extra to configure.

The long horizon also means two rollouts can end up aimed at the same devices, so a new mass upgrade that overlaps an existing one is rejected rather than letting the two collide.

Pull Requests:

• Scheduled Mass Upgrades #460
• Documentation and screenshots #481