The Automatic Metadata Extraction Pipeline

flowchart LR
     A["Try fwtool extraction"] -- Unsupported type or too large --> B["Stop no metadata"]
     A -- Fails, no trailer --> C["Use DTB result only"]
     A -- Succeeds --> D["Check DTB for a better model name"]
     D -- DTB has model --> E["Override model"]
     D -- DTB fails or no model --> F["Keep fwtool result"]
     E --> F
     F --> G["Return final result"]
     C --> G

     B:::hardstop
     C:::replace
     D:::enrich
     E:::enrich
     F:::enrich
     G:::enrich
    classDef hardstop fill:#f8d7da,stroke:#b02a37,color:#58151c
    classDef replace fill:#fff3cd,stroke:#997404,color:#664d03
    classDef enrich fill:#d1e7dd,stroke:#0f5132,color:#0f5132

Most OpenWrt sysupgrade images are built with fwtool, which appends a small trailer of JSON metadata to the end of the image file: the board identifier, the target, the firmware version, and the list of devices the image supports. Extraction looks for this trailer first, since it's the most direct and reliable source when it's present, no guessing or pattern-matching required, the image describes itself.

Not every image carries an fwtool trailer, and some are compressed in ways that make locating it unreliable. For these cases, extraction falls back to scanning the image for an embedded DTB, the same structure the Linux kernel uses at boot to describe the hardware it's running on. A DTB carries model and compatible properties that identify the board just as directly, just from a different part of the image. Even when fwtool succeeds, a DTB scan still runs afterward, since it can confirm or override the model with a more precise, human-readable name than the raw identifier fwtool provides on its own.

The Extraction State Machine

flowchart TB
    subgraph Statuses["extraction_status"]
        direction LR
        s1["unconfirmed"]
        s2["in_progress"]
        s3["failed"]
        s5["invalid"]
        s4["incomplete"]
        s6["success"]
        s7["manually_confirmed"]
    end

    subgraph Capabilities["What it unlocks"]
        direction LR
        c0["No functionality yet"]
        c3["Can be manually confirmed"]
        c1["Eligible for device pairing"]
        c2["Locked: fields uneditable"]
    end

    s1 --> c0
    s2 --> c0
    s3 --> c3
    s5 --> c3
    s4 --> c3
    s4 --> c1
    s6 --> c1
    s6 --> c2
    s7 --> c1
    s7 --> c2

    c0:::neutral
    c1:::unlock
    c2:::unlock
    c3:::unlock
    classDef unlock fill:#d1e7dd,stroke:#0f5132,color:#0f5132
    classDef neutral fill:#e2e3e5,stroke:#41464b,color:#41464b

Several new fields on FirmwareImage carry the result of extraction:

• board, the identifier used to pair the image with a device, replacing the old hardware-map lookup
• compatible, the fuller list of DTB-style identifiers an image supports
• target, the OpenWrt target platform the image was built for
• fw_version, the firmware version extracted from the image, shown whenever it differs from the build's own version
• compat_version, an internal compatibility marker that blocks device pairing outright when it exceeds 1.0, independent of extraction_status
• source, recording which method produced the metadata, fwtool, dtb, or manual, so an admin can tell at a glance how much to trust a given value
• extraction_status, tracking the image through the pipeline shown below, backed by a failure_reason and a full extraction_log for the details

Only success, manually confirmed, and incomplete images are eligible for device pairing. success and manually confirmed additionally lock the metadata fields from further edits, while failed, incomplete, and invalid images can be corrected by hand at any time. Rather than an image silently failing to pair with any device and leaving the admin to guess why, the status, reason, and log make the cause visible directly in the admin interface.

Safety Guards in the Extraction Pipeline

flowchart TB
    subgraph Reject["Reject"]
        direction LR
        A["Upload received"] --> B{"Bad file type?"}
        B -- Yes --> X["failed: unsupported type"]
    end

    subgraph SizeGuard["Size Limits"]
        direction LR
        C{"Raw file too large?"}
        C -- Yes --> Y["failed: decompression limit"]
        C -- No --> D["Decompress"]
        D -- Cap tripped --> Y
    end

    subgraph Scan["Scanning"]
        direction LR
        E["Scan fwtool trailer"] --> F["Fallback: DTB scan"]
        F --> G["Return result"]
        E -- Too large --> Y
    end

    B -- No --> C
    D -- OK --> E

    X:::fail
    Y:::fail
    G:::ok
    classDef fail fill:#f8d7da,stroke:#b02a37,color:#58151c
    classDef ok fill:#d1e7dd,stroke:#0f5132,color:#0f5132

Firmware uploads are untrusted binary input, so the extraction pipeline enforces limits at every stage, not just one. The raw file size is capped before it's even copied into a working file, decompressed size and compression ratio are capped during actual decompression to block decompression-bomb-style uploads, and a separate cap on the fwtool trailer's claimed metadata size stops an oversized JSON payload from being parsed even after it passes checksum validation. On top of the size limits, the number of trailer probes, CRC computations, and DTB scan attempts are all bounded too, so a file crafted with many fake trailers or DTB-like magic bytes can't force excessive CPU work by itself. Certain image types are rejected upfront by filename, before any parsing happens at all. Finally, the whole task carries its own hard time limit as a backstop, in case something still runs long despite every guard above.

Tripping the raw file size cap, the upfront filename rejection, the task's own time limit, or the decompression cap during a full DTB fallback marks the image failed with a specific reason in the log. The trailer size, probe, and DTB scan limits behave differently: hitting one of them just makes that step give up and fall through to whatever comes next in the pipeline, so if DTB still finds a board afterward, the image ends up incomplete rather than failed.

Recovering from Extraction Failures

flowchart TB
    subgraph Q["queue_unconfirmed_extractions"]
        direction LR
        Q1["Celery Beat (periodic)"]
        Q2["Worker startup (deduped)"]
        Q3["Find all unconfirmed images"]
        Q4["Queue extraction"]
        Q1 --> Q3
        Q2 --> Q3
        Q3 --> Q4
    end

    subgraph R["reclaim_stale_extractions"]
        direction LR
        R1["Celery Beat (periodic)"]
        R2["Find in_progress images past timeout"]
        R3["Mark failed: timeout"]
        R1 --> R2
        R2 --> R3
        R1 -.-> RC["Not scheduled? Django system check warns"]
    end

    Q4 --> Q5["Back into the pipeline:
in_progress"]
    R3 --> R4["Just another failed image:
can be confirmed manually"]

    Q4:::queued
    R3:::failed
    RC:::warn
    classDef queued fill:#d1e7dd,stroke:#0f5132,color:#0f5132
    classDef failed fill:#f8d7da,stroke:#b02a37,color:#58151c
    classDef warn fill:#fff3cd,stroke:#997404,color:#664d03

Extraction runs in the background via Celery, and the background workers can fail in ways ordinary exception handling can't catch, a worker killed by an out-of-memory condition, or one that hits Celery's hard time limit mid-extraction, both leave affected images stuck in progress indefinitely, with nothing coming back to retry it.

Two tasks handle recovery: reclaim_stale_extractions finds images stuck in progress past a configurable timeout and marks them failed so they can be manually re-extracted or corrected, and queue_unconfirmed_extractions picks up any image still sitting unconfirmed and queues it for extraction. The second one also runs automatically every time a worker starts, so images left unconfirmed after a deploy or a crash get requeued without anyone noticing, guarded by a short-lived cache lock so multiple workers restarting together don't all queue the same backlog at once.

Both tasks are idempotent and safe to run concurrently with themselves, so the recommended setup schedules them periodically via Celery Beat. reclaim_stale_extractions specifically is checked by a Django system check, since it has no other trigger, if it's missing from your CELERY_BEAT_SCHEDULE, you'll see a warning in your deployment logs. queue_unconfirmed_extractions isn't checked the same way, since the worker-startup trigger already covers it as a fallback. One more thing worth getting right: the stale-claim timeout must be set to at least the task's own time limit, otherwise the reaper can mark a still-running extraction as failed before it's actually had a chance to finish.

This isn't just cleanup: a build's mass upgrade is blocked entirely if even one of its images is still unconfirmed, in progress, failed, or invalid, so a single stuck extraction can hold up an entire rollout until it's recovered.

Each build also carries its own aggregate extraction status, rolled up from every image that belongs to it: if any image is still unconfirmed or in progress the build shows as analyzing, otherwise the worst outstanding state wins, invalid, then failed, then incomplete, then manually confirmed, and only once every image has fully resolved does the build itself show success. This gives an admin a single badge to check before attempting a mass upgrade, instead of opening every image individually to see whether it's ready.

Admin Workflow: Manual Confirmation and Bulk Re-extraction

When an extraction fails, or comes back incomplete, an admin can fill in the missing metadata by hand directly in the change form. If DTB was the only source (fwtool found nothing), board and compatible are already locked, only target and fw_version need filling in; if fwtool found a board that DTB never confirmed, board and compatible stay editable too. Saving confirms the image automatically, as long as board isn't empty, extraction status moves to 'Manually Confirmed' and the build's status updates to reflect it, without a separate confirm step. Leaving board empty shows a warning instead, and the image keeps its current status.

A bulk Re-extract metadata action is also available for images where the underlying file changed or extraction needs to be retried. It skips, and reports skipping, several categories of images so it can never undo a working state:

• images with an upgrade operation currently in progress
• images already locked (success or manually confirmed)
• images marked incomplete, since re-extraction wouldn't recover what's missing, they need manual input instead
• images already assigned to a device

Device Pairing via board

An image only becomes eligible for pairing once its extraction_status reaches success, manually confirmed or incomplete, images still unconfirmed, mid-extraction, or failed are excluded, so a device can never be paired to metadata that hasn't been verified. Pairing itself is an exact match: device.model == image.board. Once an image reaches success or manually confirmed, its metadata fields are locked and can no longer be edited, protecting a working pairing from being silently invalidated.

How it works:

• Make sure the extraction status of your firmware image is one of the following: Success, Manually Confirmed, or Incomplete.
• Go to Devices, create a device and fill in the required fields, set up credentials and then click on Save (make sure the Model field on the device page matches the Board field you see in the firmware inline)
• Go to the Firmware tab on the Device page and you will see the drop down populated with the firmware image

REST API Support

Extraction-derived fields (extraction_status, failure_reason, source, and similar) are always read-only through the API. board, compatible, target, and fw_version are read-only too, but only while the image's status keeps them locked in the admin, on creation, or while success/manually confirmed/unconfirmed/in progress. PATCH-ing those fields on a failed, invalid, or incomplete image writes them and confirms the image automatically, the same workflow available in the admin, so automation scripts and integrations can do it too, not just the browser.

Migrating Away from the Static Hardware Map

flowchart LR
    A["Built-in hardware map entries"] --> C{"Image: board empty & unconfirmed?"}
    B["Custom OPENWISP_CUSTOM_OPENWRT_IMAGES entries"] --> C
    C -- No --> Z["Left untouched"]
    C -- "Single board" --> D["Set board + source, status: success"]
    C -- "Multiple boards" --> E["Log all compatible boards, status: incomplete"]
    D --> F["Recompute build status immediately"]
    E --> F
    E --> G["Notify admin after migration completes"]

    D:::success
    E:::warn
    Z:::neutral
    classDef success fill:#d1e7dd,stroke:#0f5132,color:#0f5132
    classDef warn fill:#fff3cd,stroke:#997404,color:#664d03
    classDef neutral fill:#e2e3e5,stroke:#41464b,color:#41464b

A migration backfills board on every pre-existing firmware image it recognizes from the old hardware map. Most resolve to a single board automatically, but a small number of legacy images matched more than one board in the old map, since one file could serve several hardware variants, and for those the migration can't guess which one a given deployment actually has: they're marked Incomplete with a log entry listing every compatible board, so an admin can set the correct one manually. Any image the old map doesn't recognize at all is left untouched by this step; those get queued for real extraction once the migration finishes, going through the same fwtool/DTB pipeline as a fresh upload, so nothing is left permanently stuck unconfirmed with no attempt made to resolve it.