Code and Module Extension

During this process, we found that inheritance works well for extending Python code.

classDiagram
class OpenWISP_App
class STEER_App
class OpenWISP_App_TestCase
class STEER_App_TestCase
OpenWISP_App <|-- STEER_App
OpenWISP_App_TestCase <|-- STEER_App_TestCase

But several challenges emerged:

• Some tests contain hardcoded dependencies on OpenWISP apps that must be overridden
• The swapper tool requires many settings; we provide defaults loaded in the app's ready() method
• Certain settings must be overridden at import time, during Django initialization
• Duplicating URL configuration structures proved to be the clearest way to override views and routes
• Import order can affect Django initialization, leading to subtle and difficult issues
• Celery tasks are widely imported transitively, making overrides complex (though we have not needed this yet)

Having default settings makes juggling multiple Django apps much more manageable. It is a small and simple piece of code, but it makes extending OpenWISP a much more practical choice than it might seem at first. As an example, our defaults settings (for swapper or others) for openwisp_steer_users are defined in a dedicated file:

File: defaults.py

# Because we extend openwisp-users
# Setting models for swapper module
OPENWISP_USERS_GROUP_MODEL = "openwisp_steer_users.Group"
OPENWISP_USERS_ORGANIZATION_MODEL = "openwisp_steer_users.Organization"
OPENWISP_USERS_ORGANIZATIONUSER_MODEL = "openwisp_steer_users.OrganizationUser"
OPENWISP_USERS_ORGANIZATIONOWNER_MODEL = "openwisp_steer_users.OrganizationOwner"

These can then be loaded during django.setup() via an extended AppConfig.__init__() or AppConfig.ready(), depending on when the settings should be activated during setup time.

As is usually the case with complex Python code like Django, using the debugger is mandatory to understand what is actually happening when writing code. For instance, here is what worked for us with openwisp_steer_users:

File: app.py

class OpenwispExtensionUsersConfig(OpenwispUsersConfig):
    name = "openwisp_steer_users"
    label = "openwisp_steer_users"

    def __init__(self, app_name, app_module) -> None:
        super().__init__(app_name, app_module)

        from .defaults import (
            OPENWISP_USERS_GROUP_MODEL,
            OPENWISP_USERS_ORGANIZATION_MODEL,
            OPENWISP_USERS_ORGANIZATIONOWNER_MODEL,
            OPENWISP_USERS_ORGANIZATIONUSER_MODEL,
        )

        # Because we extend openwisp-users
        # Setting models for swapper module
        settings.OPENWISP_USERS_GROUP_MODEL = getattr(
            settings, "OPENWISP_USERS_GROUP_MODEL", OPENWISP_USERS_GROUP_MODEL
        )
        settings.OPENWISP_USERS_ORGANIZATION_MODEL = getattr(
            settings,
            "OPENWISP_USERS_ORGANIZATION_MODEL",
            OPENWISP_USERS_ORGANIZATION_MODEL,
        )
        settings.OPENWISP_USERS_ORGANIZATIONUSER_MODEL = getattr(
            settings,
            "OPENWISP_USERS_ORGANIZATIONUSER_MODEL",
            OPENWISP_USERS_ORGANIZATIONUSER_MODEL,
        )
        settings.OPENWISP_USERS_ORGANIZATIONOWNER_MODEL = getattr(
            settings,
            "OPENWISP_USERS_ORGANIZATIONOWNER_MODEL",
            OPENWISP_USERS_ORGANIZATIONOWNER_MODEL,
        )

The same pattern can be repeated for any django app extension you wish to write, provided there is no interaction with module import logic. But as always, when in doubt, trust your debugger.

Database Migration Strategy

Instead of generating new migrations and attempting to reconcile them with existing OpenWISP migrations, we adopted a different strategy:

stateDiagram-v2
direction LR
OW: OpenWISP DB (vN)
STEER: STEER DB (vN)
STEER_UP: STEER DB (vN+1)
OW --> STEER: Migrate OW to STEER (fake-apply duplicated migrations, remap ContentTypes)
STEER --> STEER_UP: Apply upstream OpenWISP migrations (via custom command for inconsistent states)
STEER --> STEER_UP: Apply custom STEER migrations

• Duplicate migrations from original OpenWISP modules, adjusting dependencies as needed

• Explicitly reference existing database tables and constraints

• Add custom changes as additional migrations with an offset for future upgrades

• Provide a management command to fake-apply migrations when equivalent OpenWISP migrations already exist

• Handle ContentType migration first by remapping original entries to extended apps while preserving foreign keys

• Ensure all migrations are reversible, allowing safe rollback and reapplication

• Develop a custom command to handle forward migration from inconsistent migration states by temporarily reverting and reapplying custom migrations

  Disclaimer: This is the approach we are currently using to manage deployments and automatic database upgrades. It might not be the best solution - or even a safe one. Do not blindly copy-paste or run code from the internet (or any AI). Use your own judgement before running any code. You are responsible for the code you execute, so make sure you understand it first.

Here is the command to fake-apply migrations when an equivalent OpenWISP migration has already been applied to the database

class Command(BaseCommand):
    help = "Mark migrations as applied for OpenWISP extended apps"

    def add_arguments(self, parser):
        parser.add_argument(
            "--database",
            default=DEFAULT_DB_ALIAS,
            help="Nominates a database to synchronize",
        )

    def handle(self, *args, **options):
        migrations = fake_migrations_registry.get_all()
        connection = connections[options["database"]]
        executor = MigrationExecutor(connection)
        state = executor.loader.project_state()

        def cmd_msg(app_label, migration_name, msg, indent=2, style=None):
            prefix = f"{' ' * indent}- {app_label}"
            if migration_name:
                prefix += f".{migration_name}"

            if style:
                self.stdout.write(style(f"{prefix}: {msg}"))
            else:
                self.stdout.write(f"{prefix}: {msg}")

        for original_app_label, (
            app_label,
            start,
            end,
            skip,
            skip_initial,
        ) in migrations.items():
            cmd_msg(app_label, None, "Faking migrations...", indent=0)

            for i in range(start, end + 1):
                if skip and i in skip:
                    continue
                migration_name = f"{i:04d}"
                try:
                    # Get the migration
                    migration = executor.loader.get_migration_by_prefix(
                        app_label, migration_name
                    )
                    if not migration:
                        continue

                except Exception as e:
                    cmd_msg(
                        app_label,
                        migration_name,
                        f"Error loading migration: {e}",
                        style=self.style.ERROR,
                    )
                    # Raise immediately, we cant even load the migration
                    raise CommandError(f"Error loading migration: {e}")

                # Check if migration is already applied
                if (
                    app_label,
                    migration.name,
                ) in executor.loader.applied_migrations:
                    cmd_msg(app_label, migration.name, "SKIPPED. Already applied.")
                    continue
                else:
                    # Check if original app migration is applied
                    if (
                        original_app_label,
                        migration.name,
                    ) not in executor.loader.applied_migrations:
                        err_msg = (
                            f"Original Migration "
                            f"{original_app_label}.{migration_name}"
                            " not applied on DB."
                        )
                        cmd_msg(
                            app_label,
                            migration.name,
                            f"ERROR. {err_msg}",
                            style=self.style.ERROR,
                        )

                        raise CommandError(
                            f"{err_msg}"
                            "\nYou might want to use `migrate` to apply "
                            f"{app_label}.{migration_name} instead."
                        )
                        # we need to stop applying migrations here, even fake ones
                        # continuing would write inconsistent history into the DB

                try:
                    # Only use fake_initial for initial migrations,
                    # otherwise use fake=True
                    if i == start and not skip_initial and migration.initial:
                        state = executor.apply_migration(
                            state,
                            migration,
                            fake=False,  # Don't use fake here,
                            # it prevents useful checks
                            fake_initial=True,
                        )

                        cmd_msg(
                            app_label,
                            migration.name,
                            "[initial] Faked successfully",
                        )

                    else:
                        state = executor.apply_migration(
                            state, migration, fake=True, fake_initial=False
                        )

                        cmd_msg(app_label, migration.name, "Faked successfully")

                except Exception as e:
                    # Raise immediately, we are applying migrations
                    # and we dont know what went wrong
                    raise CommandError(
                        f"Failed to fake migration {app_label}.{migration.name}. "
                        f"This usually means the tables from {original_app_label} "
                        f"app are not present. Are you running this on an existing "
                        f"OpenWISP 1.1.1 database?"
                        f"Error: {str(e)}"
                    )

        self.stdout.write(
            self.style.SUCCESS("\nSuccessfully processed all registered migrations")
        )

fake_migrations_registry is a registry listing the migrations that are duplicated from OpenWISP and, therefore, may have potentially already been applied to the database (depending on the version of OpenWISP being extended). This effectively allows you to deploy an extension on top of a database that was previously used with OpenWISP.

However, to make this usable, the content types and versions in the database need to be updated. To achieve this, the extension requires its own migrations. To prevent issues, these migrations should be reversible without breaking foreign keys. This way you can revert and reapply them without losing information.

Furthermore, when running ./manage.py migrate after updating the extension itself to follow OpenWISP changes, you might end up in an inconsistent state. This is because migrations (the openwisp duplicates) have been added earlier in the list of applied migrations. To resolve this, you will need to:

• First (fake-)unapply the extension specific migrations
• Then apply the openwisp-duplicated migrations to bring the DB up to date
• And finally (fake-)apply the extension specific migrations

Faking the apply/unapply process is fine when the extension specific migrations do not change. However, when your extension specific migrations need to change between versions, you'll have to actually unapply-reapply them - which is when their reversibility will come in handy.

Or you can always create another migration on top.

Specific Module Concerns

Some modules are more difficult to extend:

• openwisp-monitoring: initialization logic in database backends complicates partial overrides, especially for query handling
• openwisp-firmware-upgrader: modifying forms and extending controllers remains challenging, and not all tests pass in extended setups

We are currently collaborating with OpenWISP to make these simpler to extend and work with other customized OpenWISP modules, helping to maintain consistency between projects and ensuring tests can run with a wide variety of setups. It is this kind of boring but necessary work that keeps a project like OpenWISP meaningful for a lot of us.

Deployment and Git Workflow

We use a custom Ansible setup, partially based on the ansible-openwisp2 role, overriding tasks where necessary.

flowchart LR
UP[upstream OpenWISP]
subgraph StellarGit["Stellar Git"]
direction LR
OWDEV[master branch
vanilla OpenWISP extension]
STELLARDEV[dev branch
STEER customizations]
OWDEV -- periodic merges --> STELLARDEV
STELLARDEV -- cherrypicks --> OWDEV
end
CI[CI pipelines
OpenWISP-like QA]
ANS[Custom Ansible
based on ansible-openwisp2]
QA[STEER deployment environment for QA
GLOBBLE routers fleet]
UP -- release upgrade --> OWDEV
OWDEV -- upstream contributions --> UP
STELLARDEV --> CI
CI --> ANS --> QA

This approach allows us to:

• Keep our Django project configuration in source control
• Minimize runtime configuration variability
• Simplify deployments for our specific use case

Our Git workflow consists of:

• One branch tracking upstream OpenWISP changes
• One branch for internal development, regularly merged with upstream updates

We use CI pipelines similar to those in OpenWISP repositories, ensuring that any upstream contributions have already passed automated QA checks in our environment.

Organizational Choices

Given our limited resources and need for controlled customization:

flowchart LR
subgraph Development["1-Dev Flow"]
Develop[Develop
local LAN
NO_MANAGEMENT_IP]
UnitTests[Unit Tests]
LANTests[LAN Tests
manually per-package]
Develop --> UnitTests
UnitTests -- problems ? --> Develop
UnitTests --> LANTests
LANTests -- Issues found --> Develop
LANTests -- all good ? --> Release
end
Release[Release
tag & publish package
as frequently as needed]
InternalDeploy[Deploy
bump version in Django project
single pipeline, env vars / feature flags]
Maintain[Maintain
Goal: minimize maintenance overhead]
Release --> InternalDeploy --> Maintain
Maintain -- Issue ? --> Develop

• Each repository can run independently in a local LAN (using NO_MANAGEMENT_IP)
• Manual testing and releases are performed per package
• Deployments typically use the latest version of each package

Additional practices:

• Final Django project resides in a separate repository with recommended settings
• Limited customization via environment variables (controlled feature flags)
• Frequent releases preferred over deploying unreleased pipeline artifacts
• Single deployment pipeline with environment-based configuration
• Strong focus on minimizing maintenance overhead

Challenges and Lessons Learned

This project has been a valuable real-world experience in transitioning from a standard OpenWISP setup to a fully extended architecture.

While upstreaming changes required significant effort, it provides long-term benefits:

• Ensures licensing compliance
• Improves the ecosystem for all users
• Reduces long-term maintenance burden

We identified areas for improvement in OpenWISP:

• Not all modules fully support extensibility (e.g., firmware upgrader)
• Debugging can be difficult due to cross-module test dependencies
• Running tests across module boundaries remains challenging

Despite these challenges, this transition has been essential to maintaining the performance and reliability of our GLOBBLE routers.