I wanted to talk about my experience with Mender to provide full OS Over-the-air updates to embedded linux devices. I’ve been using Raspberry Pi because well its the Raspberry Pi and has great support and docs.

Over-the-air (OTA) updates for the full operating system to headless embedded linux devices can be challenging. Things like apt-get/yum are problematic because of potential unknown state and failures that could easily brick a device in the field. A common and well tested technique is using an A/B rootfs partition. Where you deploy a new copy of the full OS after testing then restart the device and only after some success condition mark it as a success, otherwise auto reboot and go back to the previous state. This is easier said then done.

Mender makes this problem a delight. First off I have loved using the product (though limited use) and really appreciate the hard work they put into documentation, open source, and developer experience. They Northen Tech seem to be a great company with great developers, though haven’t personally interacted with any.

Because I had been playing with Yocto previously (blog post ) I thought I try Mender integrated with Yocto. Of course they have great docs and support for that build system based distro.

I followed their board support documentation here . Though note I used `kirkstone` branch which did work. `export BRANCH=“zeus”`

From there things just worked. Building the image with:

MACHINE=raspberrypi4-64 bitbake core-image-base

It outputted the correct `sdimg` files for flashing the microsd as well as the `.mender` files for uploading to Mender.io for a new update over-the-air.

Few issues/snags I ran into:

Rootfs Size

The default sizing of the SD card assumes 1GB but resizes the /data partition after booting. Great. Mender does some math under the hood and calculates how big each A/B rootfs and the /boot partition need to be. But after trying to add Tailscale to it and a few other packages it started to complain.

Ok I thought I would increase the media size with `MENDER_STORAGE_TOTAL_SIZE_MB` since I was using a 32GB sdcard. But soon after I tired that and uploaded the `.mender` image and deployed it my device complained about running out of space and failed the update. Makes sense since the new rootfs would be so much bigger. I didn’t try to deal with updating the partion sizes on the running device and just flashed a new sdcard. Something to keep in mind.

Pi Reboots

For some reason the Raspberry Pi 4 with the yocto image i’m using does not like a software reboot with `reboot`. It never succeeds in coming back up and requires a power cycle. Since mender does this after flashing an update it always gets stuck.

MENDER_ARTIFACT_NAME

This `MENDER_ARTIFACT_NAME` env variable in Yocto build names the artifact and when uploading to Mender this is what is set in their system. It can be annoying to remember to update this to a unique each time.

Overall it was a delight experimenting with Mender. I also appreciate their commitment to Open Source. They open source most of their control plane, though looks like a bit of pain to setup, this allows you to run your own Mender. Also they support a standalone mode which allows you to build your own system to deploying images to the devices, managing them, etc… but handles the actual A/B partition side of things. This is great if you already have a system to control and manage you devices but want to incorporate the OTA into it.

That said I think their pricing is fair and doesn’t gauge you for large number of devices like i’ve seen many others do. IoT is meant to be a large scale.

I still plan to check out swupdate a similar opensource product closer to their standalone mode.

Go check them out! Mender.io