Linux and Zephyr, Sitting in a Tree

layout: true
name: inverse
class: center, middle, inverse
layout: true
.header[.floatleft[Linux and Zephyr, sitting in a tree].floatright[Christopher Biggs, .hdrlogo[Accelerando Consulting]]]
.footer[.floatleft[LCA2021, Jan 2021]]
---
layout: true
name: callout
class: center, middle, italic, bulletul
.header[.floatleft[Linux and Zephyr, sitting in a tree].floatright[Christopher Biggs, .hdrlogo[Accelerando Consulting]]]
.footer[.floatleft[LCA2021, Jan 2021]]

---
class: center, middle
template: inverse

# Linux and Zephyr, sitting in a tree (Literally!)

## A collection of hacks to aid developing for the Zephyr RTOS

.bottom.right[
Christopher Biggs, .logo[Accelerando Consulting]
<br>@unixbigot .logo[@accelerando_au]
]

???

G'day my name is Christopher and I'm here in sunny Online from my home port of Brisbane, where I run a consultancy and incubator called Accelerando which helps people build the right technolgy faster.

The internet of things is our speciality and we've found over the last 5 years that while every problem is different, we find ourselves reaching for the same tools over and over.

---

# Agenda

.vvtight[
* What and Why is Zephyr
* Getting started
* Working with hardware
* Voodoo and Voodonts
* Devops for Drones
* Field testing: Linux and Zephyr, sitting in a tree
]

???

.vvtight[
* Why Zephyr
* What is Zephyr
* Zephyr Highlights
* How you're supposed to do it
* How I do it
* Getting your dev together
* $1000 cables
* Starter hardware
* Adding a new board
* Adding a new driver
* Booting and Upgrading
* Debugging
* Sitting in a tree
]

???

After seeing presentations about Zephyr at a number of previous LCA events, about six months ago we added Zephyr to our toolbox, after looking at a number of options that occupy similar niches.

I'm not involved in the project, so this is going to be my outsider's perspective on why we chose it and what techniques we've developed to fit it into our processes.

---
layout: true
template: callout
.crumb[
# Why
]

---

# What (and Why) is Zephyr

## A quick discourse on tradeoffs

???

This isn't going to be an introduction to Zephyr, because project member Kate Stewart has done a number of great talks in recent years in that vein.     I'm going to touch on what I see as the highligts, and talk about some areas where you might get stuck.

---

# The Internet of Things

## From Chips to Clouds

* LCA keynotes continuing the streak of doing my setup for me
* Extend your vision and reach, with technology

???

I'd also like to give a massive thanks to Limor Fried and her keynote this morning.   As has been the case every time I've spoken at LCA, the keynote speakers do a fabulous job of introducing everyone to the key topics that the later speakers will go deeper on.

This time I hope you've all learned what a microcontroller is, and that there's a range of options from cheap and simple to still pretty cheap and complex.

To add to what Limor said, I want to talk about WHY we use technology, and in particular the so-called internet of things.   For me, it's because besides saving labour, technology lets us see further, and do more, extending our conscious sphere beyond our mere bodies.

---

# You have one job

* Provide a particular benefit
* -> Cooperate 
* -> Keep doing the job, and don't do anything bad

???

So what is it to be a thing on the internet.     Well, presumably you have a job to do.    Some reason why we built you.   But that's only one third of the big picture.    The other parts are cooperation and safety.

The promise of IoT is that from a number of small parts we get an overall picture that exceeds the sum of those parts.    But with ubiquity comes risks, and we need to be aware that the chance exists that our creations can be turned to the dark side.

---

# Complexity is the enemy of Reliability

???

One of the key ways we manage that risk is to avoid unneccesary complexity.

To demostrate what I mean, this is a virtual conference. We're all sitting in front of powerful computers.  Multitasking.

What's wrong?    Complexity and unpredictability.   And batteries, always with the batteries.

Encase your PC in concrete and just leave the mouse cable sticking out.

New game, upgrade video card.   New program, upgrade disk drive.

---

# The Internet is just a series of ~~tubes~~ tradeoffs

???

To go back to what Limor said this morning, there's a spectrum of complexity from single board computers that can run Linux just like any other general purpose computer, down to tiny little microcontrollers with no room for anything but your application code.    In the middle we have systems-on-a-chip like the ESP32 that's in your conference badge, the nrf51 that Limor spoke about, and a bunch of other chips that run at something near 100mhz, with tens or hundreds of k of ram, and a megabyte or four of flash memory.     Too small for Linux, in most cases.

* 8 bits, eg arduino around $5
* 32-bit SoC, under $10
* Small form-factor PCS (Pi), under $100
* General purpose PCs, around $500 and up

---

# Linux for IoT, Yes You Can

## All the best horror stories prove it

???

Some of these devices you CAN fit Linux on, like our home internet routers or many iot security cameras.    And oddly enough, that's where most of the horror stories about how the S in IoT stands for security come from.

It's not that Linux is by nature insecure, so much that developers can lose focus and forget to clean up after their development shortcuts.    So what if we made a small operating system that made it harder to screw up.

---

# GPOS drawbacks

* resource hungry
* maintenance heavy
* best-effort scheduling

???

Some other reasons why we might not want Linux, or Windows, or
Android, or MacOS on our embedded devices, is that they need a lot of
storage and battery capacity, you apply updates and clean out your
hard drive when it fills up, there's a tendency to pop up a full
screen dialog box asking you to leave an product review just as you
were in the middle of landing your spaceship, or giving a presentation.

---

# Real Time Operating systems

* Keep up with the real world
* Long-running
* Low resource usage

???

Taking those previous drawbacks in the opposite order, we invent the opposite kind of system.

It's monomaniacally focused on doing just a few things, it can be
expected to run for a long time without needing maintenance, and
hopefully it has a battery life measured in years, not hours.

---

# RTOS and RTOS-like systems

* Linux + RTLinux
* QNX and other industrial OSes
* Android
* FreeRTOS (running on your badge)
* Innumerable vendor-provided toolkits

???

So what's out there in that space.   You can cut down linux and add a real time scheduling underkernel, you can choose from a number of commercial real time operating systems that aren't too much unlike linux, you can use whatever proprietary code your vendor provides, or you can use another of the open real time oses like Mynewt or FreeRTOS which is running on your badge.

---

# ...and Zephyr

## Why another RTOS?

* Free
* Flexible
* Friendly

???

And then there's Zephyr.    I wouldn't say there's anything revolutionary about it, but we've found it to be a nice well rounded package, which ticks my boxes of free flexibile and friendly.

---

# What is Zephyr

.nolm[
* A simple real time OS, part of the linux foundation
* Similar to Mynewt, FreeRTOS, et.al.
* Portable; Avoid vendor lockin
* Wide support, easy to extend
* Helpful subystems to get started fast
* Good sensor support
* Signing, Encryption, Field updates
* Debug and interact over serial or SWD
]

???

Some of the other things that drew me to Zephyr were its association with the Linux foundation, its support for a wide number of chips which gives me the option to change silicon vendor if I need to, its pretty easy to add support for new boards without needing to wait for updates, its got a LOT of features and examples that cover common use cases, security and quality is a focus, and its a lot more convenient to debug and test than some other options.

---

# Hardware support

.nolm[
* Supports 200+ boards on 7 architectures, including your PC
* Supports the 32-bit Arduinos (ARM core)
* Supports ESP32 (eg. your LCA SwagBadge!)
* Comparable sensor support to Arduino
* Display support a little sparse
* Top-notch Bluetooth stack
* You don't have to use stinky vendor tools
* Avoid "Binary Blob Syndrome"
]

???

Let's look at that hardware support.   It runs on most of the major chip families in use for IoT, including the recent Arduino boards, which means its an excellent choice for those who find themselves outgrowing the arduino ecosystem.   It runs on the ESP32 processor that you'll find in your conference swag badge.   It supports a heckton of different sensors out of the box, and its pretty easy to add more.     I would say it doesn't support as many LCD screens as I'd like at the moment.

The bluetooth stack is good, and small.  It's not opinionated about tools, you can use whatever IDE you like, and there arent any black box vendor code blobs that you have to blindly trust.

---

# Software subsystems

???

Look I said I wasn't giving an introduction to zephyr so I'll cut this short and say if you need a particular protocol or interface its probably already there.

---

# Who's using it

.nolm[
* Adopted by Nordic as the OS for their NRF52 bluetooth and NRF91 cellular ranges
* Adafruit are onboard  (* waves at Limor *)
* Very active development
* Comprehensive (but not always detailed) documentation
]

???

I am pleased as punch that Nordic and other vendors have said you know what we're going to stop
foisting our proprietary stack on you and just ship zephyr.   Many of the major vendors are financially supporting the project. Adafruit are a silver zephyr foundation member.    The development process is open and releases are regular.   And the documentation is great.   What's not to like.

---
layout: true
template: callout
.crumb[
# Why
# Starting
]

---

# Getting Started

???

So how do you get started.

Just read the fine manual.

---

# I ❤️   docs.zephyrproject.org

## It's so good I only gave up twice!

???

Seriously, the getting started guide is really good.   Its so good I only gave up in frustration twice before I got it working.  No really that's way less than every other OS I've met.

---

# Quickstart on your Linux, Mac or Windows PC:

## Install

.hugecode[
```
pip3 install west
west init ~/zephyrproject && west update
pip3 install requirements.txt
# download and install the compiler(s) (separate instructions for Linux, Mac, Windows)
```]

## Build

???

So you want to start coding for Zephyr.  The compilers for different
chips are mostly derived from the GNU Compiler Collection, the build
system is Cmake, and there's a python wrapper that drives Cmake for
you.  Installing all that stuff is only 4 steps.

After that, building an application is one line to build it, and one
line to install it on your microcontroller.  I can't overstress how
refreshing this all is.

---

# Can you run Zephyr on a Raspberry Pi

## No, and anyway, you don't want to do that.

???

Oh, does it run on raspberry pi?    Well yes and no.   You can run your zephyr application as a process on your PC, but Zephyr isn't competing with Linux or other general purpose operating systems in that size range.

---

# Can you *Develop* for Zephyr on a Raspberry Pi

## Also, no

???

If you follow the standard installation instructions, you can't develop on a raspberry pi either.   The compilers aren't packaged by Zephyr for ARM Linux at this time.

---

# *FX: Record scratch*
## Yes (with this one* simple trick)

???

But you can do it, and I'm going to tell you how, and why I set out to do that in the first place.

---

# Zephyr dev on a Pi: Why

* save your PC's USB ports
* avoid having to install shady drivers on your PC
* upload code from anywhere
* continuous integration

???

First the why.  I've got a nice grunty macbook which compiles my zephyr OS in a few seconds, why do I want to do that on a less power system instead?

Most of the chips I work with are programmed over USB.  Working on a
mac I am already in dongle hell and I don't want more cables holding
me down.  Also, some of the USB serial port chips I encounter have
pretty sketchy drivers for mac and windows, though they seem to work
reliably from linux.  But the two big reasons are firstly that I can
set up a test rig where it needs to be, maybe outside near the steet,
next to a noisy machine, up on a roof or down a hole, and I want to
work from my nice air conditioned office.  Secondly, I'm trying to
bring embedded development out of the 1980s by using source control,
build pipelines, test suites and all the things that grown up
programmers use.

---

# Zephyr dev on Pi: How

* Disclaimer: this applies to ARM and ESP32 architectures
* Compiler for ARM is packaged packaged for ARM
* Compiler for ESP32 is (unofficially) packaged for ARM
* Python stuff is good on a pi
* Let's write python extensions in Rust!
* **BOM BOM** Rustc in 1GB of RAM is NOT. FUN.

???

So how, if it isn't officially supported, can we get zephyr onto a raspberry pi.

Firstly, I haven't tried this for every chip that zephyr supports, I mostly work with ARM processors and the ESP32, and this all applies to them.

Although Zephyr doesn't provide a package, the GCC port for ARM is provided as a binary by its project team, and the same is true for the ESP compilers.    Python runs pretty much anywhere as Limor showed this morning.

The one gotcha is that if you are going to do this the way I did, go
out and get a raspberry pi 4 with at least 4 gig of RAM.  There's one
of those python libraries that is implemented in Rust and the
compilation grinds a 1Gig pi to a complete standstill.  I'm sure you
could work around this but I am not a python guru.

---

# Zephyr dev on Pi: TL;DR

* Install Ubuntu, not Raspberry OS
* Get a SBC with 4G of Ram (because rustc)
* Use my SaltStack rules to one-step install everything

???

---
layout: true
template: callout
.crumb[
# Why
# Starting
# Hardware
]

---

# Working with Hardware

---

# Chips and boards

* Zephyr supports a bunch of different chip families
* A "board" is some chip alongside some memory and maybe peripherals
* Find your board under `zephyr/boards/<chip>/<boardname>`
* (Later: how to add a new board)

---

# Building in a nutshell

* `west build -b <boardname>`

will generate build/zephyr/zephyr.hex (**fast!**)

* `west flash`

will use whicher programming method is defined for your board

---

# How does zephyr.hex get onto your board

* USB "drag and drop"
* USB serial
* JTAG (the "standard" interface to microcontrollers)
* SWD (another "standard", stands for single* wire debug)

---

# ASIDE: Commercial microcontroller dev tools are awful

* Buy the official development kit for each chip (often $300+ ea)
* Buy a programming cable from a tool vendor ($20 - $1000)
* Buy a clone programming tool ($5)
* or, use an open source programmer

---

# Buy

* Segger J-Link - $60 educational, ~$1000 professional
* ST-Link v2 - $20
* Clones of ST-Link and J-Link $5

---

# Build

* Use OpenOCD software and the GPIO pins on a raspberry pi
* Use OpenOCD software and an ST-link or J-link clone
* Buy a BlackMagic debugger (or program its firmware onto a blank board)
* Put open source firmware onto an ST-Link or J-Link clone

---

# Programming from Pi with an ST-Link or J-link (or clone)

## Step one: Build on your PC and copy to pi

### `west build && scp build/zephyr/zephyr.hex mypi:`

(or just use your IDE's remote edit, and build directly on your Pi)

---

# Programming from Pi with an ST-Link or J-link (or clone)

## Step two: Program the compiled code onto your board

### `openocd -f interface/stlink-v2.cfg -f target/nrf52.cfg -c init -c "program zephyr.hex"`

## *or*

### `openocd -f interface/jlink.cfg -c "transport select swd" -f target/nrf52.cfg -c init -c "program zephyr.hex"`

---
layout: true
template: callout
.crumb[
# Why
# Starting
# Voodoo
]

---

# VooDos and VooDon'ts

## Things I've learned

---

# Device Tree

* PC's use "probe and pray" (or a voodoo mix of ACPI and PCI) to find hardware
* ARM invented "device tree" as a way to describe what is present
* DT describes what inputs and outputs the board has
* In Zephyr, DT acts as a hardware abstraction layer
* DT can be overlaid to describe attached sensors etc.

---

# Configuring a build

* Include only the bits of the OS you need
* `west build -t menuconfig` -- same as Linux's `make menuconfig` (but don't)
* `prj.conf` is your custom kernel settings (building on the base board)
* `nameofboard.conf` and `nameofboard.overlay` allow customisations per-board
* put custom boards and drivers inside your project tree if you want

---

# Practical configuration
* Lean heavily on the sample apps
* compare `build/zephyr/.config` before and after menuconfig, using diff
* save the key differences to `prj.conf`

.hugecode[
```
export BOARD=adafruit_feather_nrf52840
cd samples/hello_world
west build -p
cp build/zephyr/.config config.default
west build -t menuconfig
diff config.default build/zephyr/.config | grep '^> ' | cut -c3- >prj.conf
```
]
---

# So your board isn't supported

* There is good porting documentation
* I defined half a dozen new boards in the first couple of weeks
* Start from a "nearby" board, copy and modify
* You will grow to love Device Tree
* Generally you'll be defining what pins your serial, I2C, LEDs and Buttons use

---

# Learning from the samples

* Over 250 sample apps
* Sample apps also contribute to regression testing
* Easy to copy a sample outside of tree and use it as a seed

---

# Adding device drivers

* You can just directly tweak the hardware (eg I2C or GPIO)
* Well documented driver interface
* Pretty easy to add new drivers (again copy, change)
* Easy to use drivers that are outside of the zephyr tree

---

## What I did in the first few weeks of learning Zephyr

* Added support for some cheap chinese NRF5x modules
* Built my own experimentor's breakout board
* Implemented a tilt/magnetic sensor module
* Implemented a bluetooth range-direction scanner
* Written a device driver for an unsupported sensor
* Worked around lousy vendor toolkits

---
layout: true
template: callout
.crumb[
# Why
# Starting
# Voodoo
# Devops
]

---

# Devops for Drones

## Deploying and maintaining

---

# Tests

* Comprehensive test-suite
* Unit and integration tests
* Templates for your own tests

---

# Working with multiple boards (including custom)

* Set BOARD to the name of your board
* Use a separate BUILD_DIR per board
* Set BOARD_ROOT to be the zephyr tree or your app tree as appropriate

.hugecode[
```
function setboard {
	 export BOARD="$1";
	 export BUILD_DIR=$PWD/build/${BOARD}
	 if [ -d boards/*/${BOARD} ]
	 then
		export BOARD_ROOT=$PWD
	 else
		export BOARD_ROOT=$ZEPHYR_BASE
	 fi
}
```]

---

# Command shell

* Optional interactive command shell provides access to OS status
* Can exercise key OS actions
* Add test hooks for your own code
* Control logging

---

# Management protocol

* Zephyr shares MCUMGR protocol with Mynewt
* MCUMGR allows remote control over UART, Bluetooth or Debug port
* Modify configuration
* Manage logs
* Invoke shell commands
* Get memory and task stats
* Manage boot images (next slide)

---

# Bootloader and Device firmware upgrade

* uses MCUBOOT (also used by Mynewt et.al)
* multiple firmware slots for safe upgrades
* Supports signed images
* Test a new image once, before confirming it
* Fetch crash dumps

---

# Debugging

* Print to UART (like Arduino)
* Logging subsystem (multiple levels and subsystems, runtime configurable)
* UART-over-debug port (Segger RTT)
* GDB (GNU source-level debugger, breakpoints, stepping etc)
* Run native code on your workstation

---

# Bluetooth as your management interface

* Once you have working firmware on a board, update over Bluetooth
* Use a bluetooth-exporer app (Nordic nRF Connect, or Adafruit Bluefruit) to configure
* Create simple setup apps on phones with React Native
* Alas Apple is not behind web-bluetooth

---
layout: true
template: callout
.crumb[
# Why
# Starting
# Voodoo
# Devops
# Tree
]

---

# Linux and Zephyr, sitting in a tree

## Using a Linux SBC as an Ambulance

---

# I don't know, it worked in my lab

## What to do when field-testing your prototype

* If you have full IP connectivity, great
* If not, send a raspberry pi along as an "Ambulance"
* Allows you to view logs, use shell, upgrade firmware

---

# Communications with your Ambulance

* use the site's existing network
* supply your own wifi hotspot
* connect a USB 4g dongle (u-bend)
* fit an nBIoT hat

---

# Ambulance frustrations

* Carrier grade NAT is a pain
* Work around NAT with a VPN (OpenVPN, Zerotier)
* SaltStack to provision, monitor and manage ambulances
* SaltStack reverse-ssh stunt

---

# Case-study: Remote [REDACTED] Monitoring

* Battery powered device with Camera, 4G, and motion sensors
* It really does sit in a tree
* Detached miniature sensor device, communicates via bluetooth
* Fully sealed devices, no cables
* Program and configure via bluetooth
* Charge via wireless charging

---

# SEALED SECTION: CI build/test pipelines

* Repo-based CI systems allow auto-testing of commits
* Zephyr's POSIX mode can be used in cloud testing
* Gitlab is a good choice for implementing real-hardware test-farms
* Run gitlab worker on your Pi Ambulance to push test builds onto real devices
* See my LCA2017 miniconf talk "devops for dishwashers" for more detail

---
.fig30[
![](keep-calm.jpg)
]

# Recap

.nolm.vtight[
* RTOS is a point in the convenience-complexity tradeoff spectrum
* Free yourself from awful vendor tools
* Wide support, easy to extend
* Using a Raspberry Pi to compile and/or deploy frees you from being wired down
* Your Raspberry Pi is also a great field ambulance
]

---

### Related talks

- [http://christopher.biggs.id.au/#talks](http://christopher.biggs.id.au/#talks)

### Links

- [Saltstack rules on GitHub](http://github.com/unixbigot/kevin)

### Contact me

.vvtight[
- Email: .blue[christopher @ accelerando.com.au]
- Twitter: .blue[@unixbigot] 
- BNE IoT Meetup: .blue[@iotbne] or search on .blue[meetup.com]
- Accelerando Makerspace in Western Brisbane
- Accelerando Consulting - IoT, DevOps, Big Data - .blue[@accelerando_au] https://accelerando.com.au/
]