Privacy in the IoT Age

name: inverse
layout: true
class: center, middle, inverse, bulletul
.header[.floatleft[.teal[Christopher Biggs] — Solving the Pigeon Obesity Crisis].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[BSidesBrisbane2019] Jun 2019]]

---
name: callout
layout: true
class: center, middle, italic, bulletul
.header[.floatleft[.teal[Christopher Biggs] — Solving the Pigeon Obesity Crisis].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[BSidesBrisbane2019] Jun 2019]]

---
template: inverse

# Solving the Pigeon Obesity Crisis
### or, "How to put the S (for Security) in IoT"

.bottom.right[
Christopher Biggs, .logo[Accelerando Consulting]
<br>@unixbigot .logo[@accelerando_au]
]

<!-- Abstract

-->

---
layout: true
template: callout
.crumb[
# Welcome
]

---
class: vtight

# Who am I?
## Christopher Biggs — .teal[@unixbigot] —  .logo[@accelerando_au]

* Brisbane, Australia
* Convenor, Brisbane Internet Of Things Meetup
* Founder, .logo[Accelerando Consulting], a "full service" consultancy - chips to cloud
* Previously: Programmer, Architect, Developer Lead

???

G'day folks, I'm Christopher Biggs.  I've been involved off and on
with embedded systems since my first professional role over 20 years
ago, and nowadays I run a consultancy specialising in the Internet of
Things, which is the new name for embedded systems that don't work
properly.

We work with companies developing IoT devices to help them choose the
right technologies and practices to build test and deploy their
products.

---

# Everything is awful

.more[-- "The Internet of Scary Things" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)]

???
I could go on for hours about the security landscape in the internet
of things, and in fact I've spoken on the subject in the past.

The one-slide precis is that that bad people want to either steal your
stuff, or break your stuff.  It doesn't have to be many bad people,
another side effect of everything being connected to everything else
is that if you are a terrible person, you have the opportunity to be
terrible to vastly more victims.

---
.fig30[
![](dumpster_fire.jpg)]

.spacedown[
# Everything is awful
## and the awful is on fire]

.more[-- "The Internet of Scary Things" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)]

???

I'm not going to sugar coat this, the state of security in IoT, is
pretty awful.   The standard joke is that the S in IoT stands for
security.   But I'm not pessimistic.

---

# But we know what to do...

???

A few years ago I made some predictions about how bad things could
get, and what kinds of action would be needed, and the bad and good
news is that these are coming true.

I'm seeing the space rapidly mature and all the right things are being
done to address the challenges.

## ...because we've been here before

???

Fortunately we know what to do because we've been here before.   We
saw ludicrously bad security when minicomputers joined the internet in
the 1980s, factory default passwords, known but unpatched bugs.   We
saw it again in the 90s with windows 95 and then with wifi printers
and badly secured routers.

We kind of suck at learning from our mistakes.

---

# Cybersecurity improvement act 2019 (et.al.)
## Eight simple rules

???

We suck so much we're more incompetent than politicians.  Let that
sink in for a moment.

Some politicians got together and asked themselves what can we do to
stop these lazy programmers from ruining our surveillance panopticon
network dystopia?

In the United States, and the UK there is legislation in progress to
implement some basic security standards for IoT that, in violation of
all probability, are not wrong.  I mean it, these proposals are
an excellent first step.

Of course here in Australia, we perpetuated the stereotype of
everything being upside down by actually legislating for worse
security.  So thank you LNP for preserving my lack of faith in
humanity.

--
.vvtight[
* Patch
* Encrypt
* Push
* Passwords
* Disclose
* Update
* Repair
* Support
]
---
template: inverse

# Part 1: Verifications
* No known vulnerabilities
* Standard security protocols
* Secure remote updates
* No fixed or hardcoded credentials

???

Let's look at the the US proposal, since it covers everything that the
UK proposal does and a bit more.    I should mention that while I'm
told our aussie spy agencies have a research group looking at IoT security
practices, it's hard to see how this could turn into a standard here
since we already legislated pretty much the exact opposite.

The US proposal divides into two chunks.   The first chunk is called
Verifications, and these are practices that result in better devices.
The second chunk which we'll get to later are Behaviours, which are
more about effecting better support.

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
]

---
template: inverse

# V1. No known vulnerabilities

???

The first Verification rule is Don't Ship Known Bugs.   Sounds simple, right.

---

# V1: No known vulnerabilities

## Verify, update and curate your dependencies

???

Software today is basically made of dependencies.   A hello world
javascript app I created this morning contains 124kloc

---

# Don't

* Roll your own Linux distribution
* Pick a platform that only runs the vendor's custom OS build
* Use shell-scripts
* Connect stone-age tech to the internet without auditing it

???

Here's something that I've seen over and over.

A project starts with developers putting together a quick prototype by
starting from a blank OS installation, installing some tools and
editing the configuration.

Then it's six months later and there's a new version of the OS and
nobody remembers how to go from a clean OS to an app ready platform.

And thats how many embedded devices end up running eight year old kernels 
that are riddled with bugs.

PLC Exception OMG.

---

# Do

* Validate your supply chain (kudos npm, github)
* Choose a secure language
* Automate your builds
* Leave room for future updates
* Use "dependency version lock" techniques
* Actually review and update your dependencies often

???

For me a big part of not shipping bugs is making it easy to ship, and
easy to ship new dependencies.   That means a strong devops culture
and a fully automated build and test pipeline.   Its that testing
burden that makes organisations reluctant to update their dependencies.

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
]

---
template: inverse

# V2: Standard security protocols

???

Now this is the big item, or at least one one that might create the
most work.

---

# V2: Standard security protocols

## Hint: neither of the 'S's in SSL stand for "Sometimes"

???

Actually use security.   And don't invent your own, use bloody best
practices.

---

# Don't

* Leave security until "later"
* Set(up) and forget
* Bet the farm on one algorithm

???

The worst thing you can do is say "well, how hard could it possibly
be, we'll just use plaintext for now then drop a crypto library on
later".

Encryption is relatively easy, but key management is hard.

The next worse thing you can do is make everything secure but
provide no way to rotate or revoke keys.    You saved a lot of effort
now, but believe me when you have ten million devices in the field and
your root certificate expires, you will be a saaaad saaad panda.

And lastly, you have to plan for migration.   What's secure today may
be hopelessly broken tomorrow, so don't lock yourself in to one key
size or one algorithm.  Make sure it's actually possible to roll out
an upgrade to a billion devices without having to shut the internet
down for a month to do it.

---

# Do

* Use a standard crypto framework
* Plan ahead for re-keying
* Consider the worst case scenario

???

To put it bluntly, an IoT fleet is for life, not just for christmas.

---

# This is actually the hard bit of the whole talk.
## But also good for the pigeons

???

And this is where the news gets slightly better.

---

.fig70[
![](briefcase.jpg)]

.spaced[
# Symmetric and Assymetric cryptography
## Unbreakable crypto is real, but almost useless
]

???

This year marks a hundred years since unbreakable digital equipment
was patented.   The One Time Pad, used properly, is unbreakable.

Unfortunately using it properly is close to impossible.   While the
messages are safe, you need a random key the same size as the message,
shared by both parties and never reused.

This constraint leads to the familiar spy movie trope of a spy and
their controller meeting at a park bench to exchange documents.

---

.fig50[
![](spies.jpg)]

.spacedn[
# Assymmetric crypto makes key distribution easier
## But authentication is still hard
]

???

In the century since, security developments have largely focused on
reducing key size, and protocols for key distribution.

Public key cryptography was the big breakthrough about 40 years ago.
This shifted the challenge from encryption to authentication.

It's relatively easy to communicate securely, but it's still hard to
be certain of WHO you're talking to.   The risk is a Mallory in the
Middle attack, where an attacker impersonates your communicant,
communicates securely to you, and then re-encrypts your message with
her own key.

---

# You need to meet in "private" at least once
* **In the park**, pretending to feed the pigeons

???

The way we solve this is with trusted third parties, who certify
identities.   With cryptographically certified identies, I can be
certain that I'm talking to my bank, and not to Mallory who's sitting
at the end of the street with the two cut ends of my NBN cable plugged
into her laptop.

But you still need one way to establish that certification.

--
* or, **at the factory** (if there is no romance in your soul)

???

In the early days of the web, we expected this would involve
certification authorities who establish secure vaults and exercise
extreme care in certifying identities.

This rapidly turned into a shambles, of course.

At one end of the incompetence spectrum we have rogue CAs certifying
anyone with fifty bucks to spend, leading to malware being able to
successfully impersonate large companies like microsoft, google and
various banks.    At the other end we have CAs like LetsEncrypt who
have put huge amount of work into leveraging multi factor
authentication to allow remote certification.

So the state of the art in web certification right now is CA
certificates distributed inside the browser software, and identity
proven via multifactor challenges.

This solves some of our needs, but not all of them.  Most
authentication on the web is one way, and most certification still
requires some human interaction.

---

# Your "secure key distribution channel" is a cardboard box

.leftish[
1. Build and sign a root certificate
1. Upload root cert to the SaltStack master
1. Create minion certificate
1. Install minion certificate on minion
1. Upload a copy to the master
1. All automatically
]

(now ship it)

???

We need a tamper-proof channel to get a unique individual secret key
from the factory out to our device, so that we can guard against
identity theft.

Only our device was made in the factory.  So we put the key inside on
the assembly line, then ship the device.

They way I achieve this in my project is using SaltStack, which we
also use to customise and configure our initial base operating system
image.  During final assembly test we assign an identity to a device
and provide it with its security certificate.

---

# Build an automated CA with SaltStack

???

Audience participation : who does devops

Who uses: Orchestration

Ansible, Puppet, Saltstack

Saltstack bus topology, uphill connections

---

# First, make a root certificate

.smallcode[
```yaml
/etc/pki/{{pillar.realm.name}}-ca.key:
  x509.private_key_managed:
    - name: /etc/pki/{{pillar.realm.name}}-ca.key
    - bits: 4096
    - backup: True

/etc/pki/{{pillar.realm.name}}-ca.crt:
  x509.certificate_managed:
    - signing_private_key: /etc/pki/{{pillar.realm.name}}-ca.key
    - path: /etc/pki/{{pillar.realm.name}}-ca.crt
    - CN: {{pillar.realm.server_host}}
    - C: AU
    - ST: Queensland
    - L: Brisbane
    - O: BSidesDemo
    - OU: {{pillar.realm.name}}
    - basicConstraints: "critical CA:true"
    - keyUsage: "critical cRLSign, keyCertSign"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - days_valid: 7300
    - days_remaining: 300
    - backup: True
    - require:
      - file: /etc/pki
      - x509: /etc/pki/{{pillar.realm.name}}-ca.key
```
]

???
IRL more pillar

---

# Master makes the CA-cert available to minions

.smallcode[
```yaml
mine.send:
  module.run:
    - func: x509.get_pem_entries
    - kwargs:
        glob_path: /etc/pki/{{pillar.realm.name}}-ca.crt
    - onchanges:
      - x509: /etc/pki/{{pillar.realm.name}}-ca.crt
```]

# And minions can fetch the CA-cert from the master

.smallcode[
```yaml
{% set ca = pillar.realm.ca %}
{% set cacert = '/etc/pki/' + pillar.realm.name + '-ca.crt' %}

/etc/openvpn/{{pillar.openvpn.name}}-ca.crt:
  x509.pem_managed:
    - text: {{ salt['mine.get'](ca, 'x509.get_pem_entries')[ca][cacert]|replace('\n', '') }}

```]

???

Saltstack puns - when you publish something you send it to the salt
mine.  so ionic

---

## Master installs a policy for signing certificates
.smallcode[
```yaml
/etc/salt/minion.d/signing_policies.conf:
  file.managed:
    - source: salt://openvpn/signing_policies.conf
    - context:
      realm: {{pillar.realm.name}}
```]

.smallcode[
```yaml
x509_signing_policies:
  {{realm}}:
    - minions: '*'
    - signing_private_key: /etc/pki/{{realm}}-ca.key
    - signing_cert: /etc/pki/{{realm}}-ca.crt
    - C: AU
    - ST: Queensland
    - L: Brisbane
    - O: Fleetvalid
    - OU: {{realm}}
    - basicConstraints: "critical CA:false"
    - keyUsage: "critical digitalSignature,keyEncipherment"
    - extendedKeyUsage: "serverAuth,clientAuth"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - days_valid: 90
    - copypath: /etc/pki/issued_certs/
```]

???

The master doesnt need to trust minions unreservedly

---

# Minions generate own key,

.smallcode[
```yaml
/etc/pki/{{pillar.realm.name}}-{{grains.id}}.key:
  x509.private_key_managed:
    - bits: 4096
    - backup: True
  file.managed:
    - replace: False
    - mode: 600
```]

## and request signed certificate from Master.

.smallcode[```yaml
/etc/pki/{{pillar.realm.name}}-{{grains.id}}.crt:
  x509.certificate_managed:
    - ca_server: {{pillar.realm.ca}}
    - signing_policy: {{pillar.realm.name}}
    - public_key: /etc/pki/{{pillar.realm.name}}-{{grains.id}}.key
    - CN: {{grains.id}}
    - days_remaining: 30
    - backup: True
```]

???

Master keeps a copy of the issues certs

---

# Bonus feature
## Re-issuing is automatic once the expiry date looms

???

What about the CA key?   Make sure your options vest before then.

Seriously, we're good.

---

# Replicas and relays
## Breed them in the reactor

* Drawback: multi-tier masters don't quite work with the CA rules, but
* SaltStack's "reactor" defines rules that trigger on events
* Client registration and key rotation can trigger certificate replication

???

TODOs - getting rid of the hacks

---

layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
]

---
template: inverse

# V3: Secure remote updates

???

OK that's the technical summit of the talk.  I'm going to use the rest
of the time to pontificate wildly.

Our next verification action is to facilitate remote updates.

---
 
# V3: Secure remote updates

### When was the last time you had to manually check for updates on your phone?

???

Now we have trustworthy secure communications with a remote device,
that allows us to send digitally signed software updates that the
device will trust because they're signed with a certificate that it
has carried around since birth.

---

# Checking for updates

* HTTPS is fine
--

* DNS might work for you
--

* SaltStack to the rescue again

???

I'm going to feature creep the remote update mandate.   If we can
remotely update, we can remotely manage.   And again, saltstack to the
rescue here.

Now saltstack is a unix tool, and if you are using a five dollar IoT
device like this, you don't have unix.   But the techniques can still
apply.

Firstly, we can use a saltstack proxy device to install the keys in
our devices.   Secondly, we can imitate the command and control
structure we've developed over MQTT.   And thirdly there are IoT
platforms for embedded devices that feature auto-enrolment.

---

# Don't

* Require an app, or expensive equipment/personnel
* Wipe configuration or otherwise make upgrades suck
* Brick on failed update

---

# Do

* Notify when update required
* Think about fleets of thousands
* Consider a kill switch
* Evaluate offerings from AWS, GCloud, Azure

???

Shoutout - mongoose OS

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
]

---
template: inverse

# V4: No fixed or hardcoded credentials

---

# V4: No fixed or hardcoded credentials

## <s>admin / admin</s>

???

Do you want botnets, because thats how you get botnets.

---

# Don't

* Allow weak passwords
* Implement "back-doors"*
* Rely on security-by-obscurity

.footnote[Boo OzGov]

---

# Do

* Set a unique initial password for each device
* Force password change at first use
* Allow "factory reset" to erase secrets

---
template: inverse

# Part 2: Behaviours

* Mandatory disclosure
* Provide updates
* Repair or replace
* Committed support lifetime

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
# Disclose
]

---
template: inverse

# B1: Mandatory disclosure

---

# B1: Mandatory disclosure

## Be reachable.  Be open.   Be responsive.

---

# Don't

* Blame the messenger
* Deny everything
* Cover up the causes

---

# Do

* Provide a way to report vulnerabilities
* Plan for inevitable vulnerabilities
* Respond rapidly

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
# Disclose
# Update
]

---
template: inverse

# B2: Provide updates

---

# B2: Provide updates

## If you can't update it, it's not safe to sell it, or own it

---

# Do

* Make it easy to find updates
* Provide updates for all variants
* Have a plan for an un-fixable flaw

---

# Don't

* Break URLs for support pages
* Be mean to firewalls
* Use different names in different markets

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
# Disclose
# Update
# Repair
]

---
template: inverse

# B3: Repair or replace

---

# B3: Repair or replace

## Open hardware is maintainable hardware

---

# Do

* Make it easy to discover model and hardware revision
* Provide documentation
* Allow unwanted or untrusted subsystems to be disabled

---

# Don't

* Default to insecure
* Lock out open-source options
* Be at the mercy of third party services or code

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
# Disclose
# Update
# Repair
# Support
]

---
template: inverse

# B4: Committed support lifetime

---

# B4: Committed support lifetime
## with notification at End-of-Life

---

# Do

* Plan for the long haul
* Provide a support lifetime guarantee
* Alert users of support expiry

---

# Don't

* Create unnecessary dependencies
* Brick devices 
* Be Evil

---
layout: true
template: callout
.crumb[
# Welcome
# Patch
# Encrypt
# Push
# Passwords
# Disclose
# Update
# Repair
# Support
# Coda
]

---
.fig30[
![](keep-calm.jpg)
]

# Recap

.nolm.vtight[
* Start from a secure base
* Use standard best-practice cryptography
* Provide fleet management hooks
* No backdoors or default passwords
* Disclose flaws
* Provide updates
* Open hardware
* Committed support lifetime
]

---

# SHAMELESS SELF-PROMOTION

## **Brisbane Internet of Things Meetup**
* Last monday night of the month at .blue[@Avestix_Lab]
* Weekend practical workshops at .blue[@accelerando_au]
* Find us on Meetup.com

## **Hardware Startup Brisbane**
* .red[REBOOT BBQ AND WORKSHOP OPEN DAY **TOMORROW** (Sun 30th)]
* If you make stuff, we help each other make better stuff
* .blue[https://www.meetup.com/Hardware-Startup-Brisbane/]

---

# Resources, Questions

## Related talks - [http://christopher.biggs.id.au/#talks](http://christopher.biggs.id.au/#talks)

- Email: .blue[christopher @ biggs.id.au]
- Twitter: .blue[@unixbigot]
- BNE IoT Meetup: .blue[@iotbne]
- Accelerando Consulting - IoT, DevOps, Big Data - .blue[@accelerando_au] https://accelerando.com.au/