name: inverse class: center, middle, inverse layout: true .header[.floatleft[.teal[Christopher Biggs] — Test Automation in IoT].floatright[.logo[@accelerando_au]]] .footer[.floatleft[.hashtag[121test] May 2018]] --- name: callout class: center, middle, italic layout: true .header[.floatleft[.teal[Christopher Biggs] — DevOps for Dishwashers].floatright[.logo[@accelerando_au]]] .footer[.floatleft[.hashtag[121test] May 2018]] --- layout: true .header[.floatleft[.teal[Christopher Biggs] — DevOps for Dishwashers].floatright[.logo[@accelerando_au]]] .footer[.floatleft[.hashtag[121test] May 2018]] --- template: inverse # Test automation in the Internet of Things ## or, "DevOps for Dishwashers" .bottom.right[ Christopher Biggs, .logo[Accelerando Consulting]
@unixbigot .logo[@accelerando_au] ] --- class: bulletsh4 # Who am I? ## Christopher Biggs — .teal[@unixbigot] — .logo[@accelerando_au] .leftish[ #### Brisbane, Australia #### Former developer, architect, development manager #### Founder, .logo[Accelerando Consulting] #### Full service consultancy - chips to cloud #### ***IoT, DevOps, Big Data*** ] ??? G'day folks, I'm Christopher Biggs. I've been involved off and on with embedded systems since my first professional role over 20 years ago, and nowadays I run a consultancy specialising in the Internet of Things, which is the new name for embedded systems that don't work properly. We work with companies developing IoT devices to help them choose the right technologies and practices to build test and deploy their products. Now I've spoken in the past about the state of security in IoT, and while I agree that it's pretty awful, I'm not pessimistic. I made some predictions about how bad things could get, and what kinds of action would be needed, and the bad and good news is that these are coming true. I'm seeing the space rapidly mature and all the right things are being done to address the challenges. --- layout: true template: callout .crumb[ # Agenda ## Problems ] --- # Why Devops? ??? Earlier this year I talked about the Internet of Scary Things, including here in Brisbane at the Internet of Things Meetup. That presentation is about how IoT is like the Wild West right now, and why DevOps is a must-do for IoT. Traditionally embedded systems have been developed carefully and conservatively, and had long lifecycles. In my experience you need those long lifecycles to cope with the generally awful tooling in the embedded space, which also apparently traditional. These days, the internet of things brings the contradictory requirement of products that are set in stone, sometimes quite literally, but which connect to the wild Internet where sometimes a day can be an age. So *this* talk is going to be about the *how*. How *you* can apply best practices from grown up computers to the challenges of building and managing internet connected embedded systems. --- # Why Dishwashers? ??? I've called this talk DevOps for Dishwashers because recently the tale of an internet connected dishwasher made the news. A lot of people laughed because why would a dishwasher need to be on the internet. It took about 10 years for us to go from personal computers being networked only at need, to us considering any computer that isn't networked all the time to be little more than a warm rock. And I expect the same for every other electric appliance. In the long term, I think the very word computer will become archaic, because everything will be a computer. --- # *"Software is eating the world"* ## .right[-- Mark Andreesen] ### Wall St Journal, Six years ago last month. ??? Software, as Mark Andreesen predicted, really truly will have eaten the world. This will have a lot of effects, but the one I want to focus on today is that we must rethink our concept of quality in the embedded space. The price of an amazing future everything can interact with everything else, is that everything might interact with everything else. --- layout: true template: callout .crumb[ # Agenda ## Problems ## DevOps? ] --- template: inverse # Interlude: What do I mean by "DevOps"? ??? Actually, before I go too far, I should define what I mean by DevOps. This is one of those terms that has been misused so much that it's now dangerous to use it without confirming that the listener hears what you mean. --- .fig40[ !(devops.png "DevOps is NOT THIS")] .spacedown[ ## DevOps is not a ***thing you do***, ## it's the ***way you do things***. ] ??? DevOps is not a thing you do. DevOps engineer is not a job title. There's no such thing as a DevOps team. It's an easy mistake to make, in fact I once held the job title Head of Devops. What DevOps **is**, is accepting that development and operations are part of a spectrum, and that drawing a sharp line between them is counterproductive. DevOps is letting go of your job title and accepting that everyone has the same job. Your job is Making the customer happy. --- .fig40[ !(devops.png "DevOps is NOT THIS")] .spacedown4[ ## *Empower* ***everyone*** ## *to maximise* ***value.*** **DevOps**: n. "*The practice of evolving a culture and a toolset that empowers everyone to effectively maximise value through radical transparency and extreme agility*." ] ??? My definition of DevOps is: Evolving a culture and a toolset that empowers everyone to effectively maximise value through radical transparency and extreme agility. --- layout: true template: callout .crumb[ # Agenda ## Problems ## DevOps? ## Solutions ] --- template: inverse # *"When every Thing is connected, **Everything** is connected"* ## .right[-- Me] ??? So today I'm going to look at how we pay the price of a connected future. The body of this presentation is about the practical things you can do to build quality products for the internet of things. I'll look at the lifecycle of an IoT product from inception to retirement, and what you can be doing to get the best outcomes at every stage. The three areas I'm going to look at are firstly choosing platforms that support rather than hinder quality outcomes, secondly shaping your development and quality practices to foster agility while preserving reliability, and thirdly working with your data in ways that promote interoperability and reusability. --- layout: true template: callout .crumb[ # Agenda # Landscape ] --- # Welcome to the Internet of Things ## pop. 10 Trillion ??? In just three score years and ten, the span of a single human lifetime, we saw the ratio of computers to people increase by around one order of magnitude per decade. We went from one per country, through one each for the fortune 500, one for every business, one for every person, to the present day with considerably more than one per person in industrialised world. In the next decade I expect another dectupling, and at some point after that the question will become meaningless. Everything will be a computer. Every. Thing. Hence, the Internet of Things, a world where humans are a trace impurity in a living network of machines. --- layout: true template: callout .crumb[ # Agenda # Landscape # Challenges ] --- .fig30[ !(switchboard.gif)] .spacedown[ # Solve the next problem, not the last one] ??? Back about a century ago, one pundit predicted that civilisation was about to grind to a halt, because trend analysis showed that very soon all available single women would be employed as telephone exchange operators, and thus no further rollout of the telephone network would be possible. If you wonder how we're going to curate and maintain a thousand devices per person, you're making the same kind of extrapolation error as our telephone operator pundit. --- # Strategy is adapting your behaviour to circumstances. ??? When a computer cost 3 months wages, you shepherded it carefully. When computers cost less than a cup of coffee, they become consumables, bought and sold by the bushel. They're sheep, not sheepdogs. --- class: bulletsh4 .crumb[ # Landscape # Challenges ## Risks ] # Everything is awful .bottom.right[ Do you want to know **more**?
"The Internet of Scary Things" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)] ??? I could go on for hours about the security landscape in the internet of things, and in fact I've spoken on the subject in the past. The one-slide precis is that that bad people want to either steal your stuff, or break your stuff. It doesn't have to be many bad people, another side effect of everything being connected is that if you are a terrible person, you have the opportunity to be terrible to vastly more victims. --- class: bulletsh4 .crumb[ # Landscape # Challenges ## Risks ] .fig30[ !(dumpster_fire.jpg)] .spacedown6[ # Everything is awful ## and the awful is on fire] ``` curl http://my-dishwasher/../../../../../../../../../../../../etc/shadow ``` ??? I don't want to go off on a tangent about security in particular, but I find that the underlying causes of security problems in IoT are an instructive microcosm of the wider challenges. Last October, the Mirai worm spread around the globe because of well known default passwords that never get changed. In April we got the endlessly entertaining story of the compromised internet dishwasher, which demonstrates how many developers aren't trained well enough to avoid common and completely avoidable pitfalls. The downside of ubiquitous communications is that there are no safe spaces. June's WannaCry malware outbreak showed us that when everything is software, everything has to be maintainable. July's second coming of the much more damaging successor hammered the point home. That particular worm shut down a global courier for over a month, cost a container shipping company something over three hundred million dollars, and wake up people, it threatened the chocolate supply. If you can't patch it, its no longer safe to own it. --- .crumb[ # Landscape # Challenges ## Risks ## Desiderata ] .fig50[ !(edna_no_capes.jpg) ] .spacedown[ # Desiderata] ??? So lets summarise what I think a response to all these faceplants should look like, then we'll go into detail on each point. --- # Select appropriate tools and platforms ??? First, choose the right platform and tool. --- # Comprehensive identity management ??? Next, make your identity and security part of your framework so that you only build it once. --- # Automate for developer and user convenience ??? Automate everything you possibly can. --- # Testing and testability kept front-of-mind ??? Design for testability. --- # Train, and Audit, and keep doing both ??? Train your teams. Share skills, watch youtube videos, get consultants, go to conferences. Whatever your budget there's a way to improve. --- # Monitor and react (automatically) ??? And lastly maintain awareness, and use that awareness to respond rapidly. --- layout: true template: callout .crumb[ # Agenda # Landscape # Challenges # Solutions ## Platforms ] --- template: inverse # Platforms ??? So lets talk about platforms. Jez Humble who literally wrote the book on Continuous Delivery gave a keynote at Agile Australia in Sydney this June where he told a story about HP's printer teams. Their cost of engineering was off the chart, yet quality was awful, and part of the reason was they were spending over a quarter of their time porting their libraries to new platforms over and over because pretty much every printer model used a different processor. They moved to a common architecture and massively reduced their reengineering cost. They could spend recovered time on innovation and quality. --- # People are more expensive than circuits ## .right[(Sorry, robots)] ??? My point here is that every hardware business has a person whose job is look at every component and ask "can we delete that to save half a cent per unit". This is a person who's empowered to remove customer value. The correct question when designing your hardware is will this hardware platform add value, in terms of resilience, longevity and enabling quality. --- # Hardware is DevOps too ## .right[(Robots, I hope this makes it up to you :)] ??? That's right, your platform is part of your devops culture. Your goal should be to select a platform that supports your devops way of doing things --- # Open and well supported ??? This means that openness and friendliness are the first things you should look for. So, select a processor and platform that has good development tools and will likely be around for a long time. --- # Case study: .blue[**ARM v7**] and .red[**Debian Linux**] ??? Thanks to the demand from Android phones and TV media boxes, there's a huge variety of ARM systems out there which are similar enough that hardware differences aren't really important. You can develop to the one architecture and select an appropriate board from five bucks up. --- .fig40[![Pi](raspberry_pi.jpg) ] .spacedown[ # Meet the #3 top-selling computer of all time ] ??? In a number of my projects we work with Raspberry Pi as the development platform. This brings the convenience of a huge selection of platform tools and a great developer community. We can use these in the lab, and for CI workers, and even for the first hundred or so product units. Then the time comes to scale up, there's a wide selection of other ARM designs to buy or build. Here's one that starts at seven bucks, this is an Orange Pi Zero from a company that I absolutely adore named Shenzen Xunlong. These folks are bankrupting me because they release a new model almost every month and I have to buy them all. --- # Artisanal free-range small-batch Linux? ## No. ??? The same goes in software. Think about what happens next time a zero-day exploit goes epidemic on the Internet. The top tier vendors probably have a patch out in a day or two. If you picked some hipster linux distribution that has thirty users and one maintainer, you may never see a patch. --- # Without the Internet, it's just a Thing. ??? I also think it's important for devices to be online as much as possible. Power and network constraints may mean that devices can't afford to be online continuously, and this is OK, but you do need to think about how to support DevOps processes within these constraints. --- # *"Is there anybody out there?"* .right[-- Pink Floyd
"Continuous Dashboarding" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)] ??? Simplest approach - red line gauges. Artificial stupidity - ignore normal, what's left? Recent ELK stack release (in June) has ML anomaly detection. --- layout: true template: callout .crumb[ # Landscape # Challenges # Solutions ## Platforms ## Dev ## QA ## Deployment ] --- template: inverse # Deployment ??? All right, so you have some tested code that's been encapsulated in container images, and now you need a device on which to run it. --- .fig50[ !(orchestrate.jpg)] .spacedown[ # Orchestrate: Never do anything by hand. ] ??? Do not fall into the trap of turning yourself into a robot. Some people say that as a programmer if you ever have to do something more than twice, you should automate it. Well let me save you those two times. If you learn how to use orchestration systems to configure servers, you'll find that pretty soon its quicker to use them even for things that you only have to do once. Also you were wrong about only ever needing to do it once. --- # Build a provisioning workflow ## Customise a clean OS (via ethernet or emulation) ??? Here's something that I've seen over and over. A project starts with developers putting together a quick prototype by starting from a blank OS installation, installing some tools and editing the configuration. Then it's six months later and there's a new version of the OS and nobody remembers how to go from a clean OS to an app ready platform. And thats how many embedded devices end up running eight year old kernels that are riddled with bugs. --- # Robo-configure the target system from a provisioning system ## Then save a filesystem image ??? So, what's a better way. Remote admin of a target machine (saltstack over ssh) Target machine in a VM - eg vagrant Target machine in a container (dockerfile) --- # How do you create a provisioning system? ## Turtles all the way down! .bottom.right[ Do you want to know **more**?
.blue[github.com/unixbigot/kevin] ] ??? You pull yourself up by your bootstraps. From a checkout of my automation environment, you can create a master server, then use that to create provisioning servers, and use those to create your fleet. Or you can do it all from one laptop. I'm going to skip the detail today, but you can find slides and video of a longer presentation on this subject on my website. What comes out the end of my process is a linux system that is enrolled in a management hierarchy, and can be remotely commissioned and updated. --- # Hey, that all sounds a bit like PaaS ## Yeah, it does. ??? At this point you might be thinking this sounds a lot what platform as a service offerings, like OpenShift or Elastic Beanstalk do. Those systems provide a pre-prepared operating system which you don't have to care about, and you just drop your code into place. Well, you'd be right. There are options for IoT where you get something very similar. I'm going to tell you about just two examples, one big, one small from a palette of a dozen or so. --- # Amazon AWS Greengrass ## IoT PaaS built on AWS IOT + AWS Lambda ??? Amazon Greengrass is one way - if you're familiar with Amazon's Lambda, their functions as a service offering, Greengrass is basically on-premises lambda, you use the same mechanisms but instead of deploying to an invisible container host in the cloud, you're deploying to an on-premises system that you nominate. --- --- # Mongoose-OS ## Multiplatform embedded OS with cloud integration and remote upgrade .bottom.right[ Do you want to know **more**?
"IoT in two Minutes" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)
"IoT at Scale" [christopher.biggs.id.au/talk](http://christopher.biggs.id.au/talk)] ??? All right, we're coming to the end. Last thing I normally talk about is the data being emitted from your things, but there isn't time today. I'll leave you with a cross reference to a whole talk on IoT data management, and the offer to present it for you at your invitation. --- layout: true template: callout class: bulletsh4 .crumb[ # Landscape # Challenges # Solutions # Coda ## Summary ] --- .fig40[ !(keep-calm.jpg) ] # Summary .left[ #### Lots of devices, too many to administer by hand #### Swimming in a soup of malware and bad actors #### Choose tools that support quality #### Pipelines for automated build/test/stage #### (Ab)use traditional cloud management tools for IoT Fleet #### Message bus all the things #### Big data now, play later ] ??? All right, let's summarise what I've talked about. We've looked at the threat landscape. We've worked through the product lifecycle from development and testing, to deployment management and retirement. --- layout: true template: callout .crumb[ # Landscape # Challenges # Solutions # Coda ## Summary ## Resources ] --- # Resources, Questions .left[ #### My SaltStack rules for IoT - [github.com/unixbigot/kevin](https://github.com/unixbigot/kevin/) #### Related talks - [http://christopher.biggs.id.au/#talks](http://christopher.biggs.id.au/#talks) #### Me - Christopher Biggs - Twitter: .blue[@unixbigot] .logo[@accelerando_au] - Email: .blue[firstname.lastname@example.org] - Slides, and getting my advice: http://christopher.biggs.id.au/ - Brisbane Internet of Things Meetup - http://tinyurl.com/bneiot - IoTsec Australia - https://iotsec.net.au/ - Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/ ] ??? Thanks for your time today, I'm happy to take questions in the time remaining. If you want to have a longer chat, you are welcome to come along to the Brisbane Internet of Things meetup when we resume in the new year. You can also contact me using the details on the screen, or come out to my office for a coffee or a training course.