name: inverse class: center, middle, inverse layout: true .header[.floatleft[.teal[Christopher Biggs] — ELK from chips to cloud].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[elasticbne] Dec 2018]] --- name: callout class: center, middle, italic, bulletul layout: true .header[.floatleft[.teal[Christopher Biggs] — ELK from chips to cloud].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[elasticbne] Dec 2018]] --- name: toply class: center, toply, italic, bulletul layout: true .header[.floatleft[.teal[Christopher Biggs] — ELK from chips to cloud].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[elasticbne] Dec 2018]] --- layout: true template: callout .header[.floatleft[.teal[Christopher Biggs] — ELK from chips to cloud].floatright[.teal[@unixbigot] .logo[@accelerando_au]]] .footer[.floatleft[.hashtag[elasticbne] Dec 2018]] --- class: center, middle template: inverse # ELK from chips to cloud ## .green[Data streaming in the Internet of Things] .bottom.right[ Christopher Biggs, .logo[Accelerando Consulting] <br>@unixbigot .logo[@accelerando_au] ] ??? G'day folks, I'm Christopher Biggs. --- layout: true template: toply .crumb[ # Welcome ] --- # Who am I? ## Christopher Biggs — .teal[@unixbigot] — .logo[@accelerando_au] * Brisbane, Australia * Founder, .logo[Accelerando Consulting] * 20+ years in IT as developer, architect, manager * Accelerando is a "full service" consultancy - chips to cloud * ***IoT, DevOps, Big Data*** ??? I've been involved with electronic gadgets since I was a teenager, and throughout 20 years as a software engineer, architect and manager. I operate Accelerando Consulting which helps businesses use technology to reduce stress. Really, that's what I believe, that technology is the gradual process of freeing humanity from drudgery and fear, and I started Accelerando because I wanted to contribute to a future that I will be excited to live in. --- layout: true template: toply .crumb[ # Welcome # Waffling ] --- # Elk in IoT --- ## waitaminnit # I owe *what?!* -- * Devices -- * DevOps -- * Data --- # My Three Laws of IoT -- * **First Law:** Devices must cooperate for the benefit of humans -- * **Second Law:** Devices must communicate, and obey instructions -- * **Third Law:** Devices must be as simple and reliable as possible --- # Publish-subscribe -- * **producer** PUB sensors/brisbane/meetup/elastic/attendance 22 -- * **consumer A ** SUB sensors/+/meetup/# -- * **consumer B ** SUB sensors/+/meetup/+/attendance --- # Interlude: harsh truth -- ## Most of the world is not Silicon Valley --- # Hierarchical communications * Sensors talk to a building uplink -- * buildings may belong to a site -- * sites talk to a regional hub --- # YAWN ## Internet, remember? --- # Devices have a hotline to silicon heaven ## Pay no attention to the heirarchy behind the curtain --- # A little bit of heaven --  .spacedown[ ## and a sprinkling of hell ] --- # Interlude: spend the damn money -- ## You cannot have -- * too much RAM -- * too many screens -- * too many environments --- # Containers: A cluster in your lap ## Why containers? -- * Dependency control -- * Immutable infrastructure -- * ~~Security~~ --- # Docker 101 -- * Docker is a friendly wrapper around Linux's process control groups -- * Docker desktop bundles a linux virtual machine for mac and windows -- * Tools like Kubernetes, Lambda, Docker Swarm manage compute-as-a-service -- * Docker-compose is the small scale cluster-on-your-lap solution -- * But it's fine for production, too --- # Docker 102 -- * A **docker image** is a snapshot of a filesystem -- * Images are a stack of filesystem overlays -- * A **docker container** is an executing instance of an image -- * The whole point of this shell game is to make softare development... --- # ...simpler ## Believe it or not -- * A container is a boxed product, it either works or it don't -- * You (the dev team) can think at the level of interactions between containers -- * Snap the bricks together like Lego --- # Approach one ## Quick and dirty .fig50l[ ] -- .fig50r[ ] --- .fig30[ ] # Approach two ## Official containers --- .fig50[ ] # Official containers --- # Yaright -- .code.huge.huge[ ```shell docker-compose up ``` ] --- # YAML (Yet Another Markup Language) ## I hate it but not as much as I hated XML --- `cat docker-compose.yml` .smallcode[ ```yaml services: elasticsearch: container_name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4 ports: - 9200:9200 kibana: container_name: kibana image: docker.elastic.co/kibana/kibana:6.2.4 links: - elasticsearch ports: - "5601:5601" logstash: container_name: logstash environment: - ELASTICSEARCH_URL=http://elasticsearch:9200/ - ELASTICSEARCH_USER=elastic - ELASTICSEARCH_PASSWORD=MeterTweeterDataEater ports: - "5000:5000" - "5044:5044" links: - elasticsearch ``` ] --- .fig50l[ ] -- .fig50r[ ] --- .fig100[ ] --- .fig100[ ] --- # Addendum: the internet toaster -- * My sensors are typically FreeRTOS or other embedded OS -- * My site gateways are often (small) Linux boxes (256M-1GB ram) -- * WHAT THE HELL JAVA WHY ARE YOU SO HANGRY? -- * Aside: Look out for Mikael Vidstedt's YOW! 2018 talk. Soon! -- * This is why beats (and Go) rock --- # Retrofitting queueing to an IoT pipeline -- * Devices publish to local MQTT -- * No room for logstash, so we build a simple helper * (a few hundred lines of Go to subscribe to MQTT and spool to a log) -- * Filebeat streams the log to cloud with failure recovery -- * Plain vanilla logrotate daemon tidies up our log --- # App in a box .code[ ```dockerfile FROM busybox WORKDIR /app COPY logger /app/ ENTRYPOINT ["./logger"] ```] --- # Beats box .code[ ```dockerfile FROM docker.elastic.co/beats/filebeat:6.3.0 ```] --- # Oh, noes! ## What do you mean "invalid binary format" .code[ ```dockerfile FROM arm32v7/debian:jessie WORKDIR /app COPY filebeat /app/ COPY filebeat.yml /app ENTRYPOINT ["./filebeat","-e","-v"] ```] --- ## Waitaminnit # Building the binary ## as easy as Pi .code[ ```Makefile $(GOPATH)/src/github.com/elastic/beats: [ -d $@ ] || mkdir -p $@ git clone https://github.com/elastic/beats.git $@ beats: $(GOPATH)/src/github.com/elastic/beats $(GOPATH)/src/github.com/elastic/beats/filebeat/filebeat: beats make -C $(GOPATH)/src/github.com/elastic/beats/filebeat GOARCH=arm filebeat: $(GOPATH)/src/github.com/elastic/beats/filebeat/filebeat cp $< $@ ```] --- ## Compose yourself .smallcode.small[ ```yaml services: broker: container_name: broker ports: - 1883:1883 build: dockerfile: Dockerfile-broker-pi context: . logger: container_name: logger links: - broker build: dockerfile: Dockerfile-logger-pi context: . filebeat: container_name: filebeat build: dockerfile: Dockerfile-filebeat-pi context: . ```] --- layout: true template: callout .crumb[ # Welcome # Waffling # Recap ] --- .fig30[  ] # Recap .nolm[ * IoT - data <= devops <= devices * You don't care (so much) about the comms topology * Chickens not Cats * Develop fast * Deploy with (almost) the same tools * Bonus: Beat the Bloat ] -- * .red[MERRY CHRISTMAS] --- # Resources, Questions ## Related talks - [http://christopher.biggs.id.au/#talks](http://christopher.biggs.id.au/#talks) ## Me - Christopher Biggs - Twitter: .blue[@accelerando_au] - Email: .blue[christopher@biggs.id.au] - Slides, and getting my advice: http://christopher.biggs.id.au/ - Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/ --- # Links * [github.com/accelerando-consulting/iot-elk](https://github.com/accelerando-consulting/iot-elk) * [sebp/elk](https://hub.docker.com/r/sebp/elk/) [sebp/elkx](https://hub.docker.com/r/sebp/elkx/) * [Official Elastic containers](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docker.html) * [Docker Compose](https://docs.docker.com/compose/overview/)