Continuous Dashboarding - Applying Agile methods to BI

name: inverse
class: center, middle, inverse
layout: true
.header[.floatleft[.teal[Christopher Biggs] — Continuous Dashboarding].floatright[.teal[@unixbigot] .logo[@accelerando_au]]]
.footer[.floatleft[ #AgileAus Jun 2017]]
---
layout: true
.header[.floatleft[.teal[Christopher Biggs] — Continuous Dashboarding].floatright[.teal[@unixbigot] .logo[@accelerando_au]]]
.footer[.floatleft[ #AgileAus Jun 2017]]
---
class: center, middle
template: inverse

# Continuous Dashboarding

.green[How and Why to apply Agile methods to Business Intelligence.]

![](node-red-title-flow.png)

.bottom.right[
.teal[Christopher Biggs], .logo[Accelerando Consulting]
<br>.teal[@unixbigot] .logo[@accelerando_au]
]

---
.left-column[
# Agenda
]
.right-column[
### **Strategy**:
* .pink[**Why**] should you build dashboards?
* .red[**Who**] are your dashboards for?
* .orange[**What**] to monitor
* .yellow[**When**] to build your dashboards

### **Tactics**:
* .green[**How**] to develop dashboards
* .blue[**Where**] to fit dashboards into your DevOps pipeline
* .purple[**With**] some suggested tools 
]

---
class: center, middle
template: inverse

# Why?

## Agile is about .pink[more] than code

### Dashboarding helps you .pink[learn] about your .yellow[product], and .green[customers].

---
.left-column[
#Strategy
## Why
]
.right-column.arrow.vtight[
## Why build dashboards?

#### A little thing called the Scientific Method (Bacon et.al, 1620-)
#### Time and Motion (Taylor, 1911)
#### How We Think (Dewey, 1933)
#### Plan Do Check Act  (Deming/Shewhart Cycle, 1950s)
#### SMART - Specific, Measurable, Achievable, Relevant, Time-Bound (Drucker, 1954)
#### Observe, Orient, Decide, Act (Boyd, 1976)
#### The Agile Manifesto (Agile Alliance (.orange[i.e. us!]), 2001)
#### "SMARTER" - SMART + Evaluate, Review (Yemm, 2013)
#### The only difference between .red[failing] and .green[learning] is .purple[measuring]
* (Me, ripping off .blue[@donttrythis], just now)

]

---
.left-column[
#Strategy
## Why
]
.right-column.arrow[
## Education: (n) The process of discovering your own ignorance.

#### You don't know what customers want.
* Customers don't know what they want, either.
* Companies don't know what their customers want.
* Companies might not even know what their product is!

#### How to pretend to have read "The Lean Startup"
* Part 1: Define, Learn, Experiment
* Part 2: Test, Measure, Pivot
]
---
.left-column[
#Strategy
## Why
]
.right-column.arrow[
## Validated learning - how to know if we achived our goal

#### Ask (Interrogate) the customer
#### Form an Intelligence (spy) Agency
#### Let's call it "Business Intelligence" (BI)
]
---
class: center, middle
template: inverse

# Who?

## Dashboards and Business Intelligence (BI) are .red[Product]

### Think about who are the .green[customers] of your BI

---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim.vtight[
![Munin](munin-boring.png)

## Who is your BI for?

#### Analysis and summaries targeted at marketing
 * Long term trends
 * "Bottom line" numbers

#### Detailed dashboards are traditionally targeted at ops.
 * Measure **all** the things!
 * Wake me if it's on **fire**
 * Little feedback goes back **upstream**

]

---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing.
 * Development is a different **point of view**
 * **Instrument** your applications
 * Measure **trends** and **levels** in your KPIs
 * Look for **out-of-range** metrics
 * What does normal **look** like?
 * Regression-test your **bottom line**
]

---

.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK, too
 * Is the first time you know of problems when support escalates?
 * Do support call ops and ask "Is everything OK, we're getting a lot
of calls"?
 * "Well Mrs Arbuckle, I see other customers are having problems too"
]

---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK too
#### Management want reassurance (and to know you deliver)
 * The morning report for "How long till my next **Porsche**"
 * Shipped a great new feature?  You now have a **hotline** to the CEOs eyeballs.
]

---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Who are your dashboards for?

#### Traditionally targeted at ops, but
#### DevOps is a thing
#### Support and Sales need to know everything's OK too
#### Management want reassurance (and to know you deliver)
#### Customers (do you dare?)
 * What are you **ashamed** of them seeing?
 * What could you **do** about that?

]
---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Give **customers** a weather report
![](tornado.jpg)

#### Take notice of load spikes
 * "We are processing your order.   This hour that
takes around 25 seconds but could be up to two minutes"
]

---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Give **customers** a weather report

#### Take notice of load spikes
#### Respond to temporary outages
 * "Sorry, it looks like PayPal has been experiencing high error rates
 for the last 15 minutes.  May we suggest you try AliPay or Credit
Card?"
   * This could head off a flood of support calls or bad reviews
]
---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## Give **customers** a weather report

#### Take notice of load spikes
#### Respond to temporary outages
#### Assure customers that errors are being acted on
 * "We're sorry, this page experienced a problem.  Bug #3342 has
already been forwarded to our Dev Team.   You'll get an email when
it's fixed, or check our .fakelink[change log]"
   * again, turn your failures into goodwill
]
---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
 * "Mr Lee encountered .fakelink[Bug #3342] six minutes ago"

]
---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
#### Notice recent outages affecting this customer
 * "Currently experiencing elevated transaction declines for Westpac
cards"

]
---
.left-column[
#Strategy
## Why
## Who
]
.right-column.autodim[
## What's relevant to this **support** call?

#### Notice recent errors affecting this customer
#### Notice recent outages affecting this customer
#### Remember recent behaviour of this customer
 * "Dr Patel had her password reset yesterday, and presented her old
 password three times today"

]
---
class: center, middle
template: inverse

# What?

## Not just **things**, also .green[patterns] and .red[trends]

---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
![Munin](munin-boring.png)
## What should I be monitoring?

#### Traditionally - **Things**
  * system load indicators
  * rates of key events (pageloads, signups, checkouts etc.)
  * service status
  * error alerts ("pages")
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
## What should I be monitoring?
.fig50[
![Trends](kpi_nodered.png)]
#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
* Business Goals - Rates plus Trends

]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
![Blynk Example](blynk_bi_goals.png)
## What should I be monitoring?

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
.fig50[
![](emea_orders.png)]
## What should I be monitoring?

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
  * Location and traffic patterns
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
## What should I be monitoring?
![](anomaly.png)

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
  * Location and traffic patterns
  * User Experience metrics
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
![](stacked_percent.png)
## What should I be monitoring?

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
  * Location and traffic patterns
  * User Experience metrics
  * Health of your third-party interfaces
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
## What should I be monitoring?

.fig50[
![](app_reviews.png)]

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
  * Location and traffic patterns
  * User Experience metrics
  * Health of your third-party interfaces
  * App and product reviews (iTunes, Google, Yelp, Amazon...)
]
---
.left-column[
#Strategy
## Why
## Who
## What
]
.right-column.autodim[
.fig50[
![](socmed_monitor.png)]

## What should I be monitoring?

#### Traditionally - **Things**
#### Now - **Patterns** and **Trends**
  * Business Goals - Rates plus Trends
  * Unavoidable errors - look for out-of-range levels
  * Location and traffic patterns
  * User Experience metrics
  * Health of your third-party interfaces
  * App and product reviews (iTunes, Google, Yelp, Amazon...)
  * Social media (facebook, twitter, et.al.)
]
---
class: center, middle
template: inverse

# When?

## Make Business Intelligence part of your .orange[**DevOps**] processes

---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.tight[
## When should I build dashboards?
.fig50[
![](page_flow.png)]

#### Early and Often!
* Dashboards inform your planning and strategy
* Make sure you *really* know the status quo before you code
]
---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.tight[
## When should I build dashboards?
.fig50[
![Monitor trends in your KPIs](kpi_nodered.png)]

#### Early and Often!

#### Continuously
* Dashboards are (subjective) tests
* **Hopefully** you already run tests before you ship
* Think about what tests you should run **continuously**
]
---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.tight[
## When should I build dashboards?

#### Early and Often!
#### Continously: Dashboards are (subjective) tests
.fig50[
![](long_term_trend.png)]
#### Releases: Dashboards are longitudinal tests
* How does this release compare to last release?
]
---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.tight[
![Server Error Dashboard](blynk_bi_errors.png)
## When should I build dashboards?

#### Early and Often!
#### Continuously: Dashboards are (subjective) tests
#### Releases: Dashboards are longitudinal tests
#### Fridays: Dashboards are your DevOps "airbag"
* Is the bucket still under the money waterfall?
* Is it Friday?   Does that scare you?
]
---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.tight[
## When should I build dashboards?
.fig40[
![](logging_code.png)]

#### Early and Often!
#### Continuously: Dashboards are (subjective) tests
#### Releases: Dashboards are longitudinal tests
#### Fridays: Dashboards are your DevOps "airbag"
#### Coding: Dashboards help you think about Features holistically
* Who needs to know about this new feature?   Training?
* What other parts of our ecosystem will this affect?
]
---
.left-column[
#Strategy
## Why
## Who
## What
## When
]
.right-column.autodim.vtight[
## When should I build dashboards?
.fig40[
![](who_ya_gonna_call.png)]

#### Early and Often!
#### Continuously: Dashboards are (subjective) tests
#### Releases: Dashboards are longitudinal tests
#### Fridays: Dashboards are your DevOps "airbag"
#### Coding: Dashboards help you think about Features holistically
#### During test: Dashboards help you plan for adverse outcomes
* **Validate** your assumptions
* What do I **do** if this breaks?
* Who will notice a failure?   Who needs to be **notified**?
]
---
class: center, middle
template: inverse

# How?

## Encourage a .purple[**data-loving**] culture

---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
]
.right-column.bulletsh4[
## So, How do I get there from here?
#### **Experiment** with your data
#### **Develop** dashboards alongside features
#### **Test** your dashboards
#### **Deploy** your BI configuration as build artifacts.
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
]
.right-column.autodim.tight[
## So, How do I get there from here?
#### **Experiment** with your data
* Derive synergy from bringing your silos together
* Visual dataflow tools like .red[**Node-RED**] ([nodered.org](http://nodered.org/))
* Rapid dashboard tools like .green[**Blynk**] ([blynk.cc](http://blynk.cc))
* Explore your data with GUI tools like .purple[**Kibana**] ([elastic.co](http://elastic.co/))

.fig30[
.img-320w[![Node-RED](nodered-frontdoor.png)]]
.fig20[
.img-200h[![Blynk](blynk_bi_goals.png)]]
.fig40[
.img-320w[![Kibana](kibana-basics.jpg)]
]

]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
]
.right-column.autodim[
## So, How do I get there from here?
![](docker.png)
#### **Experiment** with your data
#### **Develop** dashboards alongside features
* Orchestrate your BI stack (Docker, Vagrant, whatever)
* Choose tools that produce shippable artifacts
* Demonstrate your dashboards as a deliverable
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
]
.right-column.autodim[
## So, How do I get there from here?
.fig50[
![](dashboard_test_inception.png)]

#### **Experiment** with your data
#### **Develop** dashboards alongside features
#### **Test** your dashboards against your code
* ...and vice-versa
* Dashboards are **Customer-Facing** product.
* Trustworthiness is crucial
]
---
class: center, middle
template: inverse

# Where?

## Dashboards as .blue[**code**]

---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
]
.right-column.autodim[
## Inserting dashboards into your development pipeline
.fig40[
![](dashboard_review.png)]
#### Code-review your Dashboards
* Pretty-print JSON or XML data files
* Take care to normalise formatting and field order
* Use Visual diffing tools to highlight layout changes (eg. pdiffy)
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
]
.right-column.autodim[
## Inserting dashboards into your development pipeline
#### Code-review your Dashboards
.fig40[
![](testable_dashboard_element.png)]
#### Part of your CI test runs
* Yes, you can unit-test dashboards
* Structure your BI pipeline as Data, Analysis, Display
  * Add hooks to inject test data
  * Each stage can be unit-tested with known data
* Leverage your integration tests to produce longitudinal data
* Use behavioural testing tools to confirm UI outcomes
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
]
.right-column.autodim[
## Inserting dashboards into your development pipeline
.fig40[
![](dashboard_test_inception.png)]
#### Code-review your Dashboards
#### Part of your CI test runs
#### Performance testing - daily/sprintly test

* Build a test that stresses your product to breaking point
* Confirm that the right alarms go off
* Break things, then verify your ecosystem responds
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
]
.right-column.autodim.tight[
## Inserting dashboards into your development pipeline
.fig40[
![](orchestrate_upload.png)]
#### Code-review your Dashboards
#### Part of your CI test runs
#### Performance testing - daily/sprintly test
#### Automate deployment - easier with some tools than others
* Best case - install configuration files as ordinary artifacts 
* APIs - pull and push
* Database - dump/restore
* Web scraping - here be dragons
]
---

class: center, middle
template: inverse

# With

## Case Study: Adding value to .yellow[**Logstash**] with .red[**Node-RED**] and .green[**Blynk**]

---

.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column.vtight[
## .green[Elasticsearch], .yellow[Logstash], and .purple[Kibana] (ELK)
.fig40[
![Kibana](kibana-basics.jpg)]
* .green[**Elasticsearch**] Fast Distributed NoSQL database
* .yellow[**Logstash**] (et.al) Push event streams, application logs, performance data and network stats to an Elasticsearch storage cluster
* .purple[**Kibana**] Query, Visualise and Dashboard your data
* ELK is Open Source: self-host, AWS, or SaaS
* Off-the-shelf docker image suitable for dev, CI, small deployments
* Quick interactive querying
* Interactively build visulisations and dashboards
* Good support for import/export of dashboard components
* Weak on certain kinds of analysis and alerting
]
---
class: center, middle

.fig100[
![Kibana](kibana-basics.jpg)]

---

.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column.tight[
## .red[Node-RED] - A Visual system for Wiring the Internet 
.fig40[
![](nodered_workspace.png)]
* Open source (Node.js)
* Originated at IBM, now under JS Foundation
* .red[**Flow**] based data processing
 - "Fire all the programmers", at last?
* Create flows using drag+drop flow editor
 - `http://nodered.example.com:1800/`
* Huge plugin ("node") ecosystem 
* Integrated dashboarding
 - `http://nodered.example.com:1800/ui`
 - or output values to other data stores

]
---

## Business Goal Monitor

![Business Goal Dashboard](blynk_bi_goals.png)
.fig60l[
![Business Goal Monitor](bi_goals.png)]

---

.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column[
## .red[Node-RED] - Nodes and Flows
.fig40[
![Simplest flow example](simplest_flow.png)]
* The .blue[**coloured boxes**] are .red[**Nodes**]
 * Nodes are **npm** modules
* Drag and drop nodes.  Connect with links to form .green[**Flows**]
* .green[**Flows**] persist as JSON data (`~/.node-red/flows.js`)
* .green[**Flows**] process .purple[**Messages**] (JSON)
* Over 750 node modules available
 * plus a general-purpose **function** node
 * or write-yer-own nodes

]

---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column.tight.small[
## (ab)Using .red[Node-RED] for Business Dashboarding
![Server Error Dashboard](blynk_bi_errors.png)

* Search most SQL and NoSQL databases (eg. .yellow[**Elasticsearch**])
* Filter, Smooth, Report-by-Exception
* Built in HTML .purple[**dashboard**] nodes (`node-red-dashboard`)
* Push data to .green[**Blynk**] for iPhone/Android Apps

.fig60l[
![Dashboard UI](dashboarde.png)]

]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column.small.tight[
## Blynk - Agile dashboarding for one user or a thousand
* .blue[blynk.cc] - cloud data, or self-host
* interactive dashboard editor
* share an app (with data) or clone (separate data)
* free for ad-hoc sharing, pay for app-store distribution

![Blynk Widgets](blynk_widgets.png)
![Blynk Sharing](blynk_share.png)
![Blynk Cloning](blynk_clone.png)
]
---

## Server Error Monitor

.fig60l[
![Server Error Monitor](bi_errors.png)]
.fig30[
![Server Error Dashboard](blynk_bi_errors.png)]

---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
]
.right-column.vtight[
## Social Media Integration
.fig30[
![](socmed_monitor.png)]

* Receive feeds from twitter, facebook etc.
* Recieve feeds from app stores
* Post to slack, and monitor channels
* Analyse, Summarise, Respond, Alert

.fig60l[
![](app_reviews.png)]
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
# Coda
## Summary
]
.right-column.bulletsh4[
## Summary
#### Target your dashboards, keep them focused
#### You can do a lot more than just display graphs
#### Design your dashboards early
#### Be Agile: experiment, build, ship, iterate.
#### Dashboards are code
#### Bring all your data together for synergy
]
---
.left-column[
# Strategy
## Why
## Who
## What
## When
# Tactics
## How
## Where
## With
# Coda
## Summary
## Resources
]
.right-column.small.tight.bulletsh4[
## Resources, Questions

#### Source code
* Docker configuration to run ELK and/or Node-RED in dev or CI
* Example dashboards with data generators
* [github.com/unixbigot/dashcon](https://github.com/unixbigot/dashcon)

#### ELK - [elastic.co](https://elastic.co)
#### Node-RED - [nodered.org](https://node-red.org/)
#### Blynk - [blynk.cc](https://blynk.cc/)

#### Me - Christopher Biggs
  - Twitter: .blue[@unixbigot]
  - Email: .blue[christopher@biggs.id.au]
  - Slides, and getting my advice: http://christopher.biggs.id.au/
  - Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/
]