That Conference 2016 – From Inception to Production: A Continuous Delivery Story

That Conference 2016, Kalahari Resort, Lake Delton, WI
From Inception to Production: A Continuous Delivery Story
Ian Randall

Day 3, 10 Aug 2016

Disclaimer: This post contains my own thoughts and notes based on attending That Conference 2016 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of the speakers.

Executive Summary

  • Ian shares best practices of his development group at Pushpay, from inception to deployment

Tell a continuous delivery story in context

  • Pushpay company, SaaS business
  • ACMR Growth – committed monthly revenue stream
  • to $10 million in 3 quarters (5 considered fast)
  • Everyone at Pushpay paid to deliver value to business
    • Code not yet running on production server has no value


  • Tools (top)
  • People, practices
  • Just culture & blameless postmortems
    • Bottom layer, most important

Our journey begins

  • Idea


  • Shared vision over the value to the business
  • Talk about why to build it
    • Better than talking about what


  • Who has conversation?
    • Product
    • QA – must involve QA in initial discussion
      • Can’t just test quality at end
      • “Bake quality in”
      • Build with tester (pairing)
    • Dev

Building a feature

  • Dev – how will I build this thing?
  • QA – how will I break this thing?
    • Very valuable insight
    • “Nobody will ever do that”
    • QA tester writes out what tests should be
    • Then Dev writes unit tests based on this
    • Quality Assistant (not Assurance)

Building a larger feature

  • Long-lived feature branch (git)
    • Delta gets too big
      • Smaller deltas are lower risk
    • No feedback – from user
    • (good) dry code
  • Feature switches (toggles)
    • Small deltas
    • Regular feedback
    • (bad) technical debt
      • Some code duplicatoin
      • But you have to go back later and yank out old code

Pushpay terminology

  • Delta – diff between production server & head of master branch
    • Code no yet running in production
  • Want to keep delta small and contained
  • Shipping in tinier increments, easier to figure out what the cause of the problem is

Feature Switches

  • Configuration per environment
    • Features.config
    • Features.Production.config
    • Features.Sandbox.config

Feature Switches

  • Url manipulation to toggle switches on/off
  • Deliver daily increments of (non-running) code
  • Light up a slice of feature
  • Measure – statsd
  • Re-think road-map to compmlete feature


  • Works on My Machine
  • Context switching after QA person finds problem
  • Pushpay, turned around
    • Must work on your machine
    • Tester decides this
    • Hand laptop to tester, let them test on the dev machine
    • Best value for company from dev point of view–watch tester break it right in front of them
    • Shortens the DEV/QA cycle
    • Pair testing

Code review

  • Every line of code gets reviewed
  • Code must reviewed and wommed before merging
  • “Roll forwards to victory”

Code Review

  • Do
    • Validate approach
    • Performance, security, operability
    • Cohesion, coupling
    • Be honest
  • Don’t
    • Be rude – e.g. “dude that’s gross”
      • Better – tell coder how you would have done it?
    • Seriously, don’t be rude
    • Sweat the small stuff, like bracing, spaces
      • This stuff is not important


  • Someone else does it all again!
  • Pollination – not necessarily from your cell
  • Might not fully understand the context of your feature

4 Cotinuouses

(1) Continuous Integration

  • Source control
    • PR branch
    • Pushed early, for discussion
  • Build & Test
    • TeamCity does builds, using nUnit for unit testing
    • Integration testing – anything that crosses a boundary

CI – Source Control

  • PR-based workflow
  • Review happens in the PR
  • Everything reviewed

CI – build and test

  • PR Branch: Build, unit test and integration test
  • Merge into master – build, unit test, integration and acceptance test
    • Goes to QA – then rolled back or pushed to production
    • Human decision to push to production
    • Acceptance tests in Selenium – a bit brittle, but have some value
    • Create static Model and push to View, then test
      • If it breaks there, it’s not because of back-end
      • Verified that your site is intact, visually
      • Renders view against snapshot – if diff, either a bug or you accept as new snapshot

(2) Continuous Deployment

  • Automatically build, package and deploy to QA
    • Octopus deploy (great tool)
  • Manually promote package to production
    • One button click (because of Octopus)

(3) Continuous Delivery

  • Operability
  • Value

CD – Operability

  • Exception logging
  • App logging (Log4Net)
  • App metrics (statsd)
    • Measure absolutely everything
  • Incident

CD – Value

  • Add incremental bits of value to the product
    • Need to think – is there maybe value in shipping a portion of the feature?
  • Measuring the effectiveness

(4) Continuous Improvement

  • Actively seeking out opportunities to improve
    • Fix broken windows
    • Leave codebase in better state than when you found it
    • Improve the process


  • Shipbot, Beebot, Salesbot
    • Many, many, many more
  • @C3PR in action
    • Catalog of little commands
    • People joining PRs together

Just Culture

  • sidney dekker
  • Retributive culture
    • Clarity between acceptable and unacceptable
  • Restorative culture / model
    • Focuses on learning from what went wrong
    • Safe to fail

Fear of breaking things will paralyze an organization

Toyota’s Five Whys

  • Keep asking why until you get to root cause of problem
  • Doesn’t work for Pushpay
    • No single thing that is root cause
    • And often turns into “who”

Blameless Post-Mortems

  • Talk about how to stop the thing from happening again
  • When?
    • When there is an opportunity
    • Often, after break to production
    • Or even when something brings QA server down
  • How?
    • If we had a meeting, the loudest person in the room would do the most talking
    • So we do this asynchronously in a Wiki
    • Coordinated in slack channel #morgue
    • Co-ordinated with person closest to the incident
  • What?
    • Four sections in report
      • Scenario and impact
      • Timeline – write in real-time
      • Discussion
      • Mitigation – make sure this type of thing won’t happen again
        • Actionable ticket in Jira, highest priority possible
        • Slipping feature is better than having the incident happen again


  • Easy for manager to say to staff–when stuff happens, it’s not your fault
  • Much harder to go to CEO and say that stuff just happens
    • Board reads every post-mortem

That Conference 2016 – Lean UX

That Conference 2016, Kalahari Resort, Lake Delton, WI
Not Just Arts & Crafts: A Developer’s Guide to Incorporating Lean UX Practices into the Devleopment Process
Rachel Krause

Day 3, 10 Aug 2016

Disclaimer: This post contains my own thoughts and notes based on attending That Conference 2016 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of the speakers.

Executive Summary

  • An argument for a lean/agile approach to UX design

Contact Info

@rachelbabiak // #thatconference
– On site, on SCRUM teams
– Consulting, be UX person on client team
– UX coaching


  • Not just putting a pretty picture on something
  • UX is the experience that the user goes through
  • What will help the user in the best way possible

Example: Dominoe’s Pizza Tracker

  • Visually, see where pizza order is in the process
  • Distracts the user a bit while they’re waiting
  • Really helped Dominoe’s competitively

UX Schools of Thought

  • Genius Designer
  • Mindset of experimentation

Genius Designer

  • Research, work done up front
  • Outside of development team
  • Developers have no say
  • “Best design possible”
  • If something goes wrong, blame the user

Mindset of experimentation

  • No matter how brilliant you are, you can’t figure it out
  • Release early/often

Lean UX

  • Book: Jeff Gothelf
  • Focus on experience being designed
  • Move us forward

Focusing on 3 principles from book today

  • Accept failure
    • It’s okay to be wrong
  • Reduce waste
    • Get rid of high fidelity mockups
  • Collaborate
    • Everyone gets a say in the design
    • Gets everyone on the same page

Trying to fit Lean UX into Agile

Explain via cake metaphor

  • Big Design
    • Pretty picture of cake–you’ll get this at end of project
    • During development, you get cake that looks similar
    • Pivot when necessary, make changes
  • Lean UX
    • Start with wish list, rather than picture
    • Everything about the list can and will change
    • Scary to think of UX as just a list of features
    • Start with something small, e.g. cupcake
      • Everything there, but on smaller scale
      • Will release to users
    • Users give feedback
    • Then come out with something bigger
      • More features
    • Eventually get to something bigger, but matching what users need
    • Or cupcake might be all that the users need

Example: Pokemon Go

  • Added feature after field testing
    • Take screenshot

Best UIs are designed by the entire team

  • Different roles bring different perspectives

Example: A website

  • QA might add opinion about usability
  • Dev weighs in
  • Product Manager weighs in
  • Design changes as people offer thoughts

No complicated software necessary

  • Nothing that you have to pay money for
  • Need to use something that will work for entire team
    • E.g. Just a whiteboard
  • No time for high value mockups
  • Don’t get too attached to mockups


  • Traditionally–heavily researched, validating
    • Packaged up and given to stakeholders
    • This takes a lot of time


  • Easier, lean
  • Originate from brainstorming sessions
  • Based on stakeholders’ domain expertise
  • Starting point for understanding users
  • Think about role
    • E.g. Administrator, Warehouse Worker, Read-only User

Demo: Proto-Personas

  • Four quadrants
  • Upper left–name and face
    • Gives up empathy for the user
  • Upper right–basic biographical information
    • Who, where coming from, what devices
    • Can be specific, e.g. married with 2 kids
  • Lower left–pain points & needs
    • Frustrations with current solutions / products
  • Lower right–potential solutions
    • Can be subjective, e.g. wants things super easy
  • Do this on whiteboard, can take photo
  • Stakeholders invited in on this process
  • Talk through things and zero in on who you’re building software for
  • If you can validate with real people, your assumptions were correct

Kids Experiment

  • Had kids run through this process, with superheroes

User Journeys

  • Once you have persona, give him a journey
  • Series of steps that user is going to go through when working with software
  • [Persona] wants to [action] because [need] but [friction]
    • James wants to check in an inbound shipment because a railcar just arrived, but he doesn’t have accurate weights yet.

Now map the process

  • Steps in user journey
  • What happens, what the user does
    • Can do the journey for pre-solution or post-solution
  • Needs, activities and expectations
    • At each step
  • Persona’s emotional state
    • Graph between + (delightful) and – ()
    • Give rating for each step in the journey
  • Opportunities for Improvement
    • At each step
    • Don’t worry about scope at this point
  • Can do in Google Slides
    • Something simple

Design Sessions

  • In place of Photoshop stuff
  • Get an idea of the layout
  • On whiteboard
    • User needs, check things off as you address them
    • Have small journey map to see where you’re at
  • Do this collaboratively
  • Why do this
    • We might think we’re all on the same page
    • Once we draw it out, we realize we’re thinking differently
    • Then hash through things and come to the same vision


  • Don’t have to do all of this than every single thing
  • Proto-personas
    • At start of project
    • Or later, if you don’t already have them
  • User Journeys
    • Whenever you don’t have enough information to fill out acceptance criteria
  • Design Sessions
    • When you need to be on the same page
  • Don’t do extra meetings for this
    • Can piggyback on existing Scrum meetings

Example: salesforce ux

  • Wanted to redesign their core web product
  • Had 85 designers and 60 Scrum teams
  • Used Lean UX to revisit assumptions
  • Then collaborated, included everyone on the team
  • Lightweight, just using Sharpie
  • How, with so many people?
    • Regular UX reviews on the calendar
    • Developers in design process early and often

Got a story?

  • Let Rachel know successes and failures


  • @rachelbabiak // #ThatConference


  • Remote teams?
    • Some tools out there for virtual design session
    • E.g., Google Slides
  • Comment – When documenting, ask why you’re creating the document
  • Comment – Balsamic
    • Cartoony layout keeps people from focusing on the details
    • Rachel: Challenge that, since it’s a paid application
      • Why not just use whiteboard
  • What to during design process
    • Translate some of the needs and pain points into design elements
    • Not a formal process, but doing gut check w/user’s pain points
  • Examples of turning negatives in journeys into positives
    • Often quick wins, e.g. confirmation dialog to avoid unintentional deletes
    • For bigger ones, you talk about how much value is in an improvement
  • How do you validate a design after it’s out there?
    • Identify who your users are, then do beta test
    • Do beta testings with stakeholders and users
    • Watch them walk through the journey, validate
    • If you have no users, you’ll have to make assumptions
    • Might go down hallway and use someone else, e.g. product manager