Load Testing

TechEd NA 2014 – Cloud-Based Load Testing

Posted by

0

TechEd North America 2014, Houston
Using the Cloud-Based Load Testing Service and Application Insights to Find Scale and Performance Bottlenecks in Your Applications – Vibhor Agarwal

Day 3, 14 May 2014, 10:15AM-11:30AM (DEV-B335)

Disclaimer: This post contains my own thoughts and notes based on attending TechEd North America 2014 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of either Microsoft, the presenters or the speakers.

Executive Summary—Sean’s takeaways

  • Various testing scenarios, e.g.
    • Load Testing – how will application perform at expected load?
    • Stress Testing – how/when will application “fall down”?
  • Load Test uses one or more web tests to hit web site or service
    • Lots of options for configuring test–# users, timing, etc.
    • Can configure the Load Test to match what you believe to be real-world scenario
  • User response time is a good key indicator
  • Cloud-based load testing allows scaling out in cloud
    • Allows heavier duty tests, e.g. for stress testing
    • Easy to set up and tear down test
  • Visual Studio 2013 Ultimate only
  • Application Insights—dashboard with key metrics about running site

Full video

Vibhor Agarwal – Principal Group Program Manager – Testing Tools, Microsoft

Agenda

  • Why load testing?
  • Cloud load testing
  • Application insights

Mention of healthcare web site issues—didn’t scale

Why load testing?

  • 1 – Performance testing
    • How fast will the application perform?
    • Typically when you have single user
    • Also—how fast, based on current load? Ok, see below
  • 2 – Load testing
    • How will application behave under expected load?
    • Might work fine for single user, but break under load
  • 3 – Stress testing
    • At what point will application break, i.e. at what load?
    • “Limit testing”, “scale testing”
    • If expected load is X, try out 2x, 3x, etc.
  • 4 – Capacity planning
    • Will application scale up to future capacity
    • How many more servers do we need
    • You can use load testing data to extrapolate

Demo – Performance testing

  • File | New – Web and Load Test Project
  • Create a simple webtest
  • Record some actions on web site, hit a few pages (typical browsing)
  • If there is user input required, the web test will parameterize that stuff
  • Play on local machine—repeats the same actions
  • Records how much time it takes at each steps
  • Look at dependent links that are being followed
  • Also shows # bytes being transferred

Demo – 2nd web test

  • Want to model various typical user scenarios
  • When you enter input, e.g. user input, it doesn’t capture it—so need to do full UI test

Demo – Other stuff available

  • Can data drive web test from CSV, XML or other data source
  • Can generate code based on web test, then modify stuff to customize
  • Parameterize web servers
    • Use same web test, but run on a different web server

Demo – Create Load Test

  • Project | Add | Load Test
  • Wizard
  • Think time profile
    • Use recorded – follow original time profile
    • Normal distribution based on recorded times
  • Typically do use think time, normal distribution
  • Load Pattern
    • Constant load – # users
    • Step load – start at one level, then add users. Good for Stress Testing
  • Test Mix
    • Total # of tests
  • Add web tests to load test, set distribution
  • You could create load test with many parallel scenarios
  • Browser mix
  • Load test duration
    • Set time
    • Typically, you also do warm-up – e.g. once cache is full, response gets better

Demo – Run Load Test

  • Run load test, using local machine
  • Users power of local machine to generate load
    • Works for small load
  • More realistically, you want test rig to do load test with larger # users
  • Creating test rig is sort of a complex process

What is needed to make that work?

  • Network-based load testing is
    • Expensive to set up
    • Complex provisioning
    • Slow to scale
    • Costly to maintain
  • Get PCs, physically set them up, replicate – painful

Demo – Cloud Load Testing

  • Infrastructure uses Azure cloud
  • Test Settings | Run tests using Visual Studio Online
    • That’s all you need to do
  • Test gets queued
  • You do have to connect to Visual Studio Online
  • Load test now in progress in cloud
  • Spits out data while test is running
  • Can open up load load tests that were run

Q: What is application sits behind firewall

  • You need to find hole in firewall to allow the web-based testing
  • VPN tunnel – might be too much for the tunnel, but haven’t done much with this yet

Demo – Review load test results

  • Graph showing user response time, over length of load test
  • View showing entire test, with various key indicators
  • What is most relevant is response time – 95th percentile

Demo – Excel plug-in

  • Can dump out all test data to Excel
  • Trend Analysis or Comparison view
  • Comparison view is the most interesting

Benefits of cloud-based load testing

  • 1 – Don’t have to set up test infrastructure
  • 2 – Get infrastructure in cloud when you need it
  • 3 – Use same load test project that you use on premises
    • Just single settings change
  • 4 – Scale out easily in case of doing stress testing

Cloud load testing today

  • Public preview, Nov-2013
  • General availability, Apr-2014
  • Using it: Xbox, Skype, Visual Studio Online
  • Some customers doing load testing as part of regular process, e.g. weekly
    • Rather than waiting until end of project

Visual Studio Online

  • Application Insights

Application Insights

  • 360 degree view of your application
    • Available – is application available, e.g. from various locations
    • Performing – under load conditions
    • Succeeding – usage dashboards; what are usage metrics, data

Demo – Using Application Insights along with Load Testing

  • Under Load Test | Run Settings, right-click Applications, Get Performance Data
  • Can look at various Performance Counters for your application—right in Visual Studio IDE
  • Can then export to Excel report, correlate
  • In Visual Studio Online, lots of graphs, showing metrics on your application
  • Can use webtest to test availability for long time period
  • Using load testing and Visual Studio Online to get good analytics
  • Can look at top slowest pages
    • Dbl-click into specific events; will jump back to Visual Studio and show you location of problem (in VSO)

Q: Can you manage specific tests up in the cloud? View for Test Management in Visual Studio Online?

  • Not really well represented now, but something that we have planned

Q: Do I need Visual Studio Ultimate?

  • Yes

Q: Is there any way to automatically schedule and then auto compare against various thresholds?

  • Will get to this (future things)

Q: If I run a web test that’s reading/writing to database on-premise, how will that work when running in cloud

  • Cloud test won’t be able to handle this
  • Will have to do SQL Azure storage, rather than on-premise

Upcoming features

  • Customer counters from Application Insights with load test
  • REST APIs to queue Load Tests (automate/schedule)
    • Then have your own logic to compare logic
  • Geo-specific load generation
    • Load service today always come from East U.S.
    • Coming soon—Europe
  • Richer analysis/reporting
    • Excel plug-in, reports

Who can use cloud load testing?