That Conference 2018 – What’s New in C#7

That Conference 2018, Kalahari Resort, Lake Delton, WI
What’s New in C#7 – Jason Bock

Day 2, 7 Aug 2018  4:00 PM

Disclaimer: This post contains my own thoughts and notes based on attending That Conference 2018 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of the speakers.

Executive Summary

  • Brief review of history of C# versions
  • Lots of good new stuff in C# 7
  • Importance of C# and .NET now being open source

Language Evolution

  • C# – 2002, .NET 1.0
  • v1 – classes, evenst, properties
  • v2 – generics, partial types, anonymous methods, nullable types
  • v3 – LINQ, implicit types, extension methods, lambda expressions
  • v4 – dynamic binding, embedded interop types, named arguments
  • v5 – async/await, caller info attributes
  • v6 – Roslyn, exception filters, await in catch, auto prop initializer, default values for getters, expression-bodied members, string interpolation, null propagator, nameof
  • v7 – out variables, tuples, local functions, binary literals, ref returns/locals, throw expressions
  • v7.1
  • v7.2
  • v7.3

.NET Foundation

  • Foster open source development
  • Everything in .NET is now open source, on github
  • Contributors from both inside/outside MIcrosoft

.csproj file

  • Can target LangVersion, or latest

Binary digits, digit separators

  • Underscore in integer constant is separator–doesn’t do anything
  • Can do one or more underscores anywhere you like
  • 0b – binary literals

Local functions

  • Declare functions within functions
  • Compiled as private methods, so performance is the same as private methods

Out var

  • Replaces big try/catch block
  • Or TryParse
  • 3rd way in C# 7 – declare out var directly in method call

Ref Returns and Locals

  • Copy semantics for structs
  • Traditional list, indexer returns element, copied if element is value typed
  • Indexer can use ref keyword to pass element back by reference
    • For value typed objects, you get a reference to the value-typed object
  • Far faster (if elements are large)
  • Where you assign the element, you also use ref keyword

Pattern Matching

  • Using switch statement
  • Pass in parameter of parent type, then case on child type
  • Can also add when clause to trigger on stuff specific to that subtype
  • default case can be anywhere
  • case null

Tuples

  • Not Item1, Item2 syntax from v5
  • Lets say that you want to pass two values back out of a function
  • public (int, int) ReturnTwoInts() { … }
  • Maps to ValueTuple, which has Item1, Item2, etc.
  • Can do named values
  • public (int x, int y) ReturnTwoInts
    • Can now ref result.x, result.y
  • Calling it
    • (var x, var y) = ReturnTwoInts()
  • Can use _ to indicate you don’t care
  • Deconstruct method – pull data out as tuple (?)
  • Can use tuples as concrete types in generics
  • Assign multiple properties in a single line of code with a tuple

ValueTask<T>

  • Task is a reference type, but you might want a value-typed task for performance
  • ValueTask<T> is just a struct that does everything that a Task does
  • You’d use this only if you really need the added performance

Expression bodied members

  • Define wherever you like, not just read-only properties

[C# 7.1]

async main method

Default literals

  • Instead of default(int), you can infer the type of the default
  • So just use default keyword, type is inferred

[C# 7.2]

readonly struct

Spans

  • Look at chunk of memory
  • “Safe” way to read/write direct memory
  • e.g. “11,22” .AsSpan()
  • .Slice instead of Substring
  • No allocations
  • More performant

Constraint Generic

  • To enum type, to delegate type
  • where T : Enum

Compare Tuples

  • Equality semantics is correct

Backing field not serialized

  • On auto property, do [field: NonSerialized]

Future Directions

  • Ranges – foreach, in 5..10
  • Default Interface Methods – default implementation for a method on an interface
    • Looks odd
  • Nullable reference types
    • Have to opt-in, to avoid breaks
    • Once opt-in, MyClass mc would be non-nullable
    • Have to do MyClass? to get nullable
    • n!.Length – don’t worry about null-check
  • async streams – async enumerator
  • Generic attributes – e.g. [MyAttribute<int>]
  • Records and with expressions
  • Shapes and extensions
    • Mads T
    • Extensions everywhere – e.g. properties

Questions

Q: Isn’t this scary / dangerous? What is the main use case?

A: Multiple return values, e.g. int.Parse that is more elegant because it returns multiple things. Helps to avoid defining little objects designed just to pass data around between methods

That Conference 2018 – Correcting Common Mistakes When Using async/await in .NET

That Conference 2018, Kalahari Resort, Lake Delton, WI
Correcting Common Mistakes When Using async/await in .NET – Brandon Minnick

Day 1, 6 Aug 2018  4:00 PM

Disclaimer: This post contains my own thoughts and notes based on attending That Conference 2018 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of the speakers.

Brandon Minnick

Developer Advocate, Microsoft

@TheCodeTraveler

https://www.codetraveler.io/THAT-2018-AsyncAwait/

Executive Summary

  • Helps to know underlying IL that’s generated
  • Helps to know core task-based objects that async/await use
  • ConfigureAwait(false) if you don’t need to come back to UI thread
  • .GetAwaiter().GetResult() instead of await to get synchronous execution

Multi Threading

  • e.g. ReadDataFromUrl
    • Our method is async
    • does await on DownloadDataTaskAsync
    • 1st chunk of code is run by the thread that calls this method
    • 2nd thread–calls the Download method
    • After await, execution picks up with 1st thread
  • Typically, first thread, i.e. main thread, is the UI thread
  • UI thread
    • Draws UI on screen
    • Rule-of-thumb for executing on UI thread. Because refresh rate is 60Hz, don’t take more then 1/60 sec (17 ms)

Intermediate Language

  • Compiler generates class for your original method
  • Implements IAsyncStateMachine
  • Every local variable from method becomes private field in class
  • MoveNext method
    • Giant switch statement
    • $PC keeping track of state
    • Also a big try/catch block surrounding everything

Quick Review

  • async keyword adds 100 bytes
  • Every async method becomes a class

Await Every Async Method

  • Non-awaited async methods hide exceptions
  • Ditto for Task.Run — GetAwaiter().GetResult() will ensure we catch exception

Let’s Fix Some Code

  • Can’t await in a constructor
    • ICommand–made for fire and foreget
    • In Execute method, do async lambda with await
    • So from constructor, we can call MyCommand.Execute(null)
  • Never use async void ?
    • Not entirely true
    • Don’t use async void when not on UI thread
    • But on UI thread, if you throw exception from within async void–you’ll catch the exception
  • So from constructor, call Initialize(), which is an async void and can do the await
  • NEVER use .Wait
    • Blocks calling thread entirely
    • Doesn’t unwrap an exception
  • NEVER use .Result
  • Way better method
    • GetAwaiter().GetResult()
    • Does unwrap any exceptions
    • Can get the result from the async method
  • Note: .GetAwaiter().GetResult() is to be used in non-async method
    • In async method, just use await keyword
  • ConfigureAwait(false)
    • By default, you context switch back to UI thread
    • But this is expensive–background thread has to wait for UI thread
    • Context switch takes time
    • When background thread is done, execution stays on that background thread. Does not come back to UI thread
    • But note that you do need to come basck to UI thread if that method does some stuff with the UI, e.g. modify a control
  • ConfigureAwait(true)
    • This is the default, equivalent to not having ConfigureAwait() there at all
  • Don’t call .Result
    • Replace with await that method call to get the result directly
  • Special case, returning await something that returns Task<>
    • You can return the task directly, rather than awaiting it, then doing async on the method
    • Look for “return await”
  • Exception to this rule
    • You should await if you want to catch exception in method that you’re calling
    • So you could argue not to every do that trick

Async/Await Best Practices

  • Never use .Wait or .Result
    • Use await keyword
    • Or .GetAwaiter().GetResult()
  • Fire and forget tasks
    • Use ICommand
    • (or) async void (if you know you’re going to be on main thread)
  • Avoid return await
    • Remove async keyword
    • Except in try/catch blocks or using blocks
  • Utilize ConfigureAwait(false) as much as possible
    • In methods that don’t touch the UI

Xamarin University

  • Various course, e.g. CSC350: Using async await

That Conference 2016 – C#: You Don’t Know Jack

That Conference 2016, Kalahari Resort, Lake Delton, WI
C#: You Don’t Know Jack – George Heeres

Day 2, 9 Aug 2016

Disclaimer: This post contains my own thoughts and notes based on attending That Conference 2016 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of the speakers.

Executive Summary

  • This talk presented a series of lesser known but useful C# skills

Strings

  • ANSI C
    • Array of characters
    • Ptr to null-terminiated array of chars
  • Getting length
    • More efficient to put length at start of string
  • Appending
    • Can’t just append in-place
    • Allocate new array, copy both pieces

C# Strings – What not to do

  • Long series of + operators to build up a string
  • Luckily, compiler makes this more efficient
  • A CIL diversion
    • Common Intermediate Language (aka MSIL)
    • Runs on CLI
  • 2nd example for concatenation
    • while loop that does + operator to append each piece
  • Better–use StringBuilder
    • Give it suggestion for size, should make best guess
    • Default capacity on StringBuilder is 16
    • Then doubles whenever it runs out of space

IFormatProvider

  • Format specifiers, built-in and custom
    • Putting object in format string, invokes ToString()
  • By default, ToString() returns type name
  • Can create custom IFormatProvider
    • E.g. if you want some custom format for telephone numbers
    • Bad–can’t globally register

String interpolation (C# 6)

  • Simplifies format string
  • Avoid problem with bad index
  • $”some test (varname) more string”
  • Compiler converts this to classic string format

Debugging helper

  • Attribute – DebuggerDisplay, give format string to indicate how to format in debugger

Operators: Ternary

  • ? : syntax
  • condition ? true : false
    • Return null if lefthand portion evaluates to null
    • Never nest ternaries
  • C#6: dice?.Values

Null coalescing operator

  • left ?? right
  • Return left if operand non-null, else return right
  • Good for lazy loading properties
    • return _dice ?? (_dice = init())

Operators: Implicit/Explicit

  • Write implicit operator, e.g. convert object to int, perhaps person.id
  • Allows assignment without cast operator

Operator overloading

  • Standard arithmetic operator

Enumerable: yield return

  • In method for IEnumerable, only execute code when the enumeration is traversed
  • Technically, violates MVC pattern
  • Worry about whether connection is still open at a later time
  • Bypass this–calling ToArray, ToList forces fetching all objects

Bit math |, &

  • Use in enumeration, set flags as power of twos
  • Can then store multiple values in single enumeration variable
  • Need [Flags]
  • Can create helper values in enum, combination of other enumerated values

Constructors – DRY > “Daisy Chain”

  • Don’t do the same initialization across multiple constructors
  • DRY – Don’t Repeat Yourself
  • Invoke another constructor with this keyword
  • Can go up or down (fewer or more parameters)

Extension methods

  • Just syntactic sugar
  • But very useful

Hacking: Decompiling Code

  • JetBrains et all
  • Decompile from IL, i.e. creates C# from IL

Hacking: Oh Snap

  • By default, anybody can decompile from your IL
  • Mitigate
    • Tools
    • Don’t store sensitive data
    • Move algorithms to server
    • .NET Obfuscator

Reflection

  • Review: can’t access private data
  • Can actually access using reflection
  • Example
    • GetType(), GetProperty(), GetValue()
    • BindingFlags Instance, NonPublic
    • Can also call method
    • Can also reflect on static data
      • Just need BindingFlags.Static
  • Generics
    • Get initial type, then construct generic type
    • GetConstructor or do MakeGenericType

Reflection: A Practical Example

  • Adding DisplayName attributes on enum values
  • This is a custom attribute
  • Then write extension method for the enum type, to get the display name
  • GetCustomAttributes on the type, find the attribute, cast

Crafstmanship: Refactoring

  • Best practices
  • Always leave code better than when you found it

“Magic” numbers

  • Move magic numbers to const int, define in just one place

Be expressive & declare intent

  • Move check into method, with name that tells you intent
  • E.g. IsValidTelephoneNumber()
  • Encapsulates ugly code, cleans up main code that you’re reading
  • Good code is self-documenting

Refactoring: String comparisons

  • Case insensitive comparison
  • ToLower or ToUpper–inefficient
  • Better
    • Use string.equals, specify StringComparison.OrdinalIgnoreCase

Refactoring: Intellisense

  • To add intellisense for your code, use XMLDoc in front of everything
  • [Sean: Also shows up in object explorer]

List<T> Properties

  • Returning List<T> or IList from class
    • Implies functionality present in IList that you may not have implemented
    • Instead, return IEnumerable if you’re returning read-only list

Extensions: DTO

  • In n-tier model, use DTO (Data Transfer Object)
  • Example, same DTO going from database all the way out to HTML
  • Problems
    • Leaky abstraction
    • If you change data access layer, changes go through every layer
  • Better
    • New data object at each layer
    • Tools like structure map can help map from one object to another
    • Or just use extension methods to convert from A to B
    • Remember–layer should not know anything about layer above it
    • Extension methods help with this
    • Conversion is in the outer layer, not the inner

Extensions: More.. Support IEnumerable

  • Take IEnumerable a, return IEnumerable b
  • Use yield
  • Or write with LINQ

TechEd NA 2014 – Async Best Practices for C# and VB

TechEd North America 2014, Houston
Async Best Practices for C# and VB – Mads Torgersen

Day 4, 15 May 2014, 10:15AM-11:30AM (DEV-B362)

Disclaimer: This post contains my own thoughts and notes based on attending TechEd North America 2014 presentations. Some content maps directly to what was originally presented. Other content is paraphrased or represents my own thoughts and opinions and should not be construed as reflecting the opinion of either Microsoft, the presenters or the speakers.

Executive Summary—Sean’s takeaways

  • Avoid async void
  • Don’t use parallel threads for I/O bound code
    • Don’t want to waste threads in thread pool by having them waiting on I/O
  • Avoid event handler mess by using async/await
    • Use TaskCompletionSource to hook event to lambda that sets result of task
  • Don’t wrap synchronous code in async method
  • Don’t block UI thread to wait for completion of asynchronous code

Full video

Mads Torgersen– Program Manager, C# Language, Microsoft
One of the people responsible for the async feature

Key takeaways

  • Async void is only for top-level event handlers
  • Use threadpool for CPU-bound code, but not IO-bound
  • Use TaskCompletionSource to wrap Tasks around events
  • Libraries shouldn’t lie, and should be chunky

Async void is only for event handlers

  • User: if user clicks Print button too quickly, stuff not ready

Stop using async void

  • Unless you absolutely have to

Async void only for event handlers

  • Event handlers are async void
  • In your handler, you might call your own async void method
    • Then in 2nd method, you hit await
    • That method returns
  • Event handler then also awaits
  • ** slide – with arrows **
  • Can’t predict which order the methods will resume in

Variant #2 – Exception in GetResponseAsync

  • Exception doesn’t come back to original caller
  • async void method has no Task to put exception in
  • Then UI thread crashes, because no handler

How to fix

  • Make 2nd function Task return, then await it

Example 3 – virtual methods returning void

  • E.g. override of OnNavigatedTo, LoadState
  • The override calls base
  • But OnNavigated base calls overridden LoadState
  • Again, a race condition because we hit awaits and exit methods
  • Solution
    • Can’t return Task
    • Still hand off Task from caller to callee
    • Stick result of 2nd call in variable, after calling async Task helper method
    • Then in 1st method, can await on the variable!
    • Brilliant
  • Tasks always the best way to communicate completion

Example 4 – Can’t always see when you’re doing async void

  • Lambda
  • Lambdas may map to delegate that returns void
  • If both overloads offered, it picks Task-returning
  • E.g. If you Dispatcher.RunAsync with async lambda
    • When lambda returns, the caller thinks that work is done
  • Sol’n: find another way to communicate completion
    • Factor it out into async method that returns Task
  • Search for async (), check it

Async void only for event handlers

  • Principles
    • Fire-and-forget mechanism—almost never what you want
    • Caller unable to know when async void has finished
    • Caller unable to catch exceptions
  • Guidance
    • Use only for top-level event handlers
    • Use async Task-returning methods everywhere else
    • If you need fire-and-forget, be explicit
    • When you see async lambda, verify it

Threadpool

  • User: how should I parallelize my code
    • Loading list of housing data
    • Use Threadpool, task parallel library, parallel for?
  • Diagnose/Fix
    • Users code was not CPU-bound

Threadpool – sequential

  • Sequentially, you have to wait

Threadpool – Try #1

  • Parallel.For
    • Deserialize and add to list
    • Runs on parallel cores
  • Now down to 300ms, from 500ms
  • But is it really taking 100 ms per house?
    • Nope, it’s actually I/O bound
    • Deserialize is actually blocking on I/O

Is it CPU-bound or I/O-bound?

  • CPU-bound – should parallelize
  • I/O-bound – maybe not

How it works

  • Doing two threads, two cores
  • Gradually spins up threads, as it sees first thread waiting on I/O
  • So we created more threads than we need

How to code it right

  • Parallelize I/O bound code
  • List of tasks
  • LoadFromDatabaseAsync
  • Then: await Task.WhenAll(tasks)
  • No threadpool
  • Thread should not be waiting on I/O

Threadpool – may get another bottleneck?

  • Moving off threadpool, but doing I/O in parallel may lead to I/O bottleneck if you have a large number of tasks
  • So use a queue and workers
  • Use WorkerAsync
  • Create three workers

Threadpool

  • Principles
    • CPU-bound okay on threads
    • Parallel.ForEach and Task.Run are good way to put CPU-bound work onto thread pool
    • E.g. LINQ-over-objects, computational
    • Use of threads won’t increase throughput on machine that’s under load
  • Guidelines
    • For IO-bound work, use await, rather than background threads
    • For CPU-bound work, use background threads via Parallel.ForEach or Task.Run, unless you’re writing library or scalable server-side code

Async over events

  • User
    • UI code looks like spaghetti
  • Diagnose/Fix
    • Events are the problem
    • Consider wrapping them as Tasks

Apple picking game

  • Multiple levels of events
  • Sequence of things all listed as nested lambdas
  • Ick !
  • Callback hell
  • Solution
    • State machine
    • Becomes complicated in a new way
  • Now we have everything as global events

The problem is events—they’re not going away

Solution

  • await async events
  • Trick – how to turn helper methods into async

Async over events – async helper methods

  • Use TaskCompletionSource<object>
  • Guy who creates TCS controls how task completes
  • Make your own task, rather than letting async create a task for you
  • Lambda just tells task that things are done
  • Then wire this lambda into Completed event
  • Then you await this tcs.Task
  • So—it’s about converted synchronous work with Completed handler into Task-based paradigm
  • Fantastic!

Async keyword is for creating logic around methods that are already async

  • When you want to create your own Task, use TaskCompletionSource

Wow, this is great. Learning async tricks from Mads..

Async over events

  • Principles
    • Callback-based programming, as with events, is hard
  • Guidance
    • If event handlers are largely independent, leave them as events
    • If they look like state machine, then await is maybe easier
    • To turn event into awaitable tasks, use TaskCompletionSource

Library methods shouldn’t lie

  • Signature hints at whether a method is synchronous or asynchronous

Library methods shouldn’t lie

  • Honest about synchronous
    • Some methods do actual work, occupy work
    • You should say so, synchronous
  • Synchronous methods
    • Do work
    • You’re not wasting your time waiting for me
  • Asynchronous methods
    • I’ll initiate something, but return immediately

Library methods shouldn’t lie – Example

  • Synchronous
    • PausePrint – burns CPU
  • Asynchronous
    • PausePrintAsync – await Task.Delay(10000)
    • Honest, because it returns immediately, spawns task
  • Don’t: wrap synchronous code in async method
    • Returns Task.Run (return it)
    • This method lies—it’s not really async
  • Never: wrap asynchronous code in synchronous
    • Async method that returns synchronously
    • Synchronous wrappers for asynchronous work – NO !

Dangers of wrapping synchronous in asynchronous method

  • Wrap synchronous in async
    • You’re still doing work
    • Hiding from app dev where work is being done
  • Threadpool is app-global resource
    • Scalability hurt
  • On server, spinning up threads hurts scalability
  • App is the best position to manage its threads
    • Don’t use threads in secret

Dangers of blocking – wrap asynchronous in synchronous

  • LoadAsync
  • Then wait on this in button click handler
  • And then update view
  • Rather than doing await so that handler returns immediately
  • LoadAsync works fine—creates thread
  • Blocks UI thread—bad !
  • Then LoadAsync does await and it leads to deadlock
  • Because the resumption of LoadAsync, after await, wants UI thread
  • But you’ve blocked UI thread—crap
  • A bit better in thread pool

Library methods shouldn’t lie

  • Principles
    • Threadpool is app-global resource
    • Poor use of threadpool hurts scalability
  • Guidance
    • Help callers understand how your method behaves
    • Libraries shouldn’t use threadpool in secret
    • Use async signature only for truly async methods

Async – not spinning on threads

Sync – not blocking threads

Libraries should expose chunky async APIs

  • We all know sync methods are “cheap”
    • Years of optimizations around sync methods
    • Enables refactoring at will
  • E.g. synchronous string out
    • IL is simple
  • Async method that outputs string (but doesn’t wait)
    • Body of method is 3x longer
    • Has to initialize state machine
    • Lots of plumbing
  • Important mental model
    • How many allocations are required for async state machine?
    • Allocation will eventually require GC
    • Garbage collection is what’s costly

Fast Path in awaits

  • Each async method has to allocate
    • State machine class holding method’s local variables
    • Delegate
    • Returned Task object
  • If code path doesn’t hit any awaits
    • Optimized so that state machine and delegate aren’t allocated. Just Task
  • If awaited Task has already completed, then skip actual wait
  • If you don’t have any awaits fired and you have common result (e.g. 0, 1, true, false, null, “”)
    • Compiler just grabs these pre-gen’d task and just returns it
  • You can follow this same pattern, for common return values in Tasks
    • Create wrapper

Libraries should expose chunky async APIs

  • Principles
    • Heap is an app-global resource
  • Guidance
    • Libraries should expose chunky async APIs, not chatty
    • If library has to be chatty, and GC perf is problem, and heap has lots of async allocations
      • Then optimize the fast-path
    • Generally, use async to your heart’s content and don’t worry about it
    • But just be aware of what’s going on under the hood

Consider .ConfigureAwait(false) in libraries

  • Sync context represents a “target for work”
    • E.g. DispatcherSynchronizationContext, whose .Post() does Dispatcher.BeginInvoke()
    • Sort of “where do I live”
    • When await resumes execution, it has to look at sync context to figure out what thread to run code on. E.g. On UI thread
    • Goal: after await, you should be where you were before (e.g. I’m still on the UI thread)
    • Extra level of bookkeeping can be expensive
  • Library code often doesn’t care where it’s running
    • You can ask await to not go find original context after await, but just keep running in whatever context is current
  • “Await task” uses the sync context
    • await task.ConfigureAwait(false)
    • Suppresses SyncContext.Post()

Consider .ConfigureAwait(false)

  • Principles
    • UI message-queue is app-global resource
    • Too much use will hurt UI responsiveness
  • Guidance
    • If you call chatty async APIs but doesn’t touch the UI, use ConfigureAwait(false)

Resources for async Best Practices

.NET Basics – Do Work in Background Thread to Keep GUI Responsive

One of the most important things that differentiates a “quick and dirty” application from one that has been designed well is how the application’s user interface behaves during lengthy operations.  The quick-and-dirty approach is to just do all of your work in a button’s Click event handler and not worry about the user interface.  The problem with this is that the GUI will freeze up while the application does whatever work it needs to do.

A well designed application, on the other hand, is one that is careful to do as much work as possible in background threads, keeping the GUI responsive and making sure that it makes it obvious to the user that work is going on in the background and adjusts the GUI to disallow any user actions that don’t apply until after the work finishes.

Under .NET 2.0, doing work on a background thread has become a lot easier, with the introduction of the BackgroundWorker class.  You no longer have to worry about cross-threading exceptions and checking a control’s InvokeRequired property.

A Simple Example of Using the BackgroundWorker Class

In this post, I’ll create a simple example of how you might use the BackgroundWorker class to do some work on a background thread and keep your GUI responsive.  We’ll start with a simple example that demonstrates how the GUI can become blocked and then evolve the application to make full use of the capabilities of the BackgroundWorker class.

Here are the basic players. We’ll have a FileReader class/object that reads text from a text file. And a Win Forms form with a button to initiate the file read operation and some GUI elements to display the status/results of the read operation.

Note: All code samples presented here can be found in CodePlex, at threadsafepubsub.codeplex.com

Iteration #1 – The Simplest Possible Solution

Let’s say that we just want to read a text file and return/display the number of lines found in the file. We can just make a call to our FileReader object, which returns the number of lines, and then display that number in our UI. Super simple.

This iteration is implemented in the files Form1.cs and FileReader1.cs.

Here’s what the GUI looks like.  If you click on the Read File button, you get a File Open dialog where you can select a file to read.  The file is read in and then we write out the # lines read, below the button.

001-Iter1Client

So far, so good.  This is how most simple user interfaces are written–you click on a button, which launches a Click callback, which does some work, and then returns to the caller.

Here’s what the FileReader1 class looks like, with a simple ReadTheFile method:

using System;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace ThreadSafePubSubUI
{
    public class FileReader1
    {
        // Read specified text file & return # lines
        public int ReadTheFile(string fileName)
        {
            int numLines = 0;

            using (StreamReader sr = new StreamReader(fileName))
            {
                string nextLine;
                while ((nextLine = sr.ReadLine()) != null)
                {
                    numLines++;
                }
            }

            return numLines;
        }
    }
}

And here’s the click event handler for the form: the guy that invokes ReadTheFile.

        private void btnSelect_Click(object sender, EventArgs e)
        {
            OpenFileDialog ofd = new OpenFileDialog();
            ofd.CheckFileExists = true;
            ofd.CheckPathExists = true;

            if (ofd.ShowDialog() == DialogResult.OK)
            {
                FileReader1 fr = new FileReader1();
                int numLines = fr.ReadTheFile(ofd.FileName);

                lblResults.Text = string.Format("We read {0} lines", numLines.ToString());
            }
        }

But what if the function that does the work takes a longer amount of time?  It’s pretty common for some action initiated by the user to take a little time.  What happens to the GUI while they are waiting?  We can simulate this by just adding a Thread.Sleep call in the ReadTheFile method.

            Thread.Sleep(3000);     // Simulate lengthy operation

Let’s also add a line in the btnSelect_Click method, to write a “busy” message to the GUI while we are processing.  Here is the updated click event handler:

        private void btnSelect_Click(object sender, EventArgs e)
        {
            OpenFileDialog ofd = new OpenFileDialog();
            ofd.CheckFileExists = true;
            ofd.CheckPathExists = true;

            if (ofd.ShowDialog() == DialogResult.OK)
            {
                lblResults.Text = " ... reading the file ...";
                FileReader1 fr = new FileReader1();
                int numLines = fr.ReadTheFile(ofd.FileName);

                lblResults.Text = string.Format("We read {0} lines", numLines.ToString());
            }
        }

What happens is not good.  Two bad things happen, from a user’s point of view:

  • The user interface is completely unresponsive during the file read operation
  • Our “reading the file” message is not displayed

What happened?  Well, because everything is on one thread, our user interface thread doesn’t respond to mouse clicks until ReadTheFile finishes.  Worse, even though we set the label’s Text property before we call ReadTheFile, the message loop doesn’t get a chance to process that change, and update the text, before we go out to lunch in ReadTheFile.

What we need to do to fix this is: do the file read operation on a different thread

The easiest way to do some work on a background thread, keeping the GUI responsive, is to use the BackgroundWorker class.

Iteration #2 – Using the BackgroundWorker Class

You should be doing very little actual work in GUI control event handlers like the Button.Click method.  It’s a good idea to:

  • Move code that does actual work outside of the user interface class
  • Do all work on a background thread.

We want to move code into a separate library or class, rather than having it in our Click event handler, to keep our user interface code separate from our functional code.  This is just a cleaner architecture and makes our code more maintainable, easier to test, and more extensible.

We also want to do as much work as possible on a different thread from the main thread handling the GUI.  If you do your work on the same thread, you risk locking up the user interface.  (As we saw in Iteration #1).

If you’re using the .NET Framework version 2.0 or later, the best way to do work on a background thread is to use the BackgroundWorker class.  This class gives us the ability to do some work on a background thread, provides progress and completed events “out of the box” and also ensures that these callbacks execute on the correct (GUI thread).

What do I mean by “execute on the correct thread”?  Here’s how it works.  To ensure that the GUI stays responsive, we want to do any non-trivial work on a background thread.  This thread can run in parallel to the GUI thread, so the user will still be able to interact with the GUI while the work is being done.

When the work finishes, we likely want to update something in the GUI to indicate this.  (E.g. change the text on a label to indicate that the operation is done).  Our GUI object will be notified by handling an event that the worker object fires.  But since we need to update the GUI, this event handler must be executing on the same thread as the user interface.

This last point is very important.  The core rule in Windows UI programming to remember is: the only thread that can update/change a user interface control is the thread that created it.  (This is true for Windows Forms applications, which use the Single Threaded Apartment model).

The beauty of the BackgroundWorker class is that it automatically handles all of this thread logic:

  • It does work on a background thread
  • It ensures that completed/progress events are fired on the original GUI thread

Let’s change our earlier file-reading example to use the BackgroundWorker.  This example can be found in the threadsafepubsub.codeplex.com project in the Form2/FileWorker2 classes.

Here’s the new Click event handler, where we create the background worker, attach our event handlers, and then tell it to go do some work.

private void btnSelect_Click(object sender, EventArgs e)
        private void btnSelect_Click(object sender, EventArgs e)
        {
            OpenFileDialog ofd = new OpenFileDialog();
            ofd.CheckFileExists = true;
            ofd.CheckPathExists = true;

            if (ofd.ShowDialog() == DialogResult.OK)
            {
                lblResults.Text = " ... reading the file ...";

                // Set up background worker object & hook up handlers
                BackgroundWorker bgWorker;
                bgWorker = new BackgroundWorker();
                bgWorker.DoWork += new DoWorkEventHandler(bgWorker_DoWork);
                bgWorker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bgWorker_RunWorkerCompleted);

                // Launch background thread to do the work of reading the file.  This will
                // trigger BackgroundWorker.DoWork().  Note that we pass the filename to
                // process as a parameter.
                bgWorker.RunWorkerAsync(ofd.FileName);
            }
        }

We first create an instance of the BackgroundWorker class and then wire up the DoWork and RunWorkerCompleted methods.  DoWork is the event that will fire when we call the RunWorkerAsync method.  And it will run asynchronously, in a background thread, freeing up the user interface.  Because RunWorkerAsync is launched in a background thread, control returns from the btnSelect_Click method quickly, and the UI is responsive, even while the file-read work is going on.

We also hook a handler to the RunWorkerCompleted event, which will fire when our bgWorker_DoWork method has finished doing the work.  This event, however, will execute on the original GUI thread–allowing is to update GUI elements directly within our gbWorker_RunWorkerCompleted handler.

Here’s the body of our DoWork handler.

        void bgWorker_DoWork(object sender, DoWorkEventArgs e)
        {
            FileReader2 fr = new FileReader2();

            // Filename to process was passed to RunWorkerAsync(), so it's available
            // here in DoWorkEventArgs object.
            string sFileToRead = (string)e.Argument;
            e.Result = fr.ReadTheFile(sFileToRead);
        }

Notice that we just use our earlier FileReader class to do the actual work of reading the file.  But there are two additions.

First, because this method is invoked from the BackgroundWorker object, we need to somehow get the name of the file to process.  We knew this filename back in the btnSelect_Click method and we hand it off by passing it as a parameter to RunWorkerAsync and then reading it out of the DoWorkEventArgs parameter.

Similarly, when we finish doing our work (reading the file), we need to make sure the result (# lines read) gets passed back to our RunWorkerCompleted handler.  We do this by setting the Result properly of the DoWorkEventArgs parameter.

Here’s the code for our RunWorkerCompleted event handler:

        void bgWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            if (e.Error != null)
            {
                MessageBox.Show(e.Error.Message);
            }
            else
            {
                int numLines = (int)e.Result;
                lblResults.Text = string.Format("We read {0} lines", numLines.ToString());
            }
        }

Here we see the other side of the e.Result handoff–we read the FileReader.ReadTheFile return value out of the RunWorkerCompletedEventArgs parameter.  We also check this parameter to see if an error occurred.

If you now run this example, you’ll see a couple of important things that work better than they did in iteration #1:

  • We now correctly see the “reading the file” label, indicating that work is in progress
  • While the file is being read, we can interact with the GUI normally

You can demonstrate the second part of this by clicking on the “Tell Me a Joke” button.  You’ll get a message box with a clever joke and you can then dismiss the dialog–all while the file read operation is still going on.

Iteration #3 – Application State and Cancel Logic

You might be tempted at this point to think that we’re done and our application has everything that it needs.  But we’re missing a few critical things.  Any time that you do work in a background worker thread, you should also consider:

  • Busy indicator — making it easy for the user to know when work is being done in the background
  • Application state — what can/can’t the user do while the work is in progress?
  • Progress indicator — give the user a visual sense of how much work is left to be done
  • Cancel logic — optionally, give the user a method to cancel the background work

Busy Indicator

Let’s start with the busy indicator.  It’s important to make it obvious to your users that something is happening in the background, and what that something is.

Application State

We have some subtle behavior in our current implementation that is probably not desirable.  Try the following:

  • Click on the Read File button and select a file, to initiate a file read operation
  • Before the read has completed, click on the button again and select a new file

You now have two file read operations running in parallel.  Is this really what we want?  Do we want to prohibit it?  If not, how do we handle the results of two different file read operations, when the operations complete?  How do we avoid mixing up the results?  How do we know which operation the results are coming from?  Is there a chance that the two operations will attempt to work on/with the same data?

For our purposes, let’s agree that we really only want to allow the user to do one operation at a time.  While one operation is in progress, a user cannot initiate another one.  We’ll modify the GUI to enforce this.

Progress Indicator

More than just indicating that some work is going on in the background, it would be nice to indicate how much work we’ve already done and how much work is left to do.  This lets a user judge how long the entire process will take.

Cancel Logic

Whenever you support doing some work on a background thread, you also need to consider whether a user might want to cancel this background activity.  Unless it’s something that happens quite quickly, it’s probably a good idea to allow a user to cancel the operation and return to the original state (no file is being read and they are able to select a new file to be read).

At this point, it’s probably a good idea to do a rough sketch of a state diagram, showing what a user can do and during what state:

Application State Diagram

Notice that we enter the “reading file” state when the user clicks the “Read File” button.  But while in this state, the user cannot press that button again–they either press the “Cancel” button, or we return to the original state when the file read operation completes.

Also note that we should be able to display a joke while in either state.  This confirms what we said earlier–the GUI won’t lock up during the file read operation.

Our Modified Example

Here’s how our file reader example works, after adding a progress indicator, cancel logic, and the ability to keep track of application state.  Here’s the new GUI during a file read operation:

Progress

Note that we now tell the user what file we’re reading and we display a progress indicator, showing how far into the read operation we are.  We also give them a Cancel button, allowing them to Cancel the operation before it completes normally.  Also notice that the Read File button is greyed out—the user can’t initiate another operation until the first one completes.

If the user lets the file read operation complete normally, they’ll see the following:

Success

Notice that when we finish reading the file, returning to the Idle state, we hide all of the progress/cancel widgets.  The Read File button is also enabled again.

If the user cancels the file read operation, they’ll see the following:

Cancelled

Again, all of the progress/cancel widgets are gone, since we’re back in the Idle state.  And the Read File button is available again.  But this time, we tell the user that they cancelled the operation.

The code for this iteration can be found in threadsafepubsub.codeplex.com, as Form3.cs and FileReader3.cs.

We added a couple of things at the top of the class–an enumeration to keep track of our state, and a class-level BackgroundWorker instance.  (We move this variable into class scope because our Cancel button will need access to the BackgroundWorker object.

    private enum AppStates { Idle, ReadingFile };

    private BackgroundWorker _worker;

Here’s our new Form3 constructor, where we now call a method to set the initial application state:

        public Form3()
        {
            InitializeComponent();

            // Set up initial state
            SetAppState(AppStates.Idle, null);
        }

Here’s the actual code for the new SetAppState function, as well as a helper function that sets visibility for several controls.

        // Set new application state, handling button sensitivity, labels, etc.
        private void SetAppState(AppStates newState, string filename)
        {
            switch (newState)
            {
                case AppStates.Idle:
                    // Hide progress widgets
                    SetFileReadWidgetsVisible(false);
                    btnSelect.Enabled = true;
                    break;

                case AppStates.ReadingFile:
                    // Display progress widgets & file info
                    SetFileReadWidgetsVisible(true);
                    lblProgress.Text = string.Format("Reading file: {0}", filename);
                    pbProgress.Value = 0;
                    lblResults.Text = "";
                    btnSelect.Enabled = false;
                    break;
            }
        }

        private void SetFileReadWidgetsVisible(bool visible)
        {
            lblProgress.Visible = visible;
            pbProgress.Visible = visible;
            btnCancel.Visible = visible;
        }

We’re basically just changing the visibility of the various progress widgets in the StatusStrip at the bottom of the form.  We also handle enabling/disabling the File Read button here.

The Click event handler for our File Read button is also slightly different. We add a line that sets the application state to indicate that a file is being read, we attach a handler to track progress, and we add an exception handler to ensure that the application state is set back to idle if anything goes wrong.

        private void btnSelect_Click(object sender, EventArgs e)
        {
            OpenFileDialog ofd = new OpenFileDialog();
            ofd.CheckFileExists = true;
            ofd.CheckPathExists = true;

            if (ofd.ShowDialog() == DialogResult.OK)
            {
                FileInfo fi = new FileInfo(ofd.FileName);
                SetAppState(AppStates.ReadingFile, fi.Name);

                try
                {
                    // Set up background worker object & hook up handlers
                    _worker = new BackgroundWorker();
                    _worker.DoWork += new DoWorkEventHandler(bgWorker_DoWork);
                    _worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bgWorker_RunWorkerCompleted);
                    _worker.WorkerReportsProgress = true;
                    _worker.WorkerSupportsCancellation = true;
                    _worker.ProgressChanged += new ProgressChangedEventHandler(bgWorker_ProgressChanged);

                    // Launch background thread to do the work of reading the file.  This will
                    // trigger BackgroundWorker.DoWork().  Note that we pass the filename to
                    // process as a parameter.
                    _worker.RunWorkerAsync(ofd.FileName);
                }
                catch
                {
                    SetAppState(AppStates.Idle, null);
                    throw;
                }
            }
        }

Note also that we have to explicitly tell the BackgroundWorker that it should support both progress and cancellation functionality.

We also now have a new event handler for the ProgressChanged event, which looks like this:

        // Get info on progress of file-read operation (% complete)
        void bgWorker_ProgressChanged(object sender, ProgressChangedEventArgs e)
        {
            // Just update progress bar with % complete
            pbProgress.Value = e.ProgressPercentage;
        }

This one is pretty simple—we just set the value of the progress bar, which runs from 0 to 100, to the reported % complete value.

Our DoWork handler has just a few changes.  Here is the new version:

        // Do work--runs on a background thread
        void bgWorker_DoWork(object sender, DoWorkEventArgs e)
        {
            // Note about exceptions:  If an exception originates anywhere in
            // this method, or methods that it calls, the BackgroundWorker will
            // automatically populate the Error property of the RunWorkerCompletedEventArgs
            // parameter that gets passed into the RunWorkerCompleted event handler.
            // So we can handle the exception in that method.

            FileReader3 fr = new FileReader3();

            // Filename to process was passed to RunWorkerAsync(), so it's available
            // here in DoWorkEventArgs object.
            BackgroundWorker bw = sender as BackgroundWorker;
            string sFileToRead = (string)e.Argument;

            e.Result = fr.ReadTheFile(bw, sFileToRead);

            // If operation was cancelled (triggered by CancellationPending),
            // we bailed out of ReadTheFile() early.  But still need to set
            // Cancel flag, because RunWorkerCompleted event will still fire.
            if (bw.CancellationPending)
                e.Cancel = true;
        }

I added a note to remind us that exceptions originating in this chunk of code (or on this thread) are automatically made available to us in the RunWorkerCompleted handler.

Notice also that we’re now passing the BackgroundWorker object into the ReadTheFile method.  We do this because we need access to it, within this message, to check for user cancellation and to report progress.

Finally, we see a piece of the cancellation infrastructure here.  Below is another code chunk to help us understand how the cancel operation works—the click handler for the Cancel button.

        private void btnCancel_Click(object sender, EventArgs e)
        {
            _worker.CancelAsync();
        }

This is pretty simple.  When the user clicks the Cancel button, we tell the BackgroundWorker object to initiate a cancel operation.  Here’s a summary of the entire cancel operation (what happens when):

  • User clicks Cancel button
  • btnCancel_Click handler invokes BackgroundWorker.CancelAsync on active worker object
  • Method doing actual work (reading file) periodically checks BackgroundWorker.CancellationPending and aborts if it sees this property set
  • Control returns to bgWorker_DoWork method
  • DoWork method checks CancellationPending property and sets DoWorkEventArgs.Cancel to true if operation was cancelled
  • BackgroundWorker.RunWorkerCompleted fires
  • We can check RunWorkerCompletedEventArgs.Cancelled, in our RunWorkerCompleted handler, to detect whether operation was cancelled

This is a little involved, but if you walk through the code, you’ll see how things work.

Finally, here is our RunWorkerCompleted event handler:

        void bgWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            try
            {
                if (e.Error != null)
                {
                    MessageBox.Show(e.Error.Message, "Error During File Read");
                }
                else if (e.Cancelled)
                {
                    lblResults.Text = "** Cancelled **";
                }
                else
                {
                    int numLines = (int)e.Result;
                    lblResults.Text = string.Format("We read {0} lines", numLines.ToString());
                }
            }
            finally
            {
                // State now goes back to idle
                SetAppState(AppStates.Idle, null);
            }
        }

There are just a couple of new things here.  We now check the Cancelled property and display a message if the operation was cancelled.  We also add a finally block, where we ensure that we transition back to the Idle state, whether things completed normally, the user cancelled, or there was an error.

I have one final block of code to share—the ReadTheFile method that does the actual work:

        public int ReadTheFile(BackgroundWorker bw, string fileName)
        {
            int numLines = 0;
            FileInfo fi = new FileInfo(fileName);
            long totalBytes = fi.Length;
            long bytesRead = 0;

            using (StreamReader sr = new StreamReader(fileName))
            {
                // Note: When BackgroundWorker has CancellationPending set, we bail
                // out and fall back to the _DoWork method that called us.
                string nextLine;
                while (((nextLine = sr.ReadLine()) != null) &&
                       !bw.CancellationPending)
                {
                    bytesRead += sr.CurrentEncoding.GetByteCount(nextLine);
                    numLines++;
                    int pctComplete = (int)(((double)bytesRead / (double)totalBytes)* 100);
                    bw.ReportProgress(pctComplete);
                    Thread.Sleep(10);  // ms
                }
            }

            return numLines;
        }

We’ve basically added two things here: support for cancellation and for progress reporting.

To support user-initiated cancellation, we just check to see if the operation has been cancelled, after each line in the file that we read.  The frequency with which you check for cancellation is important.  If you don’t check often enough, the application will appear to not be responding to the cancel request and the user may become frustrated.

We report progress (% complete) by invoking the ReportProgress method on the background worker.  We do this after calculating the actual progress, in terms of # bytes read so far.

A Simple .NET Twitter API Wrapper Using LINQ

In the world of software demos, doing something with Twitter has replaced Hello World as the most common target of a demo.  At the risk of polluting the world with yet another chunk of code that does something with Twitter–I’d like to play around a bit with Silverlight charting and Twitter seems a great context for demoing what is possible.

But before I can start creating a Silverlight demo, I need a basic Twitter API wrapper in .NET.  So here’s a starting point–a simple example that uses LINQ to get a list of people that you follow.  This is a good starting point for later demos.

Twitter provides a simple REST API that lets you do basically everything you’d want to do using simple HTTP GET, POST and DELETE requests.

You can learn everything you need to know about the Twitter API at the Twitter API Wiki.

Basic Concepts

I’ll assume that you generally know how Twitter works–you follow some folks, some folks follow you, and you all post status messages–which your followers can read.  That’s the beauty of Twitter–pretty simple.

But here are some things that you should know about the Twitter API.

  • How it works
    • You post an HTTP request to a URL
    • You get XML data back in the HTTP response
  • Authentication
    • Some API method calls require authentication, using HTTP Basic Authentication.
    • Any app invoking calls that require authentication will need to supply the proper credentials.
  • Rate limits
    • Your app is limited to 100 requests per hour.    (whether you’re authenticating or not)
    • You can receive a special dispensation from the Twitter gurus to be allowed up to 20,000 requests/hr.
  • Paging
    • Many API methods requires multiple requests, retrieving a page of data at a time
    • The page parameter allows you to specify which page of data to retrieve
    • The count parameter allows specifying # items per page

What happens when you hit your rate limit?  Well, basically your application (your IP address, actually) can no longer make requests of Twitter–until the rate limit resets.

The API Calls That I Use

Here are the two Twitter API calls (URLs) that I use in this example:

You can see how these work by just entering the above URLs into your browser and looking at the XML data stream that comes back.

Here’s an example of the data returned by the friends call:

Output of Friends Request

An here’s an example of the data returned from the users call:

Data Returned from Users Request

The Peep Class

Let’s start building a simple Twitter API wrapper in .NET with a very simple class to encapsulate information about a single user–either yourself, a follower, or someone that you follow.  This doesn’t cover absolutely everything that we can find out about a Twitter user, but encapsulates some of the basic stuff that we care about.

(For these examples, I’m using Visual Studio 2008 — C# 3.0).

Here’s the code for the Peep class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }
    }
}

Super simple class, made much easier through the user of C#’s automatic properties.  As an example, using me as a Twitter user, my ScreenName would be “spsexton” and my Name would be “Sean Sexton”.

The Peeps Class

Now that we have an object that wraps a “peep”, let’s create a special class that represents a collection of Peep instances–or “peeps”.  For example, an instances of Peeps could be used to store a list of everyone that we follow (or everyone that follows us).

We’ll use our old friend, the List(T) class, from System.Collections.Generic, which implements a strongly typed collection.

Basically, a collection of Peep objects will look like this:  List<Peep>.  But we’ll create a subclass so that we can add a static method for building up a list of everyone that we follow.

Here’s the full code for Peeps.cs, followed by an explanation of how we do things.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Peep collection class
    /// </summary>
    public class Peeps : List<Peep>
    {
        // Partial Twitter API
        private const string getFriendsURI = "http://twitter.com/statuses/friends/{0}.xml?page={1}";

        /// <summary>
        /// Return list of Peeps followed by a specified person
        /// </summary>
        ///
<param name="ScreenName">The Twitter username, e.g. spsexton</param>
        /// <returns></returns>
        public static Peeps PeopleFollowedBy(string ScreenName, out int RateLimit, out int LimitRemaining)
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("PeopleFollowedBy: Invalid ScreenName");

            int nPageNum = 1;
            int userCount = 0;      // # users read on last call

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docFriends;
            Peeps peeps = new Peeps();

            // Retrieve people I'm following, 100 people at a time
            // (each call to Twitter API results in one "page" of results--up to 100 users)
            try
            {
                do
                {
                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

                    // Manually create an HTTP request, so that we can pull information out of the
                    // headers in the response.  (Then later constitute the XDocument).
                    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(getFriendsURI, ScreenName, nPageNum));
                    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                    TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                    XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                    docFriends = XDocument.Load(reader);

                    IEnumerable<XElement> users = docFriends.Elements("users").Elements("user");

                    userCount = users.Count();
                    if (userCount > 0)
                    {
                        List<Peep> nextPage = (from user in users
                                               orderby (string)user.Element("screen_name")
                                               select new Peep
                                               {
                                                   ID = (int)user.Element("id"),
                                                   ScreenName = (string)user.Element("screen_name"),
                                                   Name = (string)user.Element("name"),
                                                   Location = (string)user.Element("location"),
                                                   Description = (string)user.Element("description"),
                                                   ProfileImageURL = TwitterUtility.UriFromString((string)user.Element("profile_image_url")),
                                                   URL = TwitterUtility.UriFromString((string)user.Element("url")),
                                                   NumFollowers = (int)user.Element("followers_count"),
                                                   LastUpdateDateTime = TwitterUtility.SafeUpdateDateTime(user.Element("status")),
                                                   LastUpdateText = TwitterUtility.SafeUpdateText(user.Element("status"))
                                               }).ToList();

                        peeps.AddRange(nextPage);
                    }
                    nPageNum++;
                } while (userCount > 0);
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }
            finally
            {
                RateLimit = rateLimit;
                LimitRemaining = limitRemaining;
            }

            return peeps;
        }
    }
}

Notice that all we have in the Peeps class at this point is a static method, PeopleFollowedBy, that returns a collection of Peep objects, one for each person that the specified screen name follows.

The first thing that you’ll notice about the code is a loop where we get consecutive pages from the Twitter \statuses\friends\screenname.xml page.  By default, you get only 100  users at a time when invoking this URL.  So the easiest way to get all people that someone follows (their “friends”), is to request consecutive pages until you get one back that contains no users.

At each step through the loop, we construct a List<Peep> object from the XML results and then add that collection to a master collection (which this function will return).

Before we look at the code at the top of the loop that constructs an HTTP request to get the next page, take a look at the commented out line at the top of the loop:

                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

This is actually the simplest way to get the results of the Twitter API calll into an XDocument, and ready for querying.  Using this single line, you could replace the next five lines of code that end with another XDocument.Load.  What’s going on here is the core of what we want to do–load up an XDocument from the URI that represents the Twitter API call.

But in my code, I go to a little more effort to create an HttpWebRequest and then get the HttpWebResponse for that request.  I do this solely for the purpose of getting Twitter rate limit information out of the header of the response.  If you’ll recall, Twitter has a 100 calls per hour rate limit by default.  The nice thing is that the API tells us the rate limit, as well as the # calls remaining, after each request.  So we read that from the header and keep track of it.

For now, this rate limit information is just returned to the caller.  But my intent is to use it in a future example to actually slow down my Twitter calls, as needed.  This will be helpful when we want to batch up a large # of Twitter calls, but we don’t want to risk maxing out our rate limit.  More on this later.

I get the rate limit info from the header in the TwitterUtility.GetInfoFromResponse method, described below.

Here’s Where LINQ Comes In

Now for the LINQ part.  Once we load up the XDocument from Twitter’s response, we can build up a collection of XElement objects corresponding to the list of users in the XML stream.  But this isn’t quite what we want..  To get the data from the XElement objects into our List<Peep> collection, we need to do a simple LINQ query.

(Thanks to a post by Wally McClure on the basic idea for the LINQ query: Calling the Twitter API in C#).

The LINQ query is pretty simple–we grab each user element from the XML stream and create a Peep object for that user.  We initialize all the fields of the new Peep object for which we can get data from this XML stream.  (We can’t get NumFollowing or NumUpdates–we’ll have to make a different API call to get that information).

In most cases, we’re asking for the value of an XML element that is a child of the <user> element.  E.g. the <id> element.  And we call helper methods in some cases, since the elements that we’re trying to read might by null.  (Actually, I haven’t tested this thoroughly–some of the other elements might occasionally be null and so it wouldn’t be a bad idea to use a “safe” accessor method on all of the elements).

Finally, we need to convert the result of our query–which is IEnumerable<Peep>–to a List<Peep> by calling the ToList() method.  Then we add this new list to the master list that we are building.

Handling the Rate Limit Exception

One final thing remains for this function–handling the case when we exceed our rate limit.  I added a simple handler, to make it a little more obvious to the client that we’ve exceeded our rate limit, rather than letting the underlying WebException bubble up.  This is a little bit sloppy, since there are other things that might throw a WebException.  But this is a good start at giving the caller a little info on the rate limit issue.

The Helper Class

Here is the full code for the TwitterUtility class, which just contains a handful of helper methods that we make use of in the Peeps and Peep (see below) classes.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Various global utility methods for Twitter library
    /// </summary>
    public class TwitterUtility
    {
        /// <summary>
        /// Convert a string to a valid Uri object (or null)
        /// </summary>
        ///
<param name="sUri">String represent Uri, e.g. http://blahblah.com</param>
        /// <returns></returns>
        public static Uri UriFromString(string sUri)
        {
            return ((sUri != null) && (sUri.Length > 0)) ? new Uri(sUri) : null;
        }

        /// <summary>
        /// Pull a couple fields out of the header--specifically, the Twitter API rate limit info.
        /// </summary>
        ///
<param name="resp"></param>
        ///
<param name="rateLimit"></param>
        ///
<param name="limitRemaining"></param>
        public static void GetInfoFromResponse(WebResponse resp, out int rateLimit, out int limitRemaining)
        {
            rateLimit = 0;
            limitRemaining = 0;

            for (int i = 0; i < resp.Headers.Keys.Count; i++)
            {
                string s = resp.Headers.GetKey(i);
                if (s == "X-RateLimit-Limit")
                {
                    rateLimit = int.Parse(resp.Headers.GetValues(i).First());
                }
                if (s == "X-RateLimit-Remaining")
                {
                    limitRemaining = int.Parse(resp.Headers.GetValues(i).First());
                }
            }
        }

        /// <summary>
        /// Parse twitter date string into .NET DateTime
        /// </summary>
        ///
<param name="dateString"></param>
        /// <returns></returns>
        public static DateTime ParseTwitterDate(string dateString)
        {
            return DateTime.ParseExact(dateString, "ddd MMM dd HH:mm:ss zzz yyyy", CultureInfo.InvariantCulture);
        }

        /// <summary>
        /// Return a valid DateTime for status.created_at and handle the case
        /// of the status element not being present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static DateTime SafeUpdateDateTime(XElement user)
        {
            DateTime creAt = new DateTime();        // Default constructor is 1/1/0001 12AM

            if (user != null)
            {
                XElement elemCreAt = user.Element("created_at");
                if (elemCreAt != null)
                {
                    creAt = ParseTwitterDate((string)elemCreAt);
                }
            }

            return creAt;
        }

        /// <summary>
        /// Return a valid update text string, whether or not the <status> element
        /// was present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static string SafeUpdateText(XElement user)
        {
            string sText = "";

            if (user != null)
            {
                XElement elemText = user.Element("text");
                if (elemText != null)
                {
                    sText = (string)elemText;
                }
            }

            return sText;
        }
    }
}

Here’s what’s in this class:

  • UriFromString — “Safe” assignment, creating either a valid Uri object, or null
  • GetInfoFromResponse — Read the Headers collection from the HTTP response to pull out the rate limit and # remaining API calls
  • ParseTwitterDate — Parse the funky Twitter date/time string into a DateTime object
  • SafeUpdateDateTime — Another “safe” method, filling in a DateTime object only if the created_at element exists
  • SafeUpdateText — And a “safe” assignment from the text element

(Note: Both the <created_at> and <text> elements are under the <status> element).

NumFollowing and NumUpdates

My goal when I started throwing together this example was to fully populate the Peep class that I listed at the top of the post.  This includes not just # of followers for everybody in my “friends” list, but the # of people that they follow, and their total updates.  I can get everything from the API call that we just saw–the “friends” call.  But to get # following and # updates, I need to make a different call:

http://twitter.com/users/show.xml?screen_name=screenname

This method returns a bunch of info about a particular user, including # following and # updates.  (See the XML output at the top of the post).

So the obvious thing to do would be to add an assignment in our LINQ query, calling a helper method to go off and call this other Twitter API method, right?  For each user, we could call show.xml and get the remaining two fields.

The problem with including this 2nd Twitter call in the LINQ is that we’ll blow out our Twitter rate limit.  We only get 100 requests per hour, so we’d run out of steam trying to flesh out the first 100 users.  (And any Twitter user worth his salt follows at least 100 people).

So what is to be done?  For now, I add code to the Peep class (see below) to get the remaining info on a “peep by peep” basis, rather than getting everything all at once.  This is a bit of a cop out, since we leave it up to the client to decide how often to call this method.

I’ll do a 2nd post where I actually add code to make these additional calls, but in a “rate limit safe” manner.  (Hint–we’ll use timers to slow down our use of the Twitter API).

So until we get some “rate limit smart” code, here’s the expanded code for Peep.cs, including a method that calls show.xml to get the additional info.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        private const string userInfoURI = "http://twitter.com/users/show.xml?screen_name={0}";

        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }

        /// <summary>
        /// Calculate NumFollowing & NumUpdates, since these two fields'
        /// data isn't available from an API call that gets a list of
        /// multiple users, but must be retrieve for each user individually.
        /// </summary>
        public void CalcAddlInfo()
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("CalcNumFollowing: Invalid ScreenName");

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docUser;

            try
            {
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(userInfoURI, ScreenName));
                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                docUser = XDocument.Load(reader);

                XElement user = docUser.Element("user");

                NumFollowing = (int)user.Element("friends_count");
                NumUpdates = (int)user.Element("statuses_count");
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }

        }

        /// <summary>
        /// Variant that just takes screen name, rather than acting on existing
        /// instance.
        /// </summary>
        ///
<param name="ScreenName"></param>
        public void CalcAddlInfo(string screenName)
        {
            Peep p = new Peep { ScreenName = screenName };
            p.CalcAddlInfo();
        }
    }
}

The CalcAddlInfo method just invokes the show.xml API call and then reads the friends_count and statuses_count fields.

Hey, Where’s My GUI?

Ok, so at this point, we have the following code chunks:

  • Peep.cs — wraps data for a single user and gives us method to get a few additional fields
  • Peeps.cs — subclasses List<Peep> and gives us method to get list of people that we follow
  • TwitterUtility.cs — some miscellaneous helper functions

Now let’s throw a simple Win Forms GUI on top of these classes, so that we can test things out.  Here’s what the final result will look like, after calling our PeopleFollowedBy method:

ourgui

The code for this couldn’t be simpler.  I just call the Peeps.PeopleFollowedBy method, which returns an instance of the Peeps class (which is really a List<Peep>).  And then I bind the collection to a DataGridView.  Presto.

(If you’re paying attention, you’ll also notice that my rate limit is listed as 20,000/hr, rather than the default 100/hr.  This is because I requested “white list” status and the Twitter crew kindly consented to bump my rate limit.  This applies whenever I’m making API calls from my specific IP address).

For what it’s worth, here’s the event handler code for the Load Grid button in the GUI.  Notice that I also make a test call to the CalcAddlInfo method–so we can step through the call in the debugger and see how it works.

        private void btnLoadGrid_Click(object sender, EventArgs e)
        {
            int rateLimit;
            int limitRemaining;

            Cursor = Cursors.WaitCursor;
            Peeps peeps = Peeps.PeopleFollowedBy(txtScreenName.Text, out rateLimit, out limitRemaining);

            lblNumFollowing.Text = peeps.Count.ToString();
            lblRateLimit.Text = rateLimit.ToString();
            lblRemaining.Text = limitRemaining.ToString();

            peeps[5].CalcAddlInfo();

            dgvPeeps.DataSource = peeps;
            Cursor = Cursors.Default;
        }

Wrapping Up and Next Steps

That’s all there is to it–the process of making calls to the Twitter API from .NET code and consuming the resulting XML data using LINQ is pretty straightforward.

Where am I headed next, on my way to doing some Silverlight demos?  Here’s what I’ll cover in the next post:

  • Making my API methods smart about rate limits, using timers to acquire Twitter data quietly in the background–at a rate that is just slow enough to not trigger the rate limit.
  • Possibly caching the data on the client

If we were going to productize the code that I’ve presented, we’d also want to think about:

  • Moving the Twitter API stuff out of the data objects and into a separate class
  • Better exception handling
  • Wrapping the entire Twitter API, rather than just a couple methods