In the world of software demos, doing something with Twitter has replaced Hello World as the most common target of a demo.  At the risk of polluting the world with yet another chunk of code that does something with Twitter–I’d like to play around a bit with Silverlight charting and Twitter seems a great context for demoing what is possible.

But before I can start creating a Silverlight demo, I need a basic Twitter API wrapper in .NET.  So here’s a starting point–a simple example that uses LINQ to get a list of people that you follow.  This is a good starting point for later demos.

Twitter provides a simple REST API that lets you do basically everything you’d want to do using simple HTTP GET, POST and DELETE requests.

You can learn everything you need to know about the Twitter API at the Twitter API Wiki.

Basic Concepts

I’ll assume that you generally know how Twitter works–you follow some folks, some folks follow you, and you all post status messages–which your followers can read.  That’s the beauty of Twitter–pretty simple.

But here are some things that you should know about the Twitter API.

• How it works
  • You post an HTTP request to a URL
  • You get XML data back in the HTTP response
• Authentication
  • Some API method calls require authentication, using HTTP Basic Authentication.
  • Any app invoking calls that require authentication will need to supply the proper credentials.
• Rate limits
  • Your app is limited to 100 requests per hour.    (whether you’re authenticating or not)
  • You can receive a special dispensation from the Twitter gurus to be allowed up to 20,000 requests/hr.
• Paging
  • Many API methods requires multiple requests, retrieving a page of data at a time
  • The page parameter allows you to specify which page of data to retrieve
  • The count parameter allows specifying # items per page

What happens when you hit your rate limit?  Well, basically your application (your IP address, actually) can no longer make requests of Twitter–until the rate limit resets.

The API Calls That I Use

Here are the two Twitter API calls (URLs) that I use in this example:

• Get list of people someone follows
  • http://twitter.com/statuses/friends/screenname.xml?page=1
• Get info about a specific user
  • http://twitter.com/users/show.xml?screen_name=screenname

You can see how these work by just entering the above URLs into your browser and looking at the XML data stream that comes back.

Here’s an example of the data returned by the friends call:

An here’s an example of the data returned from the users call:

The Peep Class

Let’s start building a simple Twitter API wrapper in .NET with a very simple class to encapsulate information about a single user–either yourself, a follower, or someone that you follow.  This doesn’t cover absolutely everything that we can find out about a Twitter user, but encapsulates some of the basic stuff that we care about.

(For these examples, I’m using Visual Studio 2008 — C# 3.0).

Here’s the code for the Peep class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }
    }
}

Super simple class, made much easier through the user of C#’s automatic properties.  As an example, using me as a Twitter user, my ScreenName would be “spsexton” and my Name would be “Sean Sexton”.

The Peeps Class

Now that we have an object that wraps a “peep”, let’s create a special class that represents a collection of Peep instances–or “peeps”.  For example, an instances of Peeps could be used to store a list of everyone that we follow (or everyone that follows us).

We’ll use our old friend, the List(T) class, from System.Collections.Generic, which implements a strongly typed collection.

Basically, a collection of Peep objects will look like this:  List<Peep>.  But we’ll create a subclass so that we can add a static method for building up a list of everyone that we follow.

Here’s the full code for Peeps.cs, followed by an explanation of how we do things.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Peep collection class
    /// </summary>
    public class Peeps : List<Peep>
    {
        // Partial Twitter API
        private const string getFriendsURI = "http://twitter.com/statuses/friends/{0}.xml?page={1}";

        /// <summary>
        /// Return list of Peeps followed by a specified person
        /// </summary>
        ///
<param name="ScreenName">The Twitter username, e.g. spsexton</param>
        /// <returns></returns>
        public static Peeps PeopleFollowedBy(string ScreenName, out int RateLimit, out int LimitRemaining)
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("PeopleFollowedBy: Invalid ScreenName");

            int nPageNum = 1;
            int userCount = 0;      // # users read on last call

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docFriends;
            Peeps peeps = new Peeps();

            // Retrieve people I'm following, 100 people at a time
            // (each call to Twitter API results in one "page" of results--up to 100 users)
            try
            {
                do
                {
                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

                    // Manually create an HTTP request, so that we can pull information out of the
                    // headers in the response.  (Then later constitute the XDocument).
                    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(getFriendsURI, ScreenName, nPageNum));
                    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                    TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                    XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                    docFriends = XDocument.Load(reader);

                    IEnumerable<XElement> users = docFriends.Elements("users").Elements("user");

                    userCount = users.Count();
                    if (userCount > 0)
                    {
                        List<Peep> nextPage = (from user in users
                                               orderby (string)user.Element("screen_name")
                                               select new Peep
                                               {
                                                   ID = (int)user.Element("id"),
                                                   ScreenName = (string)user.Element("screen_name"),
                                                   Name = (string)user.Element("name"),
                                                   Location = (string)user.Element("location"),
                                                   Description = (string)user.Element("description"),
                                                   ProfileImageURL = TwitterUtility.UriFromString((string)user.Element("profile_image_url")),
                                                   URL = TwitterUtility.UriFromString((string)user.Element("url")),
                                                   NumFollowers = (int)user.Element("followers_count"),
                                                   LastUpdateDateTime = TwitterUtility.SafeUpdateDateTime(user.Element("status")),
                                                   LastUpdateText = TwitterUtility.SafeUpdateText(user.Element("status"))
                                               }).ToList();

                        peeps.AddRange(nextPage);
                    }
                    nPageNum++;
                } while (userCount > 0);
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }
            finally
            {
                RateLimit = rateLimit;
                LimitRemaining = limitRemaining;
            }

            return peeps;
        }
    }
}

Notice that all we have in the Peeps class at this point is a static method, PeopleFollowedBy, that returns a collection of Peep objects, one for each person that the specified screen name follows.

The first thing that you’ll notice about the code is a loop where we get consecutive pages from the Twitter \statuses\friends\screenname.xml page.  By default, you get only 100  users at a time when invoking this URL.  So the easiest way to get all people that someone follows (their “friends”), is to request consecutive pages until you get one back that contains no users.

At each step through the loop, we construct a List<Peep> object from the XML results and then add that collection to a master collection (which this function will return).

Before we look at the code at the top of the loop that constructs an HTTP request to get the next page, take a look at the commented out line at the top of the loop:

                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

This is actually the simplest way to get the results of the Twitter API calll into an XDocument, and ready for querying.  Using this single line, you could replace the next five lines of code that end with another XDocument.Load.  What’s going on here is the core of what we want to do–load up an XDocument from the URI that represents the Twitter API call.

But in my code, I go to a little more effort to create an HttpWebRequest and then get the HttpWebResponse for that request.  I do this solely for the purpose of getting Twitter rate limit information out of the header of the response.  If you’ll recall, Twitter has a 100 calls per hour rate limit by default.  The nice thing is that the API tells us the rate limit, as well as the # calls remaining, after each request.  So we read that from the header and keep track of it.

For now, this rate limit information is just returned to the caller.  But my intent is to use it in a future example to actually slow down my Twitter calls, as needed.  This will be helpful when we want to batch up a large # of Twitter calls, but we don’t want to risk maxing out our rate limit.  More on this later.

I get the rate limit info from the header in the TwitterUtility.GetInfoFromResponse method, described below.

Here’s Where LINQ Comes In

Now for the LINQ part.  Once we load up the XDocument from Twitter’s response, we can build up a collection of XElement objects corresponding to the list of users in the XML stream.  But this isn’t quite what we want..  To get the data from the XElement objects into our List<Peep> collection, we need to do a simple LINQ query.

(Thanks to a post by Wally McClure on the basic idea for the LINQ query: Calling the Twitter API in C#).

The LINQ query is pretty simple–we grab each user element from the XML stream and create a Peep object for that user.  We initialize all the fields of the new Peep object for which we can get data from this XML stream.  (We can’t get NumFollowing or NumUpdates–we’ll have to make a different API call to get that information).

In most cases, we’re asking for the value of an XML element that is a child of the <user> element.  E.g. the <id> element.  And we call helper methods in some cases, since the elements that we’re trying to read might by null.  (Actually, I haven’t tested this thoroughly–some of the other elements might occasionally be null and so it wouldn’t be a bad idea to use a “safe” accessor method on all of the elements).

Finally, we need to convert the result of our query–which is IEnumerable<Peep>–to a List<Peep> by calling the ToList() method.  Then we add this new list to the master list that we are building.

Handling the Rate Limit Exception

One final thing remains for this function–handling the case when we exceed our rate limit.  I added a simple handler, to make it a little more obvious to the client that we’ve exceeded our rate limit, rather than letting the underlying WebException bubble up.  This is a little bit sloppy, since there are other things that might throw a WebException.  But this is a good start at giving the caller a little info on the rate limit issue.

The Helper Class

Here is the full code for the TwitterUtility class, which just contains a handful of helper methods that we make use of in the Peeps and Peep (see below) classes.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Various global utility methods for Twitter library
    /// </summary>
    public class TwitterUtility
    {
        /// <summary>
        /// Convert a string to a valid Uri object (or null)
        /// </summary>
        ///
<param name="sUri">String represent Uri, e.g. http://blahblah.com</param>
        /// <returns></returns>
        public static Uri UriFromString(string sUri)
        {
            return ((sUri != null) && (sUri.Length > 0)) ? new Uri(sUri) : null;
        }

        /// <summary>
        /// Pull a couple fields out of the header--specifically, the Twitter API rate limit info.
        /// </summary>
        ///
<param name="resp"></param>
        ///
<param name="rateLimit"></param>
        ///
<param name="limitRemaining"></param>
        public static void GetInfoFromResponse(WebResponse resp, out int rateLimit, out int limitRemaining)
        {
            rateLimit = 0;
            limitRemaining = 0;

            for (int i = 0; i < resp.Headers.Keys.Count; i++)
            {
                string s = resp.Headers.GetKey(i);
                if (s == "X-RateLimit-Limit")
                {
                    rateLimit = int.Parse(resp.Headers.GetValues(i).First());
                }
                if (s == "X-RateLimit-Remaining")
                {
                    limitRemaining = int.Parse(resp.Headers.GetValues(i).First());
                }
            }
        }

        /// <summary>
        /// Parse twitter date string into .NET DateTime
        /// </summary>
        ///
<param name="dateString"></param>
        /// <returns></returns>
        public static DateTime ParseTwitterDate(string dateString)
        {
            return DateTime.ParseExact(dateString, "ddd MMM dd HH:mm:ss zzz yyyy", CultureInfo.InvariantCulture);
        }

        /// <summary>
        /// Return a valid DateTime for status.created_at and handle the case
        /// of the status element not being present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static DateTime SafeUpdateDateTime(XElement user)
        {
            DateTime creAt = new DateTime();        // Default constructor is 1/1/0001 12AM

            if (user != null)
            {
                XElement elemCreAt = user.Element("created_at");
                if (elemCreAt != null)
                {
                    creAt = ParseTwitterDate((string)elemCreAt);
                }
            }

            return creAt;
        }

        /// <summary>
        /// Return a valid update text string, whether or not the <status> element
        /// was present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static string SafeUpdateText(XElement user)
        {
            string sText = "";

            if (user != null)
            {
                XElement elemText = user.Element("text");
                if (elemText != null)
                {
                    sText = (string)elemText;
                }
            }

            return sText;
        }
    }
}

Here’s what’s in this class:

• UriFromString — “Safe” assignment, creating either a valid Uri object, or null
• GetInfoFromResponse — Read the Headers collection from the HTTP response to pull out the rate limit and # remaining API calls
• ParseTwitterDate — Parse the funky Twitter date/time string into a DateTime object
• SafeUpdateDateTime — Another “safe” method, filling in a DateTime object only if the created_at element exists
• SafeUpdateText — And a “safe” assignment from the text element

(Note: Both the <created_at> and <text> elements are under the <status> element).

NumFollowing and NumUpdates

My goal when I started throwing together this example was to fully populate the Peep class that I listed at the top of the post.  This includes not just # of followers for everybody in my “friends” list, but the # of people that they follow, and their total updates.  I can get everything from the API call that we just saw–the “friends” call.  But to get # following and # updates, I need to make a different call:

http://twitter.com/users/show.xml?screen_name=screenname

This method returns a bunch of info about a particular user, including # following and # updates.  (See the XML output at the top of the post).

So the obvious thing to do would be to add an assignment in our LINQ query, calling a helper method to go off and call this other Twitter API method, right?  For each user, we could call show.xml and get the remaining two fields.

The problem with including this 2nd Twitter call in the LINQ is that we’ll blow out our Twitter rate limit.  We only get 100 requests per hour, so we’d run out of steam trying to flesh out the first 100 users.  (And any Twitter user worth his salt follows at least 100 people).

So what is to be done?  For now, I add code to the Peep class (see below) to get the remaining info on a “peep by peep” basis, rather than getting everything all at once.  This is a bit of a cop out, since we leave it up to the client to decide how often to call this method.

I’ll do a 2nd post where I actually add code to make these additional calls, but in a “rate limit safe” manner.  (Hint–we’ll use timers to slow down our use of the Twitter API).

So until we get some “rate limit smart” code, here’s the expanded code for Peep.cs, including a method that calls show.xml to get the additional info.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        private const string userInfoURI = "http://twitter.com/users/show.xml?screen_name={0}";

        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }

        /// <summary>
        /// Calculate NumFollowing & NumUpdates, since these two fields'
        /// data isn't available from an API call that gets a list of
        /// multiple users, but must be retrieve for each user individually.
        /// </summary>
        public void CalcAddlInfo()
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("CalcNumFollowing: Invalid ScreenName");

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docUser;

            try
            {
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(userInfoURI, ScreenName));
                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                docUser = XDocument.Load(reader);

                XElement user = docUser.Element("user");

                NumFollowing = (int)user.Element("friends_count");
                NumUpdates = (int)user.Element("statuses_count");
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }

        }

        /// <summary>
        /// Variant that just takes screen name, rather than acting on existing
        /// instance.
        /// </summary>
        ///
<param name="ScreenName"></param>
        public void CalcAddlInfo(string screenName)
        {
            Peep p = new Peep { ScreenName = screenName };
            p.CalcAddlInfo();
        }
    }
}

The CalcAddlInfo method just invokes the show.xml API call and then reads the friends_count and statuses_count fields.

Hey, Where’s My GUI?

Ok, so at this point, we have the following code chunks:

• Peep.cs — wraps data for a single user and gives us method to get a few additional fields
• Peeps.cs — subclasses List<Peep> and gives us method to get list of people that we follow
• TwitterUtility.cs — some miscellaneous helper functions

Now let’s throw a simple Win Forms GUI on top of these classes, so that we can test things out.  Here’s what the final result will look like, after calling our PeopleFollowedBy method:

The code for this couldn’t be simpler.  I just call the Peeps.PeopleFollowedBy method, which returns an instance of the Peeps class (which is really a List<Peep>).  And then I bind the collection to a DataGridView.  Presto.

(If you’re paying attention, you’ll also notice that my rate limit is listed as 20,000/hr, rather than the default 100/hr.  This is because I requested “white list” status and the Twitter crew kindly consented to bump my rate limit.  This applies whenever I’m making API calls from my specific IP address).

For what it’s worth, here’s the event handler code for the Load Grid button in the GUI.  Notice that I also make a test call to the CalcAddlInfo method–so we can step through the call in the debugger and see how it works.

        private void btnLoadGrid_Click(object sender, EventArgs e)
        {
            int rateLimit;
            int limitRemaining;

            Cursor = Cursors.WaitCursor;
            Peeps peeps = Peeps.PeopleFollowedBy(txtScreenName.Text, out rateLimit, out limitRemaining);

            lblNumFollowing.Text = peeps.Count.ToString();
            lblRateLimit.Text = rateLimit.ToString();
            lblRemaining.Text = limitRemaining.ToString();

            peeps[5].CalcAddlInfo();

            dgvPeeps.DataSource = peeps;
            Cursor = Cursors.Default;
        }

Wrapping Up and Next Steps

That’s all there is to it–the process of making calls to the Twitter API from .NET code and consuming the resulting XML data using LINQ is pretty straightforward.

Where am I headed next, on my way to doing some Silverlight demos?  Here’s what I’ll cover in the next post:

• Making my API methods smart about rate limits, using timers to acquire Twitter data quietly in the background–at a rate that is just slow enough to not trigger the rate limit.
• Possibly caching the data on the client

If we were going to productize the code that I’ve presented, we’d also want to think about:

• Moving the Twitter API stuff out of the data objects and into a separate class
• Better exception handling
• Wrapping the entire Twitter API, rather than just a couple methods