Sean’s Stuff

4 April, 2009

A Simple .NET Twitter API Wrapper Using LINQ

Filed under: LINQ — Sean @ 6:06 am
Tags: , , , ,

In the world of software demos, doing something with Twitter has replaced Hello World as the most common target of a demo.  At the risk of polluting the world with yet another chunk of code that does something with Twitter–I’d like to play around a bit with Silverlight charting and Twitter seems a great context for demoing what is possible.

But before I can start creating a Silverlight demo, I need a basic Twitter API wrapper in .NET.  So here’s a starting point–a simple example that uses LINQ to get a list of people that you follow.  This is a good starting point for later demos.

Twitter provides a simple REST API that lets you do basically everything you’d want to do using simple HTTP GET, POST and DELETE requests.

You can learn everything you need to know about the Twitter API at the Twitter API Wiki.

Basic Concepts

I’ll assume that you generally know how Twitter works–you follow some folks, some folks follow you, and you all post status messages–which your followers can read.  That’s the beauty of Twitter–pretty simple.

But here are some things that you should know about the Twitter API.

  • How it works
    • You post an HTTP request to a URL
    • You get XML data back in the HTTP response
  • Authentication
    • Some API method calls require authentication, using HTTP Basic Authentication.
    • Any app invoking calls that require authentication will need to supply the proper credentials.
  • Rate limits
    • Your app is limited to 100 requests per hour.    (whether you’re authenticating or not)
    • You can receive a special dispensation from the Twitter gurus to be allowed up to 20,000 requests/hr.
  • Paging
    • Many API methods requires multiple requests, retrieving a page of data at a time
    • The page parameter allows you to specify which page of data to retrieve
    • The count parameter allows specifying # items per page

What happens when you hit your rate limit?  Well, basically your application (your IP address, actually) can no longer make requests of Twitter–until the rate limit resets.

The API Calls That I Use

Here are the two Twitter API calls (URLs) that I use in this example:

You can see how these work by just entering the above URLs into your browser and looking at the XML data stream that comes back.

Here’s an example of the data returned by the friends call:

Output of Friends Request

An here’s an example of the data returned from the users call:

Data Returned from Users Request

The Peep Class

Let’s start building a simple Twitter API wrapper in .NET with a very simple class to encapsulate information about a single user–either yourself, a follower, or someone that you follow.  This doesn’t cover absolutely everything that we can find out about a Twitter user, but encapsulates some of the basic stuff that we care about.

(For these examples, I’m using Visual Studio 2008 — C# 3.0).

Here’s the code for the Peep class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }
    }
}

Super simple class, made much easier through the user of C#’s automatic properties.  As an example, using me as a Twitter user, my ScreenName would be “spsexton” and my Name would be “Sean Sexton”.

The Peeps Class

Now that we have an object that wraps a “peep”, let’s create a special class that represents a collection of Peep instances–or “peeps”.  For example, an instances of Peeps could be used to store a list of everyone that we follow (or everyone that follows us).

We’ll use our old friend, the List(T) class, from System.Collections.Generic, which implements a strongly typed collection.

Basically, a collection of Peep objects will look like this:  List<Peep>.  But we’ll create a subclass so that we can add a static method for building up a list of everyone that we follow.

Here’s the full code for Peeps.cs, followed by an explanation of how we do things.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Peep collection class
    /// </summary>
    public class Peeps : List<Peep>
    {
        // Partial Twitter API
        private const string getFriendsURI = "http://twitter.com/statuses/friends/{0}.xml?page={1}";

        /// <summary>
        /// Return list of Peeps followed by a specified person
        /// </summary>
        ///
<param name="ScreenName">The Twitter username, e.g. spsexton</param>
        /// <returns></returns>
        public static Peeps PeopleFollowedBy(string ScreenName, out int RateLimit, out int LimitRemaining)
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("PeopleFollowedBy: Invalid ScreenName");

            int nPageNum = 1;
            int userCount = 0;      // # users read on last call

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docFriends;
            Peeps peeps = new Peeps();

            // Retrieve people I'm following, 100 people at a time
            // (each call to Twitter API results in one "page" of results--up to 100 users)
            try
            {
                do
                {
                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

                    // Manually create an HTTP request, so that we can pull information out of the
                    // headers in the response.  (Then later constitute the XDocument).
                    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(getFriendsURI, ScreenName, nPageNum));
                    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                    TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                    XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                    docFriends = XDocument.Load(reader);

                    IEnumerable<XElement> users = docFriends.Elements("users").Elements("user");

                    userCount = users.Count();
                    if (userCount > 0)
                    {
                        List<Peep> nextPage = (from user in users
                                               orderby (string)user.Element("screen_name")
                                               select new Peep
                                               {
                                                   ID = (int)user.Element("id"),
                                                   ScreenName = (string)user.Element("screen_name"),
                                                   Name = (string)user.Element("name"),
                                                   Location = (string)user.Element("location"),
                                                   Description = (string)user.Element("description"),
                                                   ProfileImageURL = TwitterUtility.UriFromString((string)user.Element("profile_image_url")),
                                                   URL = TwitterUtility.UriFromString((string)user.Element("url")),
                                                   NumFollowers = (int)user.Element("followers_count"),
                                                   LastUpdateDateTime = TwitterUtility.SafeUpdateDateTime(user.Element("status")),
                                                   LastUpdateText = TwitterUtility.SafeUpdateText(user.Element("status"))
                                               }).ToList();

                        peeps.AddRange(nextPage);
                    }
                    nPageNum++;
                } while (userCount > 0);
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }
            finally
            {
                RateLimit = rateLimit;
                LimitRemaining = limitRemaining;
            }

            return peeps;
        }
    }
}

Notice that all we have in the Peeps class at this point is a static method, PeopleFollowedBy, that returns a collection of Peep objects, one for each person that the specified screen name follows.

The first thing that you’ll notice about the code is a loop where we get consecutive pages from the Twitter \statuses\friends\screenname.xml page.  By default, you get only 100  users at a time when invoking this URL.  So the easiest way to get all people that someone follows (their “friends”), is to request consecutive pages until you get one back that contains no users.

At each step through the loop, we construct a List<Peep> object from the XML results and then add that collection to a master collection (which this function will return).

Before we look at the code at the top of the loop that constructs an HTTP request to get the next page, take a look at the commented out line at the top of the loop:

                    // Example of constituting XDocument directly from the URI
                    // docFriends = XDocument.Load(string.Format(getFriendsURI, ScreenName, nPageNum));

This is actually the simplest way to get the results of the Twitter API calll into an XDocument, and ready for querying.  Using this single line, you could replace the next five lines of code that end with another XDocument.Load.  What’s going on here is the core of what we want to do–load up an XDocument from the URI that represents the Twitter API call.

But in my code, I go to a little more effort to create an HttpWebRequest and then get the HttpWebResponse for that request.  I do this solely for the purpose of getting Twitter rate limit information out of the header of the response.  If you’ll recall, Twitter has a 100 calls per hour rate limit by default.  The nice thing is that the API tells us the rate limit, as well as the # calls remaining, after each request.  So we read that from the header and keep track of it.

For now, this rate limit information is just returned to the caller.  But my intent is to use it in a future example to actually slow down my Twitter calls, as needed.  This will be helpful when we want to batch up a large # of Twitter calls, but we don’t want to risk maxing out our rate limit.  More on this later.

I get the rate limit info from the header in the TwitterUtility.GetInfoFromResponse method, described below.

Here’s Where LINQ Comes In

Now for the LINQ part.  Once we load up the XDocument from Twitter’s response, we can build up a collection of XElement objects corresponding to the list of users in the XML stream.  But this isn’t quite what we want..  To get the data from the XElement objects into our List<Peep> collection, we need to do a simple LINQ query.

(Thanks to a post by Wally McClure on the basic idea for the LINQ query: Calling the Twitter API in C#).

The LINQ query is pretty simple–we grab each user element from the XML stream and create a Peep object for that user.  We initialize all the fields of the new Peep object for which we can get data from this XML stream.  (We can’t get NumFollowing or NumUpdates–we’ll have to make a different API call to get that information).

In most cases, we’re asking for the value of an XML element that is a child of the <user> element.  E.g. the <id> element.  And we call helper methods in some cases, since the elements that we’re trying to read might by null.  (Actually, I haven’t tested this thoroughly–some of the other elements might occasionally be null and so it wouldn’t be a bad idea to use a “safe” accessor method on all of the elements).

Finally, we need to convert the result of our query–which is IEnumerable<Peep>–to a List<Peep> by calling the ToList() method.  Then we add this new list to the master list that we are building.

Handling the Rate Limit Exception

One final thing remains for this function–handling the case when we exceed our rate limit.  I added a simple handler, to make it a little more obvious to the client that we’ve exceeded our rate limit, rather than letting the underlying WebException bubble up.  This is a little bit sloppy, since there are other things that might throw a WebException.  But this is a good start at giving the caller a little info on the rate limit issue.

The Helper Class

Here is the full code for the TwitterUtility class, which just contains a handful of helper methods that we make use of in the Peeps and Peep (see below) classes.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    /// <summary>
    /// Various global utility methods for Twitter library
    /// </summary>
    public class TwitterUtility
    {
        /// <summary>
        /// Convert a string to a valid Uri object (or null)
        /// </summary>
        ///
<param name="sUri">String represent Uri, e.g. http://blahblah.com</param>
        /// <returns></returns>
        public static Uri UriFromString(string sUri)
        {
            return ((sUri != null) && (sUri.Length > 0)) ? new Uri(sUri) : null;
        }

        /// <summary>
        /// Pull a couple fields out of the header--specifically, the Twitter API rate limit info.
        /// </summary>
        ///
<param name="resp"></param>
        ///
<param name="rateLimit"></param>
        ///
<param name="limitRemaining"></param>
        public static void GetInfoFromResponse(WebResponse resp, out int rateLimit, out int limitRemaining)
        {
            rateLimit = 0;
            limitRemaining = 0;

            for (int i = 0; i < resp.Headers.Keys.Count; i++)
            {
                string s = resp.Headers.GetKey(i);
                if (s == "X-RateLimit-Limit")
                {
                    rateLimit = int.Parse(resp.Headers.GetValues(i).First());
                }
                if (s == "X-RateLimit-Remaining")
                {
                    limitRemaining = int.Parse(resp.Headers.GetValues(i).First());
                }
            }
        }

        /// <summary>
        /// Parse twitter date string into .NET DateTime
        /// </summary>
        ///
<param name="dateString"></param>
        /// <returns></returns>
        public static DateTime ParseTwitterDate(string dateString)
        {
            return DateTime.ParseExact(dateString, "ddd MMM dd HH:mm:ss zzz yyyy", CultureInfo.InvariantCulture);
        }

        /// <summary>
        /// Return a valid DateTime for status.created_at and handle the case
        /// of the status element not being present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static DateTime SafeUpdateDateTime(XElement user)
        {
            DateTime creAt = new DateTime();        // Default constructor is 1/1/0001 12AM

            if (user != null)
            {
                XElement elemCreAt = user.Element("created_at");
                if (elemCreAt != null)
                {
                    creAt = ParseTwitterDate((string)elemCreAt);
                }
            }

            return creAt;
        }

        /// <summary>
        /// Return a valid update text string, whether or not the <status> element
        /// was present.
        /// </summary>
        ///
<param name="user">Represents status element (child of user element)</param>
        /// <returns></returns>
        public static string SafeUpdateText(XElement user)
        {
            string sText = "";

            if (user != null)
            {
                XElement elemText = user.Element("text");
                if (elemText != null)
                {
                    sText = (string)elemText;
                }
            }

            return sText;
        }
    }
}

Here’s what’s in this class:

  • UriFromString — “Safe” assignment, creating either a valid Uri object, or null
  • GetInfoFromResponse — Read the Headers collection from the HTTP response to pull out the rate limit and # remaining API calls
  • ParseTwitterDate — Parse the funky Twitter date/time string into a DateTime object
  • SafeUpdateDateTime — Another “safe” method, filling in a DateTime object only if the created_at element exists
  • SafeUpdateText — And a “safe” assignment from the text element

(Note: Both the <created_at> and <text> elements are under the <status> element).

NumFollowing and NumUpdates

My goal when I started throwing together this example was to fully populate the Peep class that I listed at the top of the post.  This includes not just # of followers for everybody in my “friends” list, but the # of people that they follow, and their total updates.  I can get everything from the API call that we just saw–the “friends” call.  But to get # following and # updates, I need to make a different call:

http://twitter.com/users/show.xml?screen_name=screenname

This method returns a bunch of info about a particular user, including # following and # updates.  (See the XML output at the top of the post).

So the obvious thing to do would be to add an assignment in our LINQ query, calling a helper method to go off and call this other Twitter API method, right?  For each user, we could call show.xml and get the remaining two fields.

The problem with including this 2nd Twitter call in the LINQ is that we’ll blow out our Twitter rate limit.  We only get 100 requests per hour, so we’d run out of steam trying to flesh out the first 100 users.  (And any Twitter user worth his salt follows at least 100 people).

So what is to be done?  For now, I add code to the Peep class (see below) to get the remaining info on a “peep by peep” basis, rather than getting everything all at once.  This is a bit of a cop out, since we leave it up to the client to decide how often to call this method.

I’ll do a 2nd post where I actually add code to make these additional calls, but in a “rate limit safe” manner.  (Hint–we’ll use timers to slow down our use of the Twitter API).

So until we get some “rate limit smart” code, here’s the expanded code for Peep.cs, including a method that calls show.xml to get the additional info.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace TwitterLibGUI
{
    public class Peep
    {
        private const string userInfoURI = "http://twitter.com/users/show.xml?screen_name={0}";

        public string ScreenName { get; set; }
        public string Name { get; set; }
        public string Location { get; set; }
        public string Description { get; set; }
        public Uri ProfileImageURL { get; set; }
        public Uri URL { get; set; }
        public int ID { get; set; }

        public int NumFollowers { get; set; }
        public int NumFollowing { get; set; }
        public int NumUpdates { get; set; }

        public string LastUpdateText { get; set; }
        public DateTime LastUpdateDateTime { get; set; }

        public new string ToString()
        {
            return string.Format("{0} - {1}", ScreenName, Name);
        }

        /// <summary>
        /// Calculate NumFollowing & NumUpdates, since these two fields'
        /// data isn't available from an API call that gets a list of
        /// multiple users, but must be retrieve for each user individually.
        /// </summary>
        public void CalcAddlInfo()
        {
            if ((ScreenName == null) || (ScreenName == ""))
                throw new ArgumentException("CalcNumFollowing: Invalid ScreenName");

            int rateLimit = 0;          // Max # API calls per hour
            int limitRemaining = 0;     // # API calls remaining

            XDocument docUser;

            try
            {
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format(userInfoURI, ScreenName));
                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                TwitterUtility.GetInfoFromResponse(resp, out rateLimit, out limitRemaining);
                XmlReader reader = XmlReader.Create(resp.GetResponseStream());
                docUser = XDocument.Load(reader);

                XElement user = docUser.Element("user");

                NumFollowing = (int)user.Element("friends_count");
                NumUpdates = (int)user.Element("statuses_count");
            }
            catch (WebException xcp)
            {
                throw new ApplicationException(
                    string.Format("Twitter rate limit exceeded, max of {0}/hr allowed. Remaining = {1}",
                        rateLimit,
                        limitRemaining),
                    xcp);
            }

        }

        /// <summary>
        /// Variant that just takes screen name, rather than acting on existing
        /// instance.
        /// </summary>
        ///
<param name="ScreenName"></param>
        public void CalcAddlInfo(string screenName)
        {
            Peep p = new Peep { ScreenName = screenName };
            p.CalcAddlInfo();
        }
    }
}

The CalcAddlInfo method just invokes the show.xml API call and then reads the friends_count and statuses_count fields.

Hey, Where’s My GUI?

Ok, so at this point, we have the following code chunks:

  • Peep.cs — wraps data for a single user and gives us method to get a few additional fields
  • Peeps.cs — subclasses List<Peep> and gives us method to get list of people that we follow
  • TwitterUtility.cs — some miscellaneous helper functions

Now let’s throw a simple Win Forms GUI on top of these classes, so that we can test things out.  Here’s what the final result will look like, after calling our PeopleFollowedBy method:

ourgui

The code for this couldn’t be simpler.  I just call the Peeps.PeopleFollowedBy method, which returns an instance of the Peeps class (which is really a List<Peep>).  And then I bind the collection to a DataGridView.  Presto.

(If you’re paying attention, you’ll also notice that my rate limit is listed as 20,000/hr, rather than the default 100/hr.  This is because I requested “white list” status and the Twitter crew kindly consented to bump my rate limit.  This applies whenever I’m making API calls from my specific IP address).

For what it’s worth, here’s the event handler code for the Load Grid button in the GUI.  Notice that I also make a test call to the CalcAddlInfo method–so we can step through the call in the debugger and see how it works.

        private void btnLoadGrid_Click(object sender, EventArgs e)
        {
            int rateLimit;
            int limitRemaining;

            Cursor = Cursors.WaitCursor;
            Peeps peeps = Peeps.PeopleFollowedBy(txtScreenName.Text, out rateLimit, out limitRemaining);

            lblNumFollowing.Text = peeps.Count.ToString();
            lblRateLimit.Text = rateLimit.ToString();
            lblRemaining.Text = limitRemaining.ToString();

            peeps[5].CalcAddlInfo();

            dgvPeeps.DataSource = peeps;
            Cursor = Cursors.Default;
        }

Wrapping Up and Next Steps

That’s all there is to it–the process of making calls to the Twitter API from .NET code and consuming the resulting XML data using LINQ is pretty straightforward.

Where am I headed next, on my way to doing some Silverlight demos?  Here’s what I’ll cover in the next post:

  • Making my API methods smart about rate limits, using timers to acquire Twitter data quietly in the background–at a rate that is just slow enough to not trigger the rate limit.
  • Possibly caching the data on the client

If we were going to productize the code that I’ve presented, we’d also want to think about:

  • Moving the Twitter API stuff out of the data objects and into a separate class
  • Better exception handling
  • Wrapping the entire Twitter API, rather than just a couple methods
About these ads

12 Comments »

  1. Here’s a quick quiz question–what’s the nature of the data in the final Peeps collection, with respect to sorting? Is it sorted or not?

    Comment by Sean — 6 April, 2009 @ 6:03 pm | Reply

  2. Sean, thankU, I think you’ve blogged the most extensive use of the Twitter API to-date. Furthermore, you’ve provided some real insight for other C# developers interested in observing how the 3.0 language features are used for a real-world objective.

    Comment by Clinton Gallagher — 26 May, 2009 @ 10:22 pm | Reply

  3. Hi,

    it doesn’t work. It increase only my remaining rate limit, but i’ve got no output.

    Do you know any reasons?

    Best regards, mojo

    Comment by mojo_theone — 9 February, 2010 @ 3:56 pm | Reply

  4. dose it work with visaul studio 2010
    becuse i have
    Erro The name ‘Properties’ does not exist in the current context

    Comment by momo — 26 March, 2011 @ 11:26 am | Reply

    • Yes, it should work in VS2010. You have to change one small thing, after converting your project. The auto-generated settings class in Settings.Designer.cs ends up with the wrong namespace. It used the “Default Namespace” that was set in project properties. In the copy of the project that I checked in, this is TwitterCount. It should actually be TwitterLib (I changed it at some point). Just go into the Project Properties for the TwitterLibGUI project and change the default namespace from TwitterCount to TwitterLib. Then rebuild the project–everything should compile fine now. You can also open Settings.Designer.cs in the code editor to verify that it has the correct namespace now.

      Comment by Sean — 26 March, 2011 @ 3:24 pm | Reply

  5. m try to use following twitter API, but unable to get user statuses. tel me whats an error??

    WebClient connectTwitter = new WebClient();
    connectTwitter.DownloadStringCompleted += new DownloadStringCompletedEventHandler(connectTwitter_DownloadStringCompleted);

    connectTwitter.DownloadStringAsync(new Uri(“https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=BarackObama”));

    Comment by Adeel — 6 September, 2013 @ 6:38 pm | Reply

  6. i got en error in this line in peeps.cs class
    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

    Comment by Azher Ali — 7 May, 2014 @ 8:55 am | Reply

    • What error do you get?

      Comment by Sean — 7 May, 2014 @ 11:18 am | Reply

      • sir in the obove code u didn,t describe how to authenticate so thats why i got error in peeps.cs class at line
        HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
        sir plz kindly tell me how do i authenticate the code are missing
        thanx

        Comment by Azher Ali — 7 May, 2014 @ 7:59 pm

      • There is no information in this post on authentication because the API methods mentioned did not require authentication at the time that this post is written. More to the point, though, the Twitter API is likely quite different than it was four years ago. If you’re just copying/pasting this code without trying to understand what it’s doing, it’s not going to just work for you. The intent of the post was to show how to use the API that existed in 2009 and to teach you how to use LINQ to dive into the results that are returned. You can do something similiar using whatever Twitter provides today as an API.

        Comment by Sean — 7 May, 2014 @ 9:38 pm

  7. thanx sir :)

    Comment by Azher Ali — 8 May, 2014 @ 6:49 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 484 other followers

%d bloggers like this: