Sean’s Stuff

5 January, 2009

Using HttpWebRequest for Asynchronous Downloads

Filed under: Miscellaneous — Sean @ 6:06 am
Tags: , , ,

I’ve occasionally had a desire to test downloading a file via HTTP or FTP from a server—either to measure performance, or to stress test the server by kicking off a large number of simultaneous downloads.  Here’s a little Win Forms client that allows you to download a single file from a server, using either HTTP or FTP.  It shows download progress and displays the average transfer rate, in kb/sec.  It also demonstrates how to use the HttpWebRequest and FtpWebRequest classes in System.Net to do file downloads.

As an added bonus, this app is a nice example of doing a little work on a background thread and then firing callbacks back to the GUI thread to report progress.  This is done using the BeginXxx/EndXxx pattern, as well as using the Invoke method the ensure that GUI updating is done on the correct thread.  I always forget the exact syntax for this, so it’s nice to have it here to refer to.

The bulk of this code comes directly from the MSDN documentation for the HttpWebRequest.BeginGetResponse method.  I’ve created a little client app around it, adding some GUI elements to show progress.  I’ve also extended it to support downloads using FTP.

I include code snippets in this article, but you can download the entire Visual Studio 2008 solution here.

The End Result

When we’re done, we’ll have a little Win Forms app that lets us enter an HTTP or FTP path to a file and then downloads that file.  During the download, we see the progress, as well as the average transfer rate.

Download Stress Test application

For the moment, the application doesn’t actually write the file locally.  Instead, it just downloads the entire file, throwing away the data that it downloaded.  The intent is to stress the server and measure the transfer speed—not to actually get a copy of the file.

If we were to use HTTP and specify an HTML file to download, we’d basically be doing the same thing that a web browser does—downloading a web page from the server to the client.  In the example above, I download a 1.9MB Powerpoint file from the PDC conference, just so that we have something a little larger than a web page and can see some progress.

Using FTP Instead of HTTP

My little application does FTP, as well as HTTP.  If you enter an FTP-based URI, rather than an HTTP-based one, we automatically switch to using FTP to download the file.  Before the download can start, however, we need to ask the user for credentials to use to log into the FTP site.

FTP Credentials

Once we’ve gotten the FTP credentials, the download runs in much the same way that the HTTP-based download ran.

Downloading a file from an FTP server

In this case, I’m downloading an ISO image of the first CD of a Fedora distribution.  Note that the FTP response string starts with “213”, which gives file status and represents a successful response from the FTP server.  The second part of the response string is the size of the file, in bytes.  In the case of HTTP, the response was just “OK”.

Where Are We?

So what do we really have here?  A little program that downloads a single file without really writing it anywhere.  At this point, we have something that’s mildly useful for testing a server, since it tells us the transfer rate.  Furthermore, we can launch a bunch of these guys in parallel and download the same file many times in parallel, to stress the server.  (Ideally, the application would let us pick #-of-simultaneous-downloads and just kick them all off, but that’s an enhancement to be added later).

Diving Into the Source Code

More interesting than what this little program does is how you go about using the HttpWebRequest and FtpWebRequest classes to do the actual work.

Here’s a quick look at the C# solution:

Files in Solution

There’s really not much here—the main form (DownloadStressTestForm), the FTP credentials form (GetCredentialsForm) and a little helper class used to pass data around between asynchronous methods.

Most of the code lives in DownloadStressTestForm.cs.  Ideally, we’d split this out into the GUI pieces and the actual plumbing code that does the work of downloading the files.  But this is just a quick-and-dirty project.

Push the Button

Let’s take a look at the code that fires when you click the Get File button.

        private void btnGetFile_Click(object sender, EventArgs e)
        {
            try
            {
                lblDownloadComplete.Visible = false;

                WebRequest req = null;
                WebRequestState reqState = null;
                Uri fileURI = new Uri(txtURI.Text);

                if (fileURI.Scheme == Uri.UriSchemeHttp)
                {
                    req = (HttpWebRequest)HttpWebRequest.Create(fileURI);
                    reqState = new HttpWebRequestState(BUFFER_SIZE);
                    reqState.request = req;
                }
                else if (fileURI.Scheme == Uri.UriSchemeFtp)
                {
                    // Get credentials
                    GetCredentialsForm frmCreds = new GetCredentialsForm();
                    DialogResult result = frmCreds.ShowDialog();
                    if (result == DialogResult.OK)
                    {
                        req = (FtpWebRequest)FtpWebRequest.Create(fileURI);
                        req.Credentials = new NetworkCredential(frmCreds.Username, frmCreds.Password);
                        reqState = new FtpWebRequestState(BUFFER_SIZE);

                        // Set FTP-specific stuff
                        ((FtpWebRequest)req).KeepAlive = false;

                        // First thing we do is get file size.  2nd step, done later,
                        // will be to download actual file.
                        ((FtpWebRequest)req).Method = WebRequestMethods.Ftp.GetFileSize;
                        reqState.FTPMethod = WebRequestMethods.Ftp.GetFileSize;

                        reqState.request = req;
                    }
                    else
                        req = null;	// abort

                }
                else
                    MessageBox.Show("URL must be either http://xxx or ftp://xxxx");

                if (req != null)
                {
                    reqState.fileURI = fileURI;
                    reqState.respInfoCB = new ResponseInfoDelegate(SetResponseInfo);
                    reqState.progCB = new ProgressDelegate(Progress);
                    reqState.doneCB = new DoneDelegate(Done);
                    reqState.transferStart = DateTime.Now;

                    // Start the asynchronous request.
                    IAsyncResult result =
                      (IAsyncResult)req.BeginGetResponse(new AsyncCallback(RespCallback), reqState);
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show(string.Format("EXC in button1_Click(): {0}", ex.Message));
            }
        }

The basic goal here is to create an instance of either the HttpWebRequest or FtpWebRequest class.  This is done by calling the corresponding Create method, and passing it the URI that the user entered.  Note that we use the Uri class to figure out if the user is entering an HTTP or an FTP URI.  We create an instance of the base class, WebRequest, which we’ll use to kick everything off.

We also create an instance of a class used to store some state information, either HttpWebRequestState or FtpWebRequestState.  These classes both derive from WebRequestState and are defined in this project, in WebRequestState.cs.

The idea of this state object is that we’ll hand it off to the asynchronous method that we use to do the actual download.  It then will get passed back to the callback that fires when an asynchronous method completes.  Think of it as a little suitcase of stuff that we want to carry around with us and hand off between the asynchronous methods.

Notice that if we’re doing an FTP transfer, we first pop up the credentials dialog to get the Username and Password from the user.  We then store those credentials in the FtpWebRequest object.

There’s one other difference between HTTP and FTP.  In the case of HTTP, we’ll fire off a single web request, with a GET command, to download the file.  But for FTP, we actually first send a command to read the file size, followed by the command to actually download the file.  To accomplish this, we set the Method property of the FtpWebRequest to WebRequestMethods.Ftp.GetFileSize.  We don’t set this property for HttpWebRequest because it just defaults to the GET command, which is what we want.

Towards the end of this function, you’ll see that I’m loading up the suitcase—setting the various properties of the WebRequestState object.  Along with the URI, we set up some delegates to point back to three callbacks in the DownloadStressTestForm class—SetResponseInfo, Progress, and Done.  These are the callbacks that actually update our user interface—when things start, during the file transfer, and when the file has finished downloading.

Finally, we call the BeginGetResponse method to actually launch the download.  Here, we specify a response callback—the method that will get called, not when the download has completed, but just when we get the actual HTTP response, or when the FTP command completes.  In the case of HTTP, we first get the response packet and then start reading the actual file using a stream that we get from the response.

What’s important here is that we push the work into the background, on another thread, as soon as possible.  We don’t do much work in the button click event handler before calling BeginGetResponse.  And this method is asynchronous—so we return control to the GUI immediately.  From this point on, we will only update the GUI in response to a callback.

Callbacks to Update the GUI

I mentioned the three callbacks above that we use to update the user interface—SetResponseInfo, Progress, and Done.  Here’s the declaration of the delegate types:

    public delegate void ResponseInfoDelegate(string statusDescr, string contentLength);
    public delegate void ProgressDelegate(int totalBytes, double pctComplete, double transferRate);
    public delegate void DoneDelegate();

And here are the bodies of each of these three callbacks, as implemented in DownloadStressTestForm.

        /// <summary>
        /// Response info callback, called after we get HTTP response header.
        /// Used to set info in GUI about max download size.
        /// </summary>
        private void SetResponseInfo(string statusDescr, string contentLength)
        {
            if (this.InvokeRequired)
            {
                ResponseInfoDelegate del = new ResponseInfoDelegate(SetResponseInfo);
                this.Invoke(del, new object[] { statusDescr, contentLength });
            }
            else
            {
                lblStatusDescr.Text = statusDescr;
                lblContentLength.Text = contentLength;
            }
        }

        /// <summary>
        /// Progress callback, called when we've read another packet of data.
        /// Used to set info in GUI on % complete & transfer rate.
        /// </summary>
        private void Progress(int totalBytes, double pctComplete, double transferRate)
        {
            if (this.InvokeRequired)
            {
                ProgressDelegate del = new ProgressDelegate(Progress);
                this.Invoke(del, new object[] { totalBytes, pctComplete, transferRate });
            }
            else
            {
                lblBytesRead.Text = totalBytes.ToString();
                progressBar1.Value = (int)pctComplete;
                lblRate.Text = transferRate.ToString("f0");
            }
        }

        /// <summary>
        /// GUI-updating callback called when download has completed.
        /// </summary>
        private void Done()
        {
            if (this.InvokeRequired)
            {
                DoneDelegate del = new DoneDelegate(Done);
                this.Invoke(del, new object[] { });
            }
            else
            {
                progressBar1.Value = 0;
                lblDownloadComplete.Visible = true;
            }
        }

This is pretty simple stuff.  To start with, notice the common pattern in each method, where we check InvokeRequired.  Remember the primary rule about updating controls in a user interface and asynchronous programming: the controls must be updated by the same thread that created them.  InvokeRequired tells us if we’re on the right thread or not.  If not, we use the Invoke method to recursively call ourselves, but on the thread that created the control (the one that owns the window handle).

Make note of this InvokeRequired / Invoke pattern.  You’ll use it whenever you’re doing background work on another thread and then you want to return some information back to the GUI.

The work that these callbacks do is very simple.  SetResponseInfo is called when we first get the reponse packet, as we start downloading the file.  We get an indication of the file size, which we write to the GUI.  Progress is called for each packet that we download.  We update the labels that indicate # bytes received and average transfer rate, as well as the main progress bar.  Done is called when we’re all done transfering the file.

The Response Callback

Let’s go back to where we called the WebRequest.BeginGetResponse method.  We we called this method, we specified our RespCallback as the method to get invoked when the response packet was received.  Here’s the code:

        /// <summary>
        /// Main response callback, invoked once we have first Response packet from
        /// server.  This is where we initiate the actual file transfer, reading from
        /// a stream.
        /// </summary>
        private static void RespCallback(IAsyncResult asyncResult)
        {
            try
            {
                // Will be either HttpWebRequestState or FtpWebRequestState
                WebRequestState reqState = ((WebRequestState)(asyncResult.AsyncState));
                WebRequest req = reqState.request;
                string statusDescr = "";
                string contentLength = "";

                // HTTP
                if (reqState.fileURI.Scheme == Uri.UriSchemeHttp)
                {
                    HttpWebResponse resp = ((HttpWebResponse)(req.EndGetResponse(asyncResult)));
                    reqState.response = resp;
                    statusDescr = resp.StatusDescription;
                    reqState.totalBytes = reqState.response.ContentLength;
                    contentLength = reqState.response.ContentLength.ToString();   // # bytes
                }

                // FTP part 1 - response to GetFileSize command
                else if ((reqState.fileURI.Scheme == Uri.UriSchemeFtp) &&
                         (reqState.FTPMethod == WebRequestMethods.Ftp.GetFileSize))
                {
                    // First FTP command was GetFileSize, so this 1st response is the size of
                    // the file.
                    FtpWebResponse resp = ((FtpWebResponse)(req.EndGetResponse(asyncResult)));
                    statusDescr = resp.StatusDescription;
                    reqState.totalBytes = resp.ContentLength;
                    contentLength = resp.ContentLength.ToString();   // # bytes
                }

                // FTP part 2 - response to DownloadFile command
                else if ((reqState.fileURI.Scheme == Uri.UriSchemeFtp) &&
                         (reqState.FTPMethod == WebRequestMethods.Ftp.DownloadFile))
                {
                    FtpWebResponse resp = ((FtpWebResponse)(req.EndGetResponse(asyncResult)));
                    reqState.response = resp;
                }

                else
                    throw new ApplicationException("Unexpected URI");

                // Get this info back to the GUI -- max # bytes, so we can do progress bar
                if (statusDescr != "")
                    reqState.respInfoCB(statusDescr, contentLength);

                // FTP part 1 done, need to kick off 2nd FTP request to get the actual file
                if ((reqState.fileURI.Scheme == Uri.UriSchemeFtp) && (reqState.FTPMethod == WebRequestMethods.Ftp.GetFileSize))
                {
                    // Note: Need to create a new FtpWebRequest, because we're not allowed to change .Method after
                    // we've already submitted the earlier request.  I.e. FtpWebRequest not recyclable.
                    // So create a new request, moving everything we need over to it.
                    FtpWebRequest req2 = (FtpWebRequest)FtpWebRequest.Create(reqState.fileURI);
                    req2.Credentials = req.Credentials;
                    req2.UseBinary = true;
                    req2.KeepAlive = true;
                    req2.Method = WebRequestMethods.Ftp.DownloadFile;

                    reqState.request = req2;
                    reqState.FTPMethod = WebRequestMethods.Ftp.DownloadFile;

                    // Start the asynchronous request, which will call back into this same method
                    IAsyncResult result =
                      (IAsyncResult)req2.BeginGetResponse(new AsyncCallback(RespCallback), reqState);
                }
                else    // HTTP or FTP part 2 -- we're ready for the actual file download
                {
                    // Set up a stream, for reading response data into it
                    Stream responseStream = reqState.response.GetResponseStream();
                    reqState.streamResponse = responseStream;

                    // Begin reading contents of the response data
                    IAsyncResult ar = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallback), reqState);
                }

                return;
            }
            catch (Exception ex)
            {
                MessageBox.Show(string.Format("EXC in RespCallback(): {0}", ex.Message));
            }
        }

The first thing that we do in this method is to open our suitcase–our WebRequestState object, which comes back in the AsyncState property of the IAsyncResult.

The other main thing that we do in this method is to get the actual WebResponse object.  This contains the information that we actually got back from the server.  We do this by calling the EndGetResponse method.

Notice the standard Begin/End pattern for asynchronous programming here.  We could have done all of this synchronously, by calling GetResponse on the original HttpWebRequest (or FtpWebRequest object).  GetResponse would have returned an HttpWebResponse (or FtpWebResponse object).  Instead, we call BeginGetResponse to launch the asynchronous method and then call EndGetResponse in the callback to get the actual result—the WebResponse object.

At this point, the first thing that we want from the response packet is an indication of the length of the file that we’re downloading.  We get that from the ContentLength property.

It’s also at this point that we call the ResponseInfo delegate, passing it the status string and content length, to update the GUI.  (Using the respInfoCB field in the WebRequestState object).

Let’s ignore FTP for the moment and look at the final main thing that we do in this method—get a stream object and kick off a read of the first packet.  We get the stream from that WebReponse object and then go asynchronous again by calling the BeginRead method.  Are you seeing a pattern yet?  Again, if we wanted to do everything synchronously, we could just set up a loop here and call the stream’s Read method to read each buffer of data.  But instead, we fire up an asynchronous read, specifying our method that should be called when we get the first packet/buffer of data—ReadCallback.

FTP Download, Step 2

Let’s go back to how we’re doing FTP.  Remember that we set the FtpWebRequest.Method property to GetFileSize.  And in ReadCallback, if we see that we just did that first command, we send the file size back to the GUI.  And then we’re ready to launch the 2nd FTP command, which is DownloadFile.  We do this by creating a 2nd FtpWebRequest and calling the BeginGetResponse method again.  And once again, when the asynchronous method completes, we’ll get control back in ReadCallback.  We don’t risk recursing indefinitely because we store an indication of which command we’re doing in our suitcase—in WebRequestState.FTPMethod.

Gettin’ the Data

Finally, let’s take a look at the code where we actually get a chunk of data from the server.  First, a quick note about buffer size.  Notice that when I called BeginRead, I specified a buffer size using the BUFFER_SIZE constant.  For the record, I’m using a value of 1448 here, which is based on the size of a typical TCP packet (packet size less some header info).  We could really use any value here that we liked—it just seemed reasonable to ask for the data a packet at a time.

Here’s the code for our read callback, which first fires when the first packet is received, after calling BeginRead.

        /// <summary>
        /// Main callback invoked in response to the Stream.BeginRead method, when we have some data.
        /// </summary>
        private static void ReadCallback(IAsyncResult asyncResult)
        {
            try
            {
                // Will be either HttpWebRequestState or FtpWebRequestState
                WebRequestState reqState = ((WebRequestState)(asyncResult.AsyncState));

                Stream responseStream = reqState.streamResponse;

                // Get results of read operation
                int bytesRead = responseStream.EndRead(asyncResult);

                // Got some data, need to read more
                if (bytesRead > 0)
                {
                    // Report some progress, including total # bytes read, % complete, and transfer rate
                    reqState.bytesRead += bytesRead;
                    double pctComplete = ((double)reqState.bytesRead / (double)reqState.totalBytes) * 100.0f;

                    // Note: bytesRead/totalMS is in bytes/ms.  Convert to kb/sec.
                    TimeSpan totalTime = DateTime.Now - reqState.transferStart;
                    double kbPerSec = (reqState.bytesRead * 1000.0f) / (totalTime.TotalMilliseconds * 1024.0f);

                    reqState.progCB(reqState.bytesRead, pctComplete, kbPerSec);

                    // Kick off another read
                    IAsyncResult ar = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallback), reqState);
                    return;
                }

                // EndRead returned 0, so no more data to be read
                else
                {
                    responseStream.Close();
                    reqState.response.Close();
                    reqState.doneCB();
                }
            }
            catch (Exception ex)
            {
                MessageBox.Show(string.Format("EXC in ReadCallback(): {0}", ex.Message));
            }
        }

As I’m so fond of saying, this is pretty simple stuff.  Once again, we make use of recursion, because we’re asynchronously reading a packet at a time.  We get the stream object out of our suitcase and then call EndRead to get an indication of how many bytes were read.  This is the indicator that will tell us when we’re done reading the data—in which case # bytes read will be 0.

If we’re all done reading the data, we close down our stream and WebResponse object, before calling our final GUI callback to tell the GUI that we’re done.

But if we did read some data, we first call our progress callback to tell the GUI that we got another packet and then we fire off another BeginRead.  (Which will, of course, lead to our landing back in the ReadCallback method when the next packet completes).

You can see that we’re passing back some basic info to the GUI—total # bytes read, the % complete, and the calculated average transfer rate, in KB per second.

If we actually cared about the data itself, we could find it in our suitcase—in WebRequestState.bufferRead.  This is just a byte array that we specified when we called BeginRead.  In the case of this application, we don’t care about the actual data, so we don’t do anything with it.

Opening the Suitcase

We’ve looked at basically all the code, except for the implementation of the WebRequestState class that we’ve been using as our “suitcase”.  Here’s the base class:

    /// <summary>
    /// Base class for state object that gets passed around amongst async methods
    /// when doing async web request/response for data transfer.  We store basic
    /// things that track current state of a download, including # bytes transfered,
    /// as well as some async callbacks that will get invoked at various points.
    /// </summary>
    abstract public class WebRequestState
    {
        public int bytesRead;           // # bytes read during current transfer
        public long totalBytes;            // Total bytes to read
        public double progIncrement;    // delta % for each buffer read
        public Stream streamResponse;    // Stream to read from
        public byte[] bufferRead;        // Buffer to read data into
        public Uri fileURI;                // Uri of object being downloaded
        public string FTPMethod;        // What was the previous FTP command?  (e.g. get file size vs. download)
        public DateTime transferStart;  // Used for tracking xfr rate

        // Callbacks for response packet info & progress
        public ResponseInfoDelegate respInfoCB;
        public ProgressDelegate progCB;
        public DoneDelegate doneCB;

        private WebRequest _request;
        public virtual WebRequest request
        {
            get { return null; }
            set { _request = value; }
        }

        private WebResponse _response;
        public virtual WebResponse response
        {
            get { return null; }
            set { _response = value; }
        }

        public WebRequestState(int buffSize)
        {
            bytesRead = 0;
            bufferRead = new byte[buffSize];
            streamResponse = null;
        }
    }

This is just all of the stuff that we wanted to pass around between our asynchronous methods.  You’ll see our three delegates, for calling back to the GUI.  And you’ll also see where we store our WebRequest and WebResponse objects.

The final thing to look at is the code, also in WebRequestState.cs, for the two derived classes—HttpWebRequestState and FtpWebRequestState.

    /// <summary>
    /// State object for HTTP transfers
    /// </summary>
    public class HttpWebRequestState : WebRequestState
    {
        private HttpWebRequest _request;
        public override WebRequest request
        {
            get
            {
                return _request;
            }
            set
            {
                _request = (HttpWebRequest)value;
            }
        }

        private HttpWebResponse _response;
        public override WebResponse response
        {
            get
            {
                return _response;
            }
            set
            {
                _response = (HttpWebResponse)value;
            }
        }

        public HttpWebRequestState(int buffSize) : base(buffSize) { }
    }

    /// <summary>
    /// State object for FTP transfers
    /// </summary>
    public class FtpWebRequestState : WebRequestState
    {
        private FtpWebRequest _request;
        public override WebRequest request
        {
            get
            {
                return _request;
            }
            set
            {
                _request = (FtpWebRequest)value;
            }
        }

        private FtpWebResponse _response;
        public override WebResponse response
        {
            get
            {
                return _response;
            }
            set
            {
                _response = (FtpWebResponse)value;
            }
        }

        public FtpWebRequestState(int buffSize) : base(buffSize) { }
    }

The whole point of these classes is to allow us to override the request and response fields in the base class with strong-typed instances—e.g. HttpWebRequest and HttpWebResponse.

Wrapping Up

That’s about it—that’s really all that’s required to implement a very simple HTTP or FTP client application, using the HttpWebRequest and FtpWebRequest classes in System.Net.

This is still a pretty crude application and there are a number of obvious next steps that we could take if we wanted to improve it:

  • Allow user to pick # downloads and kick off simultaneous downloads, each with their own progress bar
  • Prevent clicking Get File button if a download is already in progress.  (Try it—you do actually get a 2nd download, but the progress bar goes whacky trying to report on both at the same time).
  • Add a timer so that we can recover if a transfer times out
  • Allow the user to actually store the data to a local file
  • Log the results somewhere, especially if we launched multiple downloads
About these ads

46 Comments »

  1. Hi Sean,

    Great site and great explanation. I’ve implemented your code near verbatum, with the only changes being the default text values for the ftp server / file name and the credentials. Without exception, as the BeginGetResponse is started, I get a 550 error (File Unavailable, no access).

    I am able to download the file with synchronus code and any standard FTP client, but no matter what form of Async code I use (i.e. BeginGetResponse, yours and other examples), I always get this error. I’ve tried umpteen different combos of the file name / URI, but I get the same thing every time.

    Any ideas?

    Thanks, Sean!

    Carl

    Comment by Carl — 5 February, 2009 @ 5:38 am | Reply

  2. Huh, that’s interesting. So you get the error right away on the GetFileSize command? Do you successfully get into your response callback and then see 550 when looking at the FtpWebResponse?

    Nothing jumps out at me, but I think I’d try swapping out a few different things to see if you get consistent behavior, e.g.

    – Do you get the same error, no matter the FTP site and file?
    – Try a shorter path in FTP url and simpler filename
    – Have you tried a file that just requires anonymous access, rather than using a specific user?
    – If using your own FTP server, try a simpler password, e.g. w/no special characters
    – Any sort of firewall or security software that would be blocking port 21? Try turning opening up access to port 21, or turning everything temporarily off
    – If you have access to the server, what do the logs on the server-side say?
    – Try running as a privileged (e.g. Administrators group) user
    – Try using a network monitor or packet sniffing tool to look at the actual traffic and see if you can see the difference between your working and non-working situation
    – Everything in the FtpWebRequest is really the same, i.e. the only difference between fail/no-fail is GetResponse vs. BeginGetResponse?

    I’d be very interested to hear what you run into..

    Comment by Sean — 5 February, 2009 @ 11:30 am | Reply

  3. Great stuff – thanks for posting. I was googling for some sample code on async downloads and timing downloads and found this – saved me many minutes of time!

    Comment by Scott Whigham — 18 March, 2009 @ 5:19 pm | Reply

  4. I am developing aplication which will use asynchronous webrequests. This article is very interesting, but it seems that the link to zip file (DownloadStressTest.zip) which contains all code doesn’t work. Please fix this bug.

    Comment by svit — 7 May, 2009 @ 7:16 am | Reply

  5. Svit,

    Maybe if you shared some more details on what about the example code doesn’t work for you, I could help you.

    Sean

    Comment by Sean — 7 May, 2009 @ 10:27 am | Reply

  6. Interesting code – I’ve implemented very similar HTTP code, and it is working great, until I get a “chunked” response from a server. In that case, in my ReadCallback method, when my bytesRead == 0, I’ve only received the header, and I have no idea how to get the rest of the response.

    Any ideas? If I force the request to http 1.0, the server won’t chunk the response, and things work great, but I really don’t like that work-around.

    Comment by Kent_Geek — 26 June, 2009 @ 3:46 pm | Reply

  7. Nice code but the 550 error comes up on nearly half the sites I accessed. Unfortunately I think this is just lack of foresight on Microsoft’s part. If you intend to use this and be relatively assured of if working on most sites then this is not the way to do it unless anyone else has a fix.

    Comment by Ziggy — 13 October, 2009 @ 1:01 am | Reply

  8. Hi Sean,

    This is just what i was looking. You did mention you have not implemented following feature
    “Allow the user to actually store the data to a local file ”

    Can you advise
    a.

    which function i can open the file and save the asynchronous bytes.

    b. How can i cancel the download

    Comment by nkg — 16 November, 2009 @ 1:19 am | Reply

  9. excellent article:
    one question is why do you always call
    IAsyncResult ar = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallback), reqState);

    with 0 as offset parameter ? should it be the actual offset of current reading block ?

    thanx very much

    Comment by danceoften — 2 March, 2010 @ 1:32 pm | Reply

  10. Reply to danceoften,

    Excellent question! I should have been more clear in the code, or added a comment to this effect: In the Stream.Read and Stream.BeginRead methods, the Offset parameter specifies an offset into the buffer that you’re writing into (1st parameter), rather than into the stream. You don’t need to specify an offset into the stream, because when you read from a stream, the Stream object automatically advances the position into the source buffer after each Read operation. In other words, you can just do a bunch of consecutive Stream.Read operations and you’ll automatically march through the data in the stream.

    Sean

    Comment by Sean — 10 March, 2010 @ 3:06 pm | Reply

  11. Hi Sean.

    Sorry my english…
    I like much your code…very cool…

    But I have a problem to save the doenload file.
    I dont understanding all code because I am study and I have difficult.

    Can you show mw how I can save file? Not programing…Only any step that I need…

    Thanks…

    Comment by Bisewski — 10 May, 2010 @ 11:08 am | Reply

    • Hello.
      I make this and write bytes:
      Private Sub teste(ByVal lopps As Integer, ByVal bufer As Integer, ByVal valida As Boolean)
      If Me.InvokeRequired Then
      Dim del As New testeDelegate(AddressOf teste)
      Me.Invoke(del, New Object() {lopps, bufer, valida})

      Else
      ListBox1.Items.Add(BUFFER_SIZE.ToString)

      Dim buffer As Byte() = New Byte(bufer) {}

      ListBox2.Items.Add(bufer.ToString)

      If lopps = “0” Then
      memstream = New FileStream(“d:\testestream.doc”, FileMode.Create)
      memstream.Write(buffer, 0, bufer)
      Else
      If valida = True Then
      memstream.Close()
      Else
      memstream.Write(buffer, 0, bufer)
      End If
      End If
      End If
      End Sub

      But, I have one problem, my doc is a word document but not recognised, I receive a messenge box question me if I want convert my doc…

      Comment by Bisewski — 10 May, 2010 @ 10:53 pm | Reply

      • It’s hard to say what’s going on without seeing the rest of your code. But maybe the first step you should take would be to do a byte-for-byte compare of the source file and the destination file, to see how the destination is different or what is missing. My guess is that you’re failing to write the final block to the destination file.

        Comment by Sean — 11 May, 2010 @ 2:55 am

  12. ok, thanks for your attention Seans. My error is because in this line:
    Dim buffer As Byte() = New Byte(bufer) {}
    the value is always 0.
    Now, I use your bufferRead in delegate and function ok.

    Comment by Bisewski — 11 May, 2010 @ 12:11 pm | Reply

  13. And for uploadind Seans, what I need make, steps…?

    Comment by Bisewski — 11 May, 2010 @ 3:30 pm | Reply

  14. Try
    Dim req As WebRequest = Nothing
    Dim fileURI As New Uri(txtURI.Text & “/” & TextBox2.Text)
    req = DirectCast(FtpWebRequest.Create(fileURI), FtpWebRequest)
    req.Credentials = New NetworkCredential(campousuario.Text, camposenha.Text)
    DirectCast(req, FtpWebRequest).KeepAlive = False
    DirectCast(req, FtpWebRequest).Method = WebRequestMethods.Ftp.UploadFile
    Dim requestStream As Stream = req.GetRequestStream
    Dim FileStream As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)
    Dim buffer As Byte() = New Byte(1448) {}
    Dim bytesread As Integer
    While True
    bytesread = FileStream.Read(buffer, 0, buffer.Length)
    If bytesread = 0 Then
    Exit While
    Else
    requestStream.Write(buffer, 0, bytesread)
    End If
    End While
    Catch ex As Exception
    MsgBox(String.Format(“EXC in button1_Click(): {0}”, ex.Message))
    End Try

    This code upload a file in FTP but not async.

    Comment by Bisewski — 11 May, 2010 @ 6:07 pm | Reply

    • With this sequence code, I make loop, and progress bar ok…I write the file in ftp, but wiyh size 0.

      Private Sub sendfile()
      Try

      conecte.Text = “Conectado…”
      lblDownloadComplete.Visible = False
      Dim req As WebRequest = Nothing
      Dim reqState As WebRequestState = Nothing
      ‘Dim buffer As Byte() = New Byte(1448) {}
      reqState = New FtpWebRequestState(BUFFER_SIZE)
      Dim fileURI As New Uri(txtURI.Text & “/” & TextBox2.Text)
      req = DirectCast(FtpWebRequest.Create(fileURI), FtpWebRequest)
      req.Credentials = New NetworkCredential(campousuario.Text, camposenha.Text)
      DirectCast(req, FtpWebRequest).KeepAlive = False
      DirectCast(req, FtpWebRequest).Method = WebRequestMethods.Ftp.UploadFile
      reqState.FTPMethod = WebRequestMethods.Ftp.UploadFile

      If req IsNot Nothing Then
      reqState.fileURI = fileURI
      reqState.respInfoCB = New ResponseInfoDelegate(AddressOf SetResponseInfo)
      reqState.progCB = New ProgressDelegate(AddressOf Progress)
      reqState.doneCB = New DoneDelegate(AddressOf Done)
      ‘ reqState.getlist = New testeDelegate(AddressOf teste)
      reqState.transferStart = DateTime.Now

      reqState.request = req

      Dim result As IAsyncResult = DirectCast(req.BeginGetResponse(New AsyncCallback(AddressOf ReadCallback1), reqState), IAsyncResult)

      ‘ Start the asynchronous request.

      ‘ End If

      ‘ While True
      ‘ bytesread = FileStream.Read(buffer, 0, buffer.Length)
      ‘ MsgBox(bytesread.ToString)
      ‘ If bytesread = 0 Then
      ‘Exit While
      ‘ Else
      ‘ requestStream.Write(buffer, 0, bytesread)
      End If
      ‘ End While
      Catch ex As Exception
      MsgBox(String.Format(“EXC in button1_Click(): {0}”, ex.Message))
      End Try
      End Sub

      Public Shared Sub ReadCallback1(ByVal asyncResult As IAsyncResult)
      Try
      Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
      Dim req As WebRequest = reqState.request
      Dim statusDescr As String = “”
      Dim contentLength As String = “”

      Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)

      Dim resp As FtpWebResponse = DirectCast((req.EndGetResponse(asyncResult)), FtpWebResponse)
      reqState.response = resp

      statusDescr = resp.StatusDescription
      reqState.totalBytes = FileStreamm.Length
      ‘ # bytes
      contentLength = reqState.totalBytes

      ‘ Get this info back to the GUI — max # bytes, so we can do progress bar
      If statusDescr “” Then

      reqState.respInfoCB(statusDescr, contentLength)
      End If

      Dim responseStream As Stream = FileStreamm
      reqState.streamResponse = responseStream

      ‘ Begin reading contents of the response data

      Dim ar As IAsyncResult = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback2), reqState)
      Catch ex As Exception
      MsgBox(String.Format(“EXC in ReadCallback(): {0}”, ex.Message))
      End Try
      End Sub
      Public Shared Sub ReadCallback2(ByVal asyncResult As IAsyncResult)

      Try
      Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
      Dim responseStream As Stream = reqState.streamResponse
      ‘ Get results of read operation
      Dim bytesRead As Integer = responseStream.EndRead(asyncResult)

      Dim buffer As Byte() = New Byte(bytesRead) {}
      responseStream.Write(buffer, 0, bytesRead)

      ‘ Got some data, need to read more
      If bytesRead > 0 Then
      ‘ Report some progress, including total # bytes read, % complete, and transfer rate
      reqState.bytesRead += bytesRead
      Dim pctComplete As Double = (CDbl(reqState.bytesRead) / CDbl(reqState.totalBytes)) * 100.0F
      ‘ Note: bytesRead/totalMS is in bytes/ms. Convert to kb/sec.
      Dim totalTime As TimeSpan = DateTime.Now – reqState.transferStart
      Dim kbPerSec As Double = (reqState.bytesRead * 1000.0F) / (totalTime.TotalMilliseconds * 1024.0F)
      reqState.progCB(reqState.bytesRead, pctComplete, kbPerSec)
      ‘ Kick off another read
      Dim ar As IAsyncResult = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback2), reqState)
      Exit Sub
      Else
      MsgBox(“dentro do else”)
      ‘Dim ms As FileStream = New FileStream(“d:\testestream.doc”, FileMode.Create)
      ‘ ms.Write(reqState.bufferRead, 0, BUFFER_SIZE)
      ‘ reqState.getlist(reqState.iforloop, bytesRead, True, reqState.bufferRead)
      ‘ responseStream.Write(, 0, BUFFER_SIZE)
      ‘memStream.Close()

      ‘ EndRead returned 0, so no more data to be read
      responseStream.Close()
      reqState.response.Close()
      reqState.doneCB()
      End If

      Catch ex As Exception
      MsgBox(String.Format(“EXC in ReadCallback(): {0}”, ex.Message))
      End Try
      End Sub

      Comment by Bisewski — 12 May, 2010 @ 10:39 pm | Reply

      • With this sequence code, the loop is perfectly, progress bar ok but I dont know Hou can I write in ftp my file…

        Here I think that is write file here…
        Private Sub teste1(ByVal lopps As Integer, ByVal bufer As Integer, ByVal valida As Boolean, ByVal bufferRead As Byte(), ByVal reqstated As FileStream)
        If Me.InvokeRequired Then
        Dim del As New teste1Delegate(AddressOf teste1)
        Me.Invoke(del, New Object() {lopps, bufer, valida, bufferRead, reqstated})
        Else
        Dim buffer As Byte() = New Byte(bufer) {}
        If lopps = “0” Then
        ‘ Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)
        ‘FileStreamm.Write(bufferRead, 0, bufer)
        Else
        If valida = True Then
        ‘ FileStreamm.Close()
        Else
        ‘ FileStreamm.Write(bufferRead, 0, bufer)
        End If
        End If
        End If
        End Sub

        Here, I get de file size and pass ithis to reqstate.toltalbytes…

        Private Sub sendfile()
        Try

        conecte.Text = “Conectado…”
        lblDownloadComplete.Visible = False
        Dim req As WebRequest = Nothing
        Dim reqState As WebRequestState = Nothing
        ‘Dim buffer As Byte() = New Byte(1448) {}
        reqState = New FtpWebRequestState(BUFFER_SIZE)
        Dim fileURI As New Uri(txtURI.Text & “/” & TextBox2.Text)
        req = DirectCast(FtpWebRequest.Create(fileURI), FtpWebRequest)
        req.Credentials = New NetworkCredential(campousuario.Text, camposenha.Text)
        DirectCast(req, FtpWebRequest).KeepAlive = False
        DirectCast(req, FtpWebRequest).Method = WebRequestMethods.Ftp.UploadFile
        reqState.FTPMethod = WebRequestMethods.Ftp.UploadFile

        Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)
        reqState.totalBytes = FileStreamm.Length
        reqState.streamResponse = FileStreamm

        If req IsNot Nothing Then
        reqState.fileURI = fileURI
        reqState.respInfoCB = New ResponseInfoDelegate(AddressOf SetResponseInfo)
        reqState.progCB = New ProgressDelegate(AddressOf Progress)
        reqState.doneCB = New DoneDelegate(AddressOf Done)
        reqState.getlist1 = New teste1Delegate(AddressOf teste1)
        reqState.transferStart = DateTime.Now

        reqState.request = req

        Dim result As IAsyncResult = DirectCast(req.BeginGetResponse(New AsyncCallback(AddressOf ReadCallback1), reqState), IAsyncResult)

        ‘ Start the asynchronous request.

        ‘ End If

        ‘ While True
        ‘ bytesread = FileStream.Read(buffer, 0, buffer.Length)
        ‘ MsgBox(bytesread.ToString)
        ‘ If bytesread = 0 Then
        ‘Exit While
        ‘ Else
        ‘ requestStream.Write(buffer, 0, bytesread)
        End If
        ‘ End While
        Catch ex As Exception
        MsgBox(String.Format(“EXC in button1_Click(): {0}”, ex.Message))
        End Try
        End Sub

        Here I get the first bytes of file…
        Public Shared Sub ReadCallback1(ByVal asyncResult As IAsyncResult)
        Try
        Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
        Dim req As WebRequest = reqState.request
        Dim statusDescr As String = “”
        Dim contentLength As String = “”

        ‘ Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)

        Dim resp As FtpWebResponse = DirectCast((req.EndGetResponse(asyncResult)), FtpWebResponse)
        reqState.response = resp

        statusDescr = resp.StatusDescription
        ‘ reqState.totalBytes = FileStreamm.Length
        ‘ # bytes
        contentLength = reqState.totalBytes

        ‘ Get this info back to the GUI — max # bytes, so we can do progress bar
        If statusDescr “” Then

        reqState.respInfoCB(statusDescr, contentLength)
        End If

        ‘ Dim responseStream As Stream = FileStreamm
        Dim responseStream As Stream = reqState.streamResponse
        reqState.streamResponse = responseStream

        ‘ Begin reading contents of the response data

        Dim ar As IAsyncResult = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback2), reqState)
        Catch ex As Exception
        MsgBox(String.Format(“EXC in ReadCallback(): {0}”, ex.Message))
        End Try
        End Sub
        And here I make a loop…call the teste1 fo write file…
        How Can I finish this code for finaly write in ftp?

        Comment by Bisewski — 13 May, 2010 @ 10:25 pm

      • With this sequence code, the loop is perfectly, progress bar ok but I dont know Hou can I write in ftp my file…

        Here I think that is write file here…
        Private Sub teste1(ByVal lopps As Integer, ByVal bufer As Integer, ByVal valida As Boolean, ByVal bufferRead As Byte(), ByVal reqstated As FileStream)
        If Me.InvokeRequired Then
        Dim del As New teste1Delegate(AddressOf teste1)
        Me.Invoke(del, New Object() {lopps, bufer, valida, bufferRead, reqstated})
        Else
        Dim buffer As Byte() = New Byte(bufer) {}
        If lopps = “0” Then
        ‘ Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)
        ‘FileStreamm.Write(bufferRead, 0, bufer)
        Else
        If valida = True Then
        ‘ FileStreamm.Close()
        Else
        ‘ FileStreamm.Write(bufferRead, 0, bufer)
        End If
        End If
        End If
        End Sub

        Here, I get de file size and pass ithis to reqstate.toltalbytes…

        Private Sub sendfile()
        Try

        conecte.Text = “Conectado…”
        lblDownloadComplete.Visible = False
        Dim req As WebRequest = Nothing
        Dim reqState As WebRequestState = Nothing
        ‘Dim buffer As Byte() = New Byte(1448) {}
        reqState = New FtpWebRequestState(BUFFER_SIZE)
        Dim fileURI As New Uri(txtURI.Text & “/” & TextBox2.Text)
        req = DirectCast(FtpWebRequest.Create(fileURI), FtpWebRequest)
        req.Credentials = New NetworkCredential(campousuario.Text, camposenha.Text)
        DirectCast(req, FtpWebRequest).KeepAlive = False
        DirectCast(req, FtpWebRequest).Method = WebRequestMethods.Ftp.UploadFile
        reqState.FTPMethod = WebRequestMethods.Ftp.UploadFile

        Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)
        reqState.totalBytes = FileStreamm.Length
        reqState.streamResponse = FileStreamm

        If req IsNot Nothing Then
        reqState.fileURI = fileURI
        reqState.respInfoCB = New ResponseInfoDelegate(AddressOf SetResponseInfo)
        reqState.progCB = New ProgressDelegate(AddressOf Progress)
        reqState.doneCB = New DoneDelegate(AddressOf Done)
        reqState.getlist1 = New teste1Delegate(AddressOf teste1)
        reqState.transferStart = DateTime.Now

        reqState.request = req

        Dim result As IAsyncResult = DirectCast(req.BeginGetResponse(New AsyncCallback(AddressOf ReadCallback1), reqState), IAsyncResult)

        ‘ Start the asynchronous request.

        ‘ End If

        ‘ While True
        ‘ bytesread = FileStream.Read(buffer, 0, buffer.Length)
        ‘ MsgBox(bytesread.ToString)
        ‘ If bytesread = 0 Then
        ‘Exit While
        ‘ Else
        ‘ requestStream.Write(buffer, 0, bytesread)
        End If
        ‘ End While
        Catch ex As Exception
        MsgBox(String.Format(“EXC in button1_Click(): {0}”, ex.Message))
        End Try
        End Sub

        Here I get the first bytes of file…
        Public Shared Sub ReadCallback1(ByVal asyncResult As IAsyncResult)
        Try
        Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
        Dim req As WebRequest = reqState.request
        Dim statusDescr As String = “”
        Dim contentLength As String = “”

        ‘ Dim FileStreamm As FileStream = File.Open(“d:\rel1.xls”, FileMode.Open)

        Dim resp As FtpWebResponse = DirectCast((req.EndGetResponse(asyncResult)), FtpWebResponse)
        reqState.response = resp

        statusDescr = resp.StatusDescription
        ‘ reqState.totalBytes = FileStreamm.Length
        ‘ # bytes
        contentLength = reqState.totalBytes

        ‘ Get this info back to the GUI — max # bytes, so we can do progress bar
        If statusDescr “” Then

        reqState.respInfoCB(statusDescr, contentLength)
        End If

        ‘ Dim responseStream As Stream = FileStreamm
        Dim responseStream As Stream = reqState.streamResponse
        reqState.streamResponse = responseStream

        ‘ Begin reading contents of the response data

        Dim ar As IAsyncResult = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback2), reqState)
        Catch ex As Exception
        MsgBox(String.Format(“EXC in ReadCallback(): {0}”, ex.Message))
        End Try
        End Sub
        And here I make a loop…call the teste1 fo write file…
        How Can I finish this code for finaly write in ftp?

        Public Shared Sub ReadCallback2(ByVal asyncResult As IAsyncResult)

        Try
        Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
        Dim responseStream As Stream = reqState.streamResponse
        ‘ Get results of read operation
        Dim bytesRead As Integer = responseStream.EndRead(asyncResult)

        Dim buffer As Byte() = New Byte(bytesRead) {}

        reqState.getlist1(reqState.iforloop, bytesRead, False, reqState.bufferRead, reqState.streamResponse)
        reqState.iforloop = CInt(reqState.iforloop + 1)
        ‘ Got some data, need to read more
        If bytesRead > 0 Then
        ‘ Report some progress, including total # bytes read, % complete, and transfer rate
        reqState.bytesRead += bytesRead
        Dim pctComplete As Double = (CDbl(reqState.bytesRead) / CDbl(reqState.totalBytes)) * 100.0F
        ‘ Note: bytesRead/totalMS is in bytes/ms. Convert to kb/sec.
        Dim totalTime As TimeSpan = DateTime.Now – reqState.transferStart
        Dim kbPerSec As Double = (reqState.bytesRead * 1000.0F) / (totalTime.TotalMilliseconds * 1024.0F)
        reqState.progCB(reqState.bytesRead, pctComplete, kbPerSec)
        ‘ Kick off another read
        Dim ar As IAsyncResult = responseStream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback2), reqState)
        Exit Sub
        Else
        MsgBox(“dentro do else”)
        reqState.getlist1(reqState.iforloop, bytesRead, True, reqState.bufferRead, reqState.streamResponse)
        ‘Dim ms As FileStream = New FileStream(“d:\testestream.doc”, FileMode.Create)
        ‘ ms.Write(reqState.bufferRead, 0, BUFFER_SIZE)
        ‘ reqState.getlist(reqState.iforloop, bytesRead, True, reqState.bufferRead)
        ‘ responseStream.Write(, 0, BUFFER_SIZE)
        ‘memStream.Close()

        ‘ EndRead returned 0, so no more data to be read
        responseStream.Close()
        reqState.response.Close()
        reqState.doneCB()
        End If

        Catch ex As Exception
        MsgBox(String.Format(“EXC in ReadCallback(): {0}”, ex.Message))
        End Try
        End Sub

        Comment by Bisewski — 13 May, 2010 @ 10:25 pm

  15. Certantly my loop is incorrectly because my progress is very fast…

    Comment by Bisewski — 13 May, 2010 @ 11:12 pm | Reply

  16. I thanks for your code, now, I upload and download files in async mode in ftp with no problem…And update progress bar and more…

    Only one problem, no send zip files…
    Can you help-me? I send total bytes in ftp, but my file is damaged…more people have the same problem but any solutions…

    Dim reqState As WebRequestState = DirectCast((asyncResult.AsyncState), WebRequestState)
    Dim requestStream As Stream = reqState.streamResponse

    Dim stream As FileStream = reqState.filestreamup
    Dim bytesRead As Integer = stream.EndRead(asyncResult)

    If bytesRead > 0 Then

    Dim buffer() As Byte = New Byte(bytesRead) {}
    requestStream.Write(buffer, 0, bytesRead)

    ‘ reqState.uploadfile(reqState.iforloop, bytesRead, False, reqState.bufferRead, requestStream)
    reqState.iforloop = CInt(reqState.iforloop + 1)
    ‘ Report some progress, including total # bytes read, % complete, and transfer rate
    reqState.bytesRead += bytesRead
    Dim pctComplete As Double = (CDbl(reqState.bytesRead) / CDbl(reqState.totalBytes)) * 100.0F
    ‘ Note: bytesRead/totalMS is in bytes/ms. Convert to kb/sec.
    Dim totalTime As TimeSpan = DateTime.Now – reqState.transferStart
    Dim kbPerSec As Double = (reqState.bytesRead * 1000.0F) / (totalTime.TotalMilliseconds * 1024.0F)
    reqState.progCB(reqState.bytesRead, pctComplete, kbPerSec)
    ‘ Kick off another read

    Dim ar As IAsyncResult = stream.BeginRead(reqState.bufferRead, 0, BUFFER_SIZE, New AsyncCallback(AddressOf ReadCallback1), reqState)

    Exit Sub
    Else

    I you can help-me, I apologise….

    Comment by Bisewski — 20 May, 2010 @ 4:42 pm | Reply

  17. Excellent code. I was trying to do the same when uploading files by POST. In this case, it looks like RespCallback is only called after the file has finished uploading. Not sure how to show a progress bar in between. Can you please post an article about uploading as well?

    Comment by Sreejith K — 6 April, 2011 @ 9:54 am | Reply

    • Uploading is pretty easy–just the reverse of downloading. Instead of calling Stream.BeginRead, you call Stream.BeginWrite. You can also specify a write callback, in the same way that you specified a read callback. And the write callback would be where you’d update your progress bar. Doing the uploading, using all of my downloading code as an example, should be pretty easy–and a good way for you to learn what the code is doing. :O)

      Comment by Sean — 6 April, 2011 @ 2:32 pm | Reply

  18. Any way of implementing this with ASP.NET ?
    I need to create an Asynchronous huge file download but it must be asp.net I cant install anything on the client.

    any help will be appreciated

    Comment by Gal pasternak — 8 May, 2011 @ 9:01 pm | Reply

  19. Hey Sean,

    Thanks a lot for this article. It’s been invaluable in understanding the shady depths of asynchronous execution of HttpWebRequest’s in .NET.

    Keep up the awesome work!

    Cheers

    Comment by Bruno de Carvalho — 26 July, 2011 @ 7:59 pm | Reply

    • De nada. Writing C# code is almost as fulfilling as listening to Fado !

      Comment by Sean — 26 July, 2011 @ 8:44 pm | Reply

  20. Very nice code, although the zip file is not available (your whole website seems to be out atm).

    I save to a file by adding a FileStream variable to the WebRequestState, then initialize it with something like
    outFile = new FileStream(fileName, FileMode.Create, FileAccess.Write, FileShare.None);
    in the ctor and a
    reqState.outFile.Write(reqState.bufferRead, 0, bytesRead);
    into the ReadCallback() function.

    Works fine, even with doing multiple consecutive downloads (the labels and progress bar don’t handle it, but that’s easy to fix).

    Comment by Einar Jón — 30 September, 2011 @ 12:14 pm | Reply

  21. I can’t see where to place the delegate declarations.

    Comment by PEte — 19 January, 2012 @ 6:03 pm | Reply

    • I think pretty much anywhere you like–just as long as they’re in scope for the code that uses the delegate type.

      Comment by Sean — 19 January, 2012 @ 8:24 pm | Reply

  22. [...] found this link but i am not sure how it is applicable to my problem. Any help or idea for how to go about [...]

    Pingback by How to make Asynchronous Web requests in C# WPF | FaceColony.org – Developers Network — 21 January, 2012 @ 10:51 am | Reply

  23. The download of the full zip is broken. Doesn’t look like http://www.seans.com/samples/DownloadStressTest.zip points to anything.

    Comment by Andrew — 4 February, 2013 @ 9:36 pm | Reply

    • Should be working now. Sorry about that.

      Comment by Sean — 1 March, 2013 @ 8:33 pm | Reply

  24. Your code looks amazing, I want to download the file and simultaneously write into file. Can you please help me writing the code and where exactly I should place in above code. I use c# language.

    Comment by Darshan — 18 February, 2013 @ 2:12 pm | Reply

    • Sure, my example reads the data into the buffer in WebRequestState, so you could just access that data from the ReadCallback method, as each packet becomes available.

      Comment by Sean — 20 February, 2013 @ 7:08 pm | Reply

  25. Hi, I’m trying to do this on silverlight and i’m getting an out-of-memory exception when using my code(see below). Any help will really be appreciated.

    WebRequest documentRequest = (HttpWebRequest)HttpWebRequest.Create(fileUri);
    WebRequestState requestState = new HttpWebRequestState(8192);
    requestState.Request = documentRequest;

    requestState.FileUri = fileUri;
    requestState.ResponseInformationCallback = new WebRequestState.ResponseInfoDelegate(requestStateInformation_CallBack);
    requestState.ProgressCallback = new WebRequestState.ProgressDelegate(ProgressChanged_CallBack);
    requestState.DownloadCompleteCallback = new WebRequestState.DownloadCompleteDelegate(DownloadCompleted_Callback);
    requestState.TransferStart = DateTime.Now;

    IAsyncResult result = (IAsyncResult)documentRequest.BeginGetResponse(new AsyncCallback(ResponseCallBack), requestState);

    private void requestStateInformation_CallBack(string statusDescr, string contentLength)
    {
    }

    private void ProgressChanged_CallBack(int totalBytes, double pctComplete, double transferRate)
    {
    if (this.Dispatcher.CheckAccess())
    {
    this.UpdateProgressBar(totalBytes, pctComplete, transferRate);
    }
    else
    {
    Deployment.Current.Dispatcher.BeginInvoke(() =>
    {
    UpdateProgressBar(totalBytes, pctComplete, transferRate);
    });
    }

    }

    private void DownloadCompleted_Callback(byte[] bufferRead)
    {
    if (this.Dispatcher.CheckAccess())
    {
    AddWatermarkIfAnyExists(bufferRead);
    }
    else
    {
    Deployment.Current.Dispatcher.BeginInvoke(() =>
    {
    AddWatermarkIfAnyExists(bufferRead);
    });
    }
    }

    private void ResponseCallBack(IAsyncResult asyncResult)
    {
    WebRequestState requestState = ((WebRequestState)(asyncResult.AsyncState));
    WebRequest request = requestState.Request;

    ((HttpWebRequest)request).AllowReadStreamBuffering = false;
    ((HttpWebRequest)request).AllowAutoRedirect = false;

    HttpWebResponse response = ((HttpWebResponse)(request.EndGetResponse(asyncResult)));
    requestState.Response = response;
    requestState.TotalBytes = requestState.Response.ContentLength;

    string statusDescription = response.StatusDescription;
    string contentLength = requestState.Response.ContentLength.ToString();

    if (statusDescription != “”)
    requestState.ResponseInformationCallback(statusDescription, contentLength);

    Stream responseStream = requestState.Response.GetResponseStream();
    requestState.StreamResponse = responseStream;

    IAsyncResult _asyncResult = responseStream.BeginRead(requestState.BufferRead, 0, 8192, new AsyncCallback(ReadStream_CallBack), requestState);
    }

    private void ReadStream_CallBack(IAsyncResult asyncResult)
    {
    WebRequestState requestState = ((WebRequestState)(asyncResult.AsyncState));
    Stream responseStream = requestState.StreamResponse;
    int bytesRead = responseStream.EndRead(asyncResult);

    if (bytesRead > 0)
    {
    requestState.BytesRead += bytesRead;
    double pctComplete = ((double)requestState.BytesRead / (double)requestState.TotalBytes) * 100.0f;

    TimeSpan totalTime = DateTime.Now – requestState.TransferStart;
    double kbPerSecond = (requestState.BytesRead * 1000.0f) / (totalTime.TotalMilliseconds * 1024.0f);

    requestState.ProgressCallback(requestState.BytesRead, pctComplete, kbPerSecond);

    //byte[] fileBytes = new byte[requestState.File.Length + bytesRead];

    //System.Buffer.BlockCopy(requestState.File, 0, fileBytes, 0, requestState.File.Length);
    //System.Buffer.BlockCopy(requestState.BufferRead, 0, fileBytes, requestState.File.Length, bytesRead);

    requestState.File = CombineBytes(requestState.File, requestState.BufferRead, bytesRead);
    //requestState.File = fileBytes;

    //Trigger another read
    IAsyncResult _asyncResult = responseStream.BeginRead(requestState.BufferRead, 0, 8192, new AsyncCallback(ReadStream_CallBack), requestState);
    }
    else
    {
    responseStream.Close();
    requestState.Response.Close();
    requestState.DownloadCompleteCallback(requestState.File);
    }
    }

    private void UpdateProgressBar(int totalBytes, double pctComplete, double transferRate)
    {
    this.pbDownload.IsIndeterminate = false;
    pbDownload.Value = pctComplete;
    lblDownload.Content = string.Format(“Downloading {0}kb of {1}kb”, ((totalBytes * pctComplete) / 1024), totalBytes / 1024);
    }

    private void AddWatermarkIfAnyExists(byte[] document)
    {
    if (loadWaterMark)
    LoadDocumentWithWaterMark(document);
    else
    LoadDocument(document);
    }

    public byte[] CombineBytes(byte[] first, byte[] second, int secondLimit)
    {
    byte[] ret = new byte[first.Length + second.Length];
    Buffer.BlockCopy(first, 0, ret, 0, first.Length);
    Buffer.BlockCopy(second, 0, ret, first.Length, secondLimit);
    return ret;
    }

    Comment by Allan Chua — 29 March, 2013 @ 1:33 am | Reply

  26. Hi

    I am facing a trouble in downloading a text file hosted in dropbox. Could you please check my post in
    http://stackoverflow.com/questions/20237968/error-in-dropbox-file-download-using-webclient. It would be of great help if you can provide some breakthrough for me.

    Regards
    Midhun

    Comment by midhun — 27 November, 2013 @ 9:22 am | Reply

    • I think you’ll need to first verify that the URL you’re trying to use, as well as the authentication, works in general and is a URL that links directly to a file. It looks like someone already mentioned this on Stack Overflow.

      Comment by Sean — 28 November, 2013 @ 12:15 am | Reply

  27. Hi Sean ,

    The Solution download link seems to be down, can you please check

    Regards
    Sanath

    Comment by Sanath Shetty — 1 January, 2014 @ 4:12 am | Reply

  28. Hi Sean,

    The Download link for the solution seems to be down can you check it

    Regards
    Sanath

    Comment by Sanath Kumar shetty — 1 January, 2014 @ 1:59 pm | Reply

    • Hello Sanath,

      Yes, the server that hosts the full solution is down at the moment. You can, however, just use the code directly that is presented in the blog post. All of the code should be there.

      Sean

      Comment by Sean — 2 January, 2014 @ 2:27 pm | Reply

  29. HI,

    I’m trying to uplaod files and get speed, mb uploaded, ecc … is there a way of doing so? Any help would be greatly appreciated.

    Cheers!

    Comment by Bruno — 13 January, 2014 @ 12:08 am | Reply

    • Certainly you can keep track of time that it takes to upload each packet in order to determine. And because you’re writing the code to upload your data, you know exactly how much you’ve uploaded.

      Comment by Sean — 13 January, 2014 @ 2:00 pm | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Theme: Rubric. Get a free blog at WordPress.com

Follow

Get every new post delivered to your Inbox.

Join 483 other followers

%d bloggers like this: