Loading Web Pages

I don't usually put up techie pages with code in them, but today I present something that I think might be useful and I've not found anywhere else. It is part of my RSS reader for the XNA display program, and it solves a couple of problems that you might hit. My feed reader needs to read feeds that are on a password protected site, and it also needs to read feeds that are compressed. This method does both:

private StringBuilder readWebPage(
    string url,
    string username,
    string password,
    string domain)
{
    // used to build entire input
    StringBuilder sb = new StringBuilder();

    // used on each read operation
    byte[] buf = new byte[8192];

    try
    {
        // prepare the web page we will be asking for
        HttpWebRequest request = 
            (HttpWebRequest)WebRequest.Create(url);

        if (username.Length > 0)
        {
            request.Proxy = null;
            NetworkCredential credential = 
               new NetworkCredential(username,
                                     password,
                                     domain);
            request.Credentials = credential;
        }

        // execute the request
        HttpWebResponse response =
            (HttpWebResponse)request.GetResponse();

        // we will read data via the response stream
        Stream resStream;
        Stream inputStream =
            response.GetResponseStream();

        // Might have a compressed stream coming back

        switch (response.ContentEncoding)
        {
            case "gzip":
                resStream =
                   new GZipStream(inputStream, 
                          CompressionMode.Decompress);
                break;
            case "":
                resStream = inputStream;
                break;
            default :
                debugOutput.PutMessageLine(
                    "Invalid content encoding: " +
                    response.ContentEncoding + " in: "
                    + url);
                return null;
        }

        string tempString = null;
        int count = 0;

        do
        {
            // fill the buffer with data
            count = resStream.Read(buf, 0, buf.Length);

            // make sure we read some data
            if (count != 0)
            {
              // translate from bytes to ASCII text
              tempString = 
                Encoding.ASCII.GetString(buf,0,count);

              // continue building the string
              sb.Append(tempString);
            }
        }
        while (count > 0); // any more data to read?

        return sb;
    }
    catch (System.Net.WebException w)
    {
        debugOutput.PutMessageLine(w.Message);
        return null;
    }
    catch (System.Net.ProtocolViolationException p)
    {
        debugOutput.PutMessageLine(p.Message);
        return null;
    }
    catch (Exception e)
    {
        debugOutput.PutMessageLine(e.Message);
        return null;
    }
}

This is not very elegant code, but it does work. I've pulled it straight out of the program and so there are a few things you need to know. I have a class called debugOutput which I use to send messages to the user. You either need to create one of your own, or remove those lines. If you need to use a username and password you need to add those to the call, otherwise you can leave those fields as empty strings.

Comments

[Ankur Gurha]

You might also want to consider attaching RemoteCertificateValidationCallback to the Service PointManager before you create the HTTPWebRequest, as a result your RSS reader could stream data from a HTTPS bound Endpoint.


ServicePointManager.
ServerCertificateValidationCallback =
new RemoteCertificateValidationCallback(AuthenticateServerCertificate);


and maybe delegate which just returns true even if we have some policy errors.


public static bool AuthenticateServerCertificate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
return true;
}

Hope this helps.

[Rob]

Thanks for that (and I'm pleased to see that you still read my blog!). I've not tried it with an https feed, I'll put your suggestion in and have a go.