Http debugging....tools and practices

Hi to all,
this is my first blog, so don't be to strict in jugging my words.
I would try to present my point of view in HTTP debugging process.

There is a bunch of free and payable tools that life of one web developer can make more comfortable. My opinion is that this post can be helpful to all web developers. no matter on which technology are based. Asp.net, java, php and other technologies developers are at least one in their work enforced to do some kind of low level HTTP debugging. No more talk, let's do something "in medias res".

In process of developing some .net service for Google documents API, and more for Google documents based on Protocol 3, I needed to see HTTP request and HTTP response , in depth.
Native idea was to use some of large number add ins for Firefox and Internet Explorer. So, let's start this interesting journey, naming it one by one.

• Fiddler

Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP(S) traffic, set breakpoints, and "fiddle" with incoming or outgoing data. Fiddler includes a powerful event-based scripting subsystem, and can be extended using any .NET language.

Fiddler is freeware and can debug traffic from virtually any application, including browsers like Internet Explorer, Mozilla Firefox, Opera.

Fiddler can be downloaded from http://www.fiddler2.com/fiddler2/version.asp
To compare this tools and browser add-ins I will send request from my app raised on button click that will contact Google docs and try to retieve all folders in Google docs account for logged user

Trace traffic in Fiddler, see details about request, in Result column you can see responce status code; you can see Protocol used, targeted Host, target URL and much more.


For some request, you can see details like request header and related informations in columns on right side Auth,Raw,Xml...



You can see response header and informations about response: response header content like content type,content lenght, or in some other columns you can see details about Auth,Cache,Privacy,Raw and xml.



You can see xml content of response and wanted details like "did I get response as I expected".


Fiddler is offcourse, much,much more, in my array of work I exploerd just a surface of this powerfull tool. You can set AutoResponder or you can use nice subtool Request Builder to create low level requests for some resouce... You can tune performances or you can read more on msdn.

• HTTP Debugger Pro

Let see one tool in action that is unfortinatelly expensive, but I love this tool UI. Offcourse, you can allways use 14-day trial version like I did. So,let's click on the button that contacts Google docs and enjou in muchness of details.

You can see nice UI with details that you want to inspect: like program that send request,method of request;get,post,..., domain, full targeted url, type od data:text,xml,json,....,etc.

Offcourse, you can see details about request


You can see details about response header


You can see details about Request stream


You can see details about Response stream


Http Debugger Pro has extremly user friendly interface, so if in your work you have intensive http debugging tasks, it gets all my recomendations.
See detailed tutorial here.

• Web Development Helper
  Web Development Helper is a free browser extension for Internet Explorer that provides a set of tools and utilities for the Web developer, esp. Ajax and ASP.NET developers. The tool provides features such as a DOM inspector, an HTTP tracing tool, and script diagnostics and immediate window.
  Web Development Helper works against IE6+, and requires the .NET Framework 2.0 or greater to be installed on the machine.
  The tool can be activated using the Tools -> Web Development Helper command. You can also customize your browser's toolbar to add a button for this command to facilitate frequest use. Clicking on the menu command or browser button brings up the tool's console window and set of commands.

Let's dive into this IE tool:

Start tool, select request that you want to inspect, you can see details and if you want more data,click on the right side on Show Details link button.

Request header details


Response header details


More details on Response content


So, if you are the "add-in type of developer" use this solution, it is a bit light weight, but it is free, and detailed as it should be.

• Tcp Trace
  

The simpliest lightweiht http debugging tool and tracing tool for http/tcp. This is a tool written by Shawn Ostermann at Ohio University, for analysis of TCP dump files. It can take as input the files produced by several popular packet-capture programs, including tcpdump, snoop, etherpeek, HP Net Metrix, and WinDump. tcptrace can produce several different types of output containing information on each connection seen, such as elapsed time, bytes and segments sent and recieved, retransmissions, round trip times, window advertisements, throughput, and more. It can also produce a number of graphs for further analysis.

Set destination port as 3639 if your app is running on http://localhost:3639/ControlWebSite/Default.aspx
You have also to set other data like destination server,2 ports etc

The HTTP module can be used to analyze web (HTTP 1.0 [8] and HTTP 1.1 [5]) traffic from dumpfiles. The module can be run by passing in the -xhttp[P] option to tcptrace , where P represents the HTTP port number. By default, the module looks for web traffic in the TCP well known port 80. If your dumpfile has web traffic in port number P (not 80), you may pass it in as P in the command line as shown above.
Let's dive in...
GET /main.asp HTTP/1.0
Response Code: 200 (OK)
Request Length: 438
Reply Length: 30730
Content Length: 30447
Content Type : text/html
Time request sent: Thu May 1 14:29:41.700691 2003 (1051813781.701)
Time reply started: Thu May 1 14:29:45.189720 2003 (1051813785.190)
Time reply ACKed: Thu May 1 14:29:46.394593 2003 (1051813786.395)
Elapsed time: 3489 ms (request to first byte sent)
Elapsed time: 4694 ms (request to content ACKed)

You have detailed User manual available.
Let us see the information reported as part of the GET /main.asp HTTP request.

These are the data exposed to you
Response Code field displays the HTTP response code indicating the type of HTTP response received from the server.

Request Length tracks the total length (in bytes) of this HTTP request.

Reply Length tracks the length of the HTTP response received from the server for this HTTP request.

Content Length field reports the length of the response found from the Content Length field part of the HTTP response. If the Content Length field is not found in the response, the Content Length is reported as the length of the remaining data (after the headers) in the server-to-client data file.

Content Type reports the content type of the data being sent in the response.

Time request sent, Time reply started, Time reply ACKed fields, as is obvious from their names, track the time the request was sent, the time when the first segment carrying the response was received, and when the response was ACKed by the client, respectively. Note that the times are printed in human readable text format along with their absolute values.

Elapsed time fields indicate the time elapsed between seeing the request and receiving the first byte of the response, and the time elapsed between seeing the request and having the response ACKed.

Interesting detail is that tcptrace is compiled and tested at Ohio University on Solaris and GNU/Linux machines. It is also known to work under [NetFreeOpen] BSD, MacOSX, AIX, HP-UX and IRIX.

• Tamper Data

This is first Firefox addin that I will present. It can be very usefull to view and modify HTTP/HTTPS headers and post parameters, Trace and time http response/requests, Security test web applications by modifying POST parameters.

After you start your add-in in Firefox, Tools -> Tamper Data, set Start Tamper to intercept request , and dive into data by clicking on our old frend button - Get data from Google, the game can start on:

Key feature is that this addin enables tampering data on the fly, and by that I mean you can capture request before it is send to the server ,modify it and send it modified to the recepient.

Details that can be changed for the sending request


There is Request Header view


There is detailed Response header view


There is a nice list of settings for our friendly tool

• Firefox Web Developer Toolbar

This Firefox add in is not rich tool for http debugging, or better say not tool for this at all. But it is so options rich, so I want to mention it as a helper for various web development issues that may come.You can do some http introspection by clicking on Tools->Web Developer->Information -> View Response Header. This the Web Developer extension adds a menu and a toolbar with various web developer tools.

• IE HTTP Analyzer

This is a tool which can serve as HTTP monitor, HTTP sniffer and HTTP Tracer.
IEInspector HTTP Analyzer is such a handy tool that allows you to monitor, trace, debug and analyze HTTP/HTTPS traffic in real-time.
HTTP Analyzer includes two Editions: Stand-alone Edition and Add-on Edition. Stand-alone Edition : Window stand-alone EXE application. It allows you to capture and view HTTP/HTTPS traffic from a specific process or user/session/system wide. Support IE, Safari, Chrome, Firefox and other win32 web application.Add-on Edition: An add-on that integrates into the lower part of your IE or Firefox window and can be opened/closed from the IE or Firefox toolbar. It can only capture and view HTTP/HTTPS traffic of current IE or Firefox process.

Unfortinatelly, this powerfull tool is not free. Let's see is it money worth. Click on the legendary button and call Get all google documents.You can see that request and some others that our machine regulary sends.

You can use standalone tool or you can use Firefox addin, both are same by features.

Inspect your request

Inspect your response

There is a big list of usefull features in this tool, like Http Request Builder, JSON Viewer( HTTP Analyzer can deserialize SOAP and JSON traffic into the easy-to-use object tree) etc

• Http Watch

There is free basic version and there is one version that you have to pay. Features:Plug-in for Internet Explorer ,Standalone log file viewer, Shows HTTPS requests, Handles HTTP compression, Page Level Time Charts, Request Level Time Charts, Automatic Recording, Exports to XML and CSV, Displays basic HTTP information including URLs, status codes, response size, mime type & time, Displays detailed HTTP information including headers, cookies, streams, etc...
Certain functions in HttpWatch Basic Edition are restricted to the following set of well known domains - amazon.com, ebay.com, google.com, httpwatch.com, microsoft.com & slashdot.org

Start tool in firefox, Tools-> HttpWatch, press Record, an let the music play...

Call Google and see response

See details

More details you can see by saving this .hwl log file, and after that you can open it in your desktop application Http Watch Studio. There is a large number typical details for http request and response like cache,content,stream data etc

More details about response


To summarize this post, this is list of tools, standalone applications, Firebug extensions and IE extensions that can be usefull in Http debugging process. Some of them you have to pay, some of them are free. But all are usefull in work or at least, they could be...