About the Author

Chris Shiflett

Hi, I’m Chris: web craftsman, community leader, husband, father, and partner at Fictive Kin.


Inspecting and Hacking HTTP

There are numerous reasons you might want to inspect HTTP when debugging a problem. If you've ever tried to debug problems with sessions, cookies, or redirects, I'm sure you can appreciate how hard it is without taking a close look at what's going on behind the scenes.

There are numerous tools to help you inspect HTTP. If you're a Firefox user, you can use LiveHTTPHeaders or HttpFox. (Please feel free to suggest others.) If you use Safari, you can enable the debug menu:

$ defaults write com.apple.Safari IncludeDebugMenu 1

This gives you access to the Web Inspector, which displays HTTP headers under Network. Unfortunately, it seems to be limited to the headers, omitting everything else. (Safari users, are there better options?)

If you're using Safari 3, replace IncludeDebugMenu with IncludeDevelopMenu. You'll know it's enabled when you see the Develop menu.

Sometimes it's useful to make a slight modification to the request and try it again to see if it fixes the problem. Many HTTP tools let you do this, and tamperdata for Firefox is particularly useful.

Despite having numerous tools available, I still find myself using telnet pretty regularly, partly because I want to minimize the risk that the tool I'm using is giving me false information, whether it's due to a bug in the tool or simply an error in my interpretation or use of it.

If you've never used telnet to send requests to a web server, you might be surprised by how easy it is:

$ telnet google.com 80

Once connected, you can enter a simple GET request to try it out:

GET / HTTP/1.1
Host: google.com

Because Google prefers the superfluous www subdomain, you'll get a 301 response. If you want to follow the redirect like a browser would, simply change the Host header in your request, and you should get a 200 response that includes the content.

When debugging a secure URL, telnet won't help you. Luckily, openssl provides similar simplicity:

openssl s_client -connect google.com:443

You might prefer -quiet if you don't find the s_client output useful.

You can use the same GET request as before to try it out, or you can copy a real request your browser sends by using one of the HTTP inspectors like LiveHTTPHeaders.

Do you have any related tips for budding web developers?

About this post

Inspecting and Hacking HTTP was posted on Sun, 24 Aug 2008. If you liked it, follow me on Twitter or share:

31 comments

1.Evert said:

I can highly recommend Charles (http://www.charlesproxy.com/)

Charles is an HTTP debugging proxy, and I pretty much can't live without it.. I'm making all the developers install it and Q/A people when they find a problem

Sun, 24 Aug 2008 at 20:23:17 GMT Link


2.Chris Shiflett said:

Thanks for the suggestion, Evert.

By the way, URLs on their own line are linked automatically for you:

http://charlesproxy.com/

Sun, 24 Aug 2008 at 20:26:17 GMT Link


3.Matthew Turland said:

I find the Net tab in Firebug to be pretty useful. Lists request parameters, request and response headers, and response body.

http://getfirebug.com

Sun, 24 Aug 2008 at 20:43:44 GMT Link


4.Tim said:

I second Charles. It's an incredible tool that saves you a lot of time debugging. it's ability to match urls to local files is incredibly useful.

Sun, 24 Aug 2008 at 21:07:17 GMT Link


5.Pretty Coder said:

"HEAD example.com" (lwp-request) or "curl -I example.com" is what i use when in a hurry.

Sun, 24 Aug 2008 at 21:25:44 GMT Link


6.Jared said:

Microsoft's Fiddler is a free scriptable proxy/reverse proxy

http://www.fiddlertool.com/fiddler/

And there is little that beats HttpWatch, http://httpwatch.com/ , which it appears HttpFox has taken its inspiration from.

Sun, 24 Aug 2008 at 21:38:38 GMT Link


7.Chris Shiflett said:

Thanks for those URLs, Jared.

(I'll look into why the first URL isn't automatically linked per the style guide and get that fixed.)

Sun, 24 Aug 2008 at 21:41:46 GMT Link


8.Dejan Kozina said:

May I suggest Fiddler, a free and decent proxy?

http://www.fiddler2.com/

It does the trick for me.

Sun, 24 Aug 2008 at 21:54:02 GMT Link


9.Lorna Mitchell said:

I always start with curl for everything, it isn't the world's friendliest tool but it does rule out anything you can't see happening. I recently wrote a post on using curl: http://www.lornajane.net/posts/2008/Curl-Cheat-Sheet

Sun, 24 Aug 2008 at 22:20:07 GMT Link


10.Vid Luther said:

Great post Chris, I've been using telnet, curl and livehttpheaders, I wasn't aware of the openssl utility, I used to just disable https in the dev environment to test.. this will help me

Sun, 24 Aug 2008 at 22:24:11 GMT Link


11.Alexey Kupershtokh said:

I had been using an Http Analyzer utility for a long time before I moved from Windows to Ubuntu:

http://www.ieinspector.com/httpanalyzer/

It's paid though :(

Now using only the Firebug. Fortunately I don't need to sniff too much recently.

Mon, 25 Aug 2008 at 04:01:02 GMT Link


12.Andre Gironda said:

I'm with Matthew and Alexey -- FireBug is the only tool that I use. Surprisingly, I'm from a pure network engineering background where Wireshark and tcpdump ruled me for years. I was doing a little analysis today, and found that I don't even have Wireshark or windump installed, so I jumped right to FireBug to help me (the Net and Flash tabs helped instantly and I didn't have to wait for another tool to install or load).

After I downloaded the SWF file I was looking for, I used Trillix Flash Decompiler for an overview and then Sothink SWF Decompiler to dig deeper (or other Flash tools such as flasm, flare, swf-tools, swfmill, and all the other tools listed on flashsec.org). If it were my own Flash, I'd probably use the Firefox add-on, FlashTracer. Other than Flash, I can do almost everything else in FireBug.

FireBug has been invaluable as I move towards application and development roles. I use the FireCookie, YSlow, and FirePHP (all FireBug add-ons) whenever I can. One of my favorite features is right-click -> Copy XPath (in the HTML view). Combining this with `xmllint --html --shell' is almost/more useful than even curl on the command-line. The best tutorial for FireBug can be found in a SitePoint book called "The Art & Science of Javascript".

I also like LiveHttpHeaders (and the additional HeaderMonitor add-on is very nice!), but prefer ModifyHeaders to TamperData. TamperData does not allow you to modify both the request and response headers. Ideally, I'd move to a tool such as Burp Suite (http://portswigger.net) if I'm performing any tasks that require this level of complexity. Or possibly http://www.grendel-scan.com or http://code.google.com/p/ratproxy/

For Safari users, how about WebInspector, included with WebKit?

http://trac.webkit.org/wiki/Web%20Inspector

which also includes Drosera, Safari's Javascript debugger:

http://trac.webkit.org/wiki/Drosera

You can see an example of WebInspector UI with HTTP Headers here:

http://trac.webkit.org/attachment/w...urces%203.1.png

Mon, 25 Aug 2008 at 04:53:57 GMT Link


13.Mohammad said:

The Best HTTP Inspecting That I know.

1- http://httpwatch.com/

2- http://www.fiddlertool.com/fiddler/

3- http://wscarabeclipse.sourceforge.net/

4- http://www.httpdebugger.com/

Mon, 25 Aug 2008 at 05:15:24 GMT Link


14.Michael B Allen said:

WireShark is a little overkill for HTTP but that would be my first choice when debugging any network communication. Frequently problems with HTTP are not identified by looking at a single message. WireShark will show you framing information, TCP flags (SYN, ACK, RST, ...), DNS queries, chunk boundries, and so on.

To capture packets you can use WireShark's builtin capture controls. But since I'm usually working on several machines remotely at the same time, I'm in the habit if just using tcpdump to collect the capture:

# tcpdump -s 0 -w out.pcap ! port ssh and ! port 3389

The above could be read as "capture packets on all ports other than the ssh port and 3389 (Remote Desktop Protocol)". Add other ports as necessary to filter noise. Add -i lo to capture packets on the loopback interface.

After you run the above command as root it will wait and collect packets. After you perform the operation being observed, hit Ctrl-C to stop the capture. Then open the resulting out.pcap in WireShark.

Mon, 25 Aug 2008 at 06:29:22 GMT Link


15.Jordan Wiens said:

Similar to using telnet, I prefer netcat. You do have to sometimes be careful when dealing with nc/telnet on the command line as depending on the operating system you're connecting from and the one you're connecting to, you might have to translate linebreaks. Still, it's nice to be able to quickly and easily:

echo -e "GET / HTTP/1.0\nHOST: here\n\n"|nc there 80

Mon, 25 Aug 2008 at 17:22:13 GMT Link


16.Chris Shiflett said:

Thanks for the suggestions, everyone. These are great.

Jordan, I never considered using netcat, but that could be very useful when you need to be precise about line endings. Plus, it lets you inline the request, which means you can repeat it easily using your command history. Nice. :-)

Mon, 25 Aug 2008 at 18:03:58 GMT Link


17.Andre Gironda said:

I tested more add-ons with Firefox 3.0.1. Live HTTP Headers affects the Page Info (in a similar way that View Dependencies does). Used along with the Page Info Button add-on, this can be really nice (especially since Ctrl + I sends you to that screen).

CookieWatcher only works with FF3 if the cookie is an integer. HeaderMonitor is not working at all. JSView used to update Page Info, but it doesn't anymore.

And FireBug was updated today with better/full support for FF3!

Mon, 25 Aug 2008 at 23:40:57 GMT Link


18.Chris Shiflett said:

Thanks for the update, Andre. I didn't realize LiveHTTPHeaders altered Page Info.

(For Mac users, command-I is the shortcut.)

Tue, 26 Aug 2008 at 00:22:25 GMT Link


19.Cicatriz said:

I always use Burp as a penetration testing proxy. But in Firefox, I use TamperData.

Wed, 27 Aug 2008 at 20:52:44 GMT Link


20.Ed Finkler said:

Hate to come late and just third a recommendation, but Charles has been invaluable to me when developing desktop RIAs. Very nice tool.

Thu, 28 Aug 2008 at 01:59:44 GMT Link


21.Ask Bjoern Hansen said:

Bah. I wrote a longer comment (variations of using netcat) but it was tossed away with a "Your name may only contain alphabetic characters, spaces, hyphens, and apostrophes." error. Bad programmer, Chris!

Sun, 31 Aug 2008 at 09:47:07 GMT Link


22.Chris Shiflett said:

Crap. Sorry about that, Ask. It should at least save your form data. I'll look into that.

The reason I only allow certain characters is so that I can easily create a URL for the people who provide an OpenID to claim their identity. For example, here's mine:

http://shiflett.org/community/members/chris-shiflett

Do you know a good way to convert characters like ø to o, so that I can let people use their proper name and convert it to the closest URL-friendly equivalent?

Sun, 31 Aug 2008 at 16:53:45 GMT Link


23.Martin Jansen said:

Chris: You could probably give the //TRANSLIT feature of PHP's iconv() a try for converting foreign characters to plain ASCII.

Tue, 02 Sep 2008 at 12:59:42 GMT Link


24.Chris Shiflett said:

Thanks for the suggestion, Martin. I'll give that a go.

Thu, 04 Sep 2008 at 17:22:33 GMT Link


25.Brian Morin said:

I've used Fiddler2 forever and only recently started Charles.

Another combination I use for testing HTTP and repeating senarios is Badboy and JMeter.

Badboy : http://www.badboy.com.au/

JMeter: http://jakarta.apache.org/jmeter/

I can script out an advanced HTTP session with Badboy, then export the script to JMeter for load testing or unit tests with JUnit/JHTTPUnit/JMeter.

These packages make development fun! or... maybe I'm just a nerd.. :)

Mon, 08 Sep 2008 at 15:50:10 GMT Link


26.Tomalak Geret'kal said:

Firebug is a critical extension for any developer working on Firefox. Absolutely critical. Especially if you're developing with Javascript.

Wed, 10 Sep 2008 at 12:40:09 GMT Link


27.Matt Robinson said:

Wotcha Chris, thanks for the tip about headers in the web inspector, I hadn't noticed them! (Actually, took me a while to find them even when I knew they were there - you have to click on a row, but not on the filename or the action bubble.)

One thing though: in Safari 3, there's no need to mess with the defaults command; the option to show the Develop menu is right there in Preferences, under Advanced.

Fri, 14 Nov 2008 at 10:47:22 GMT Link


28.Andre Gironda said:

Also for Mac OS X:

http://ditchnet.org/httpclient/

Thu, 18 Dec 2008 at 16:08:06 GMT Link


29.Chris Shiflett said:

For my own reference, you can debug SNI using openssl with -servername. For example:

openssl s_client -connect example.org:443 -servername example.org

Fri, 10 Dec 2010 at 22:33:28 GMT Link


30.Stock Market Today said:

I suppose you gotta do it.

192.168.1.1

Sun, 12 Jun 2016 at 09:26:30 GMT Link


31.Kvasin Leonid said:

good job guys, found a lot of useful information!

192.168.1.1

Sun, 26 Jun 2016 at 08:51:30 GMT Link


Hello! What’s your name?

Want to comment? Please connect with Twitter to join the discussion.