1. "-i en1" which can be omitted or changed to "-i en0" if you're
not using Airport to get to the net.
2. "www.mytargethost.com" which should be changed to the name or IP
address of the site you're connecting to. This will help filter out
all the other network noise coming out of your computer.
I also suggest running this from a Terminal whose "scrollback buffer"
has been set to unlimited. Then, when you're about to do a test
load, press Cmd-K to clear the Terminal. Go do the test load, then
examine the terminal contents. You can cmd-F to search for helpful
strings like "POST" or "GET" to see the most significant parts of
your HTTP requests. For instance, just below the "GET" you'll see the
headers, including User-Agent. This is what it looks like if I load
http://www.apple.com/ from Safari:
GET / HTTP/1.1
Accept: */*
Accept-Language: en
Accept-Encoding: gzip, deflate
Cookie: [Omitted From this Email]
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5
If-Modified-Since: Thu, 27 Oct 2005 17:37:46 GMT
Connection: keep-alive
Host: www.apple.com
If I do the same request from Curl, I get the much more meager:
Curl can of course be manipulated to be less meager. In your case,
you might do something like this:
curl -A "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5" --cookie "bob=nancy;
blah=pookie; your=cookiesHere;" http://www.apple.com/
Which pops up in tcpdump as the following:
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5
Host: www.apple.com
Pragma: no-cache
Accept: */*
Cookie: bob=nancy; blah=pookie; your=cookiesHere;
Looking a lot more like Safari! Just copy the cookies text out of
the Safari dump and into your "--cookie" parameter to Safari. This
should be a good start for you. If it doesn't quite work, you may
have a situation where redirections need to be followed, or where
curl needs to accept updated cookies from the server. If you provide
a --cookie-jar parameter to curl, it will not only read cookies from
that file, but will write them back
to the file if the server updates them. This is the holy grail in
simulating a "real browser."
Good luck!
Daniel
On Oct 28, 2005, at 9:52 AM, John R. wrote:
I assume I have lynx, and I'd love it if it works. I tried curl and
perl's LWP modules. It seems to me that Lynx would hang me up for
the same reason: the need for precise emulation of the http
parameters needed for my proprietary databases, as I said the prior
postings. I could not even get access to the site with what I
thought were correct password authentication and user-agent info.
Clearly the problem is my own lack of experience reverse
engineering http protocol exchanges. Is there a tool for debugging
and/or listening to the data transfers that you guys use to figure
out what is going on?
