Re: Applescript-users Digest, Vol 2, Issue 720
Re: Applescript-users Digest, Vol 2, Issue 720
- Subject: Re: Applescript-users Digest, Vol 2, Issue 720
- From: Daniel Jalkut <email@hidden>
- Date: Fri, 28 Oct 2005 10:22:36 -0400
I wouldn't assume you have lynx. I don't.
I think you probably have tcpdump, however, which can be used to
examine the network interchange between client and server.
It can be difficult to figure out which options to use to get the
"readable dump" you're looking for. I recommend something like the
following:
tcpdump -Atq -s 0 -i en1 host www.mytargethost.com
Two parts of that you want to change are:
1. "-i en1" which can be omitted or changed to "-i en0" if you're
not using Airport to get to the net.
2. "www.mytargethost.com" which should be changed to the name or IP
address of the site you're connecting to. This will help filter out
all the other network noise coming out of your computer.
I also suggest running this from a Terminal whose "scrollback buffer"
has been set to unlimited. Then, when you're about to do a test
load, press Cmd-K to clear the Terminal. Go do the test load, then
examine the terminal contents. You can cmd-F to search for helpful
strings like "POST" or "GET" to see the most significant parts of
your HTTP requests. For instance, just below the "GET" you'll see the
headers, including User-Agent. This is what it looks like if I load
http://www.apple.com/ from Safari:
GET / HTTP/1.1
Accept: */*
Accept-Language: en
Accept-Encoding: gzip, deflate
Cookie: [Omitted From this Email]
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5
If-Modified-Since: Thu, 27 Oct 2005 17:37:46 GMT
Connection: keep-alive
Host: www.apple.com
If I do the same request from Curl, I get the much more meager:
GET / HTTP/1.1
User-Agent: curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1
OpenSSL/0.9.7g zlib/1.2.3
Host: www.apple.com
Pragma: no-cache
Accept: */*
Curl can of course be manipulated to be less meager. In your case,
you might do something like this:
curl -A "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5" --cookie "bob=nancy;
blah=pookie; your=cookiesHere;" http://www.apple.com/
Which pops up in tcpdump as the following:
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/
412.7 (KHTML, like Gecko) Safari/412.5
Host: www.apple.com
Pragma: no-cache
Accept: */*
Cookie: bob=nancy; blah=pookie; your=cookiesHere;
Looking a lot more like Safari! Just copy the cookies text out of
the Safari dump and into your "--cookie" parameter to Safari. This
should be a good start for you. If it doesn't quite work, you may
have a situation where redirections need to be followed, or where
curl needs to accept updated cookies from the server. If you provide
a --cookie-jar parameter to curl, it will not only read cookies from
that file, but will write them back
to the file if the server updates them. This is the holy grail in
simulating a "real browser."
Good luck!
Daniel
On Oct 28, 2005, at 9:52 AM, John R. wrote:
I assume I have lynx, and I'd love it if it works. I tried curl and
perl's LWP modules. It seems to me that Lynx would hang me up for
the same reason: the need for precise emulation of the http
parameters needed for my proprietary databases, as I said the prior
postings. I could not even get access to the site with what I
thought were correct password authentication and user-agent info.
Clearly the problem is my own lack of experience reverse
engineering http protocol exchanges. Is there a tool for debugging
and/or listening to the data transfers that you guys use to figure
out what is going on?
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Applescript-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden