• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Applescript-users Digest, Vol 2, Issue 720
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Applescript-users Digest, Vol 2, Issue 720


  • Subject: Re: Applescript-users Digest, Vol 2, Issue 720
  • From: Daniel Jalkut <email@hidden>
  • Date: Fri, 28 Oct 2005 10:22:36 -0400

I wouldn't assume you have lynx. I don't.

I think you probably have tcpdump, however, which can be used to examine the network interchange between client and server.

It can be difficult to figure out which options to use to get the "readable dump" you're looking for. I recommend something like the following:

tcpdump -Atq -s 0 -i en1 host www.mytargethost.com

Two parts of that you want to change are:

1. "-i en1" which can be omitted or changed to "-i en0" if you're not using Airport to get to the net.
2. "www.mytargethost.com" which should be changed to the name or IP address of the site you're connecting to. This will help filter out all the other network noise coming out of your computer.


I also suggest running this from a Terminal whose "scrollback buffer" has been set to unlimited. Then, when you're about to do a test load, press Cmd-K to clear the Terminal. Go do the test load, then examine the terminal contents. You can cmd-F to search for helpful strings like "POST" or "GET" to see the most significant parts of your HTTP requests. For instance, just below the "GET" you'll see the headers, including User-Agent. This is what it looks like if I load http://www.apple.com/ from Safari:

GET / HTTP/1.1
Accept: */*
Accept-Language: en
Accept-Encoding: gzip, deflate
Cookie: [Omitted From this Email]
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/ 412.7 (KHTML, like Gecko) Safari/412.5
If-Modified-Since: Thu, 27 Oct 2005 17:37:46 GMT
Connection: keep-alive
Host: www.apple.com


If I do the same request from Curl, I get the much more meager:

GET / HTTP/1.1
User-Agent: curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g zlib/1.2.3
Host: www.apple.com
Pragma: no-cache
Accept: */*


Curl can of course be manipulated to be less meager. In your case, you might do something like this:

curl -A "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/ 412.7 (KHTML, like Gecko) Safari/412.5" --cookie "bob=nancy; blah=pookie; your=cookiesHere;" http://www.apple.com/

Which pops up in tcpdump as the following:

GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/ 412.7 (KHTML, like Gecko) Safari/412.5
Host: www.apple.com
Pragma: no-cache
Accept: */*
Cookie: bob=nancy; blah=pookie; your=cookiesHere;


Looking a lot more like Safari! Just copy the cookies text out of the Safari dump and into your "--cookie" parameter to Safari. This should be a good start for you. If it doesn't quite work, you may have a situation where redirections need to be followed, or where curl needs to accept updated cookies from the server. If you provide a --cookie-jar parameter to curl, it will not only read cookies from that file, but will write them back
to the file if the server updates them. This is the holy grail in simulating a "real browser."


Good luck!

Daniel

On Oct 28, 2005, at 9:52 AM, John R. wrote:

I assume I have lynx, and I'd love it if it works. I tried curl and perl's LWP modules. It seems to me that Lynx would hang me up for the same reason: the need for precise emulation of the http parameters needed for my proprietary databases, as I said the prior postings. I could not even get access to the site with what I thought were correct password authentication and user-agent info. Clearly the problem is my own lack of experience reverse engineering http protocol exchanges. Is there a tool for debugging and/or listening to the data transfers that you guys use to figure out what is going on?


_______________________________________________ Do not post admin requests to the list. They will be ignored. Applescript-users mailing list (email@hidden) Help/Unsubscribe/Update your Subscription: This email sent to email@hidden
  • Follow-Ups:
    • Re: Applescript-users Digest, Vol 2, Issue 720
      • From: Daniel Jalkut <email@hidden>
References: 
 >Re: Applescript-users Digest, Vol 2, Issue 720 (From: "John R." <email@hidden>)

  • Prev by Date: Re: Applescript-users Digest, Vol 2, Issue 720
  • Next by Date: Re: curl equivalence
  • Previous by thread: Re: Applescript-users Digest, Vol 2, Issue 720
  • Next by thread: Re: Applescript-users Digest, Vol 2, Issue 720
  • Index(es):
    • Date
    • Thread