Re: Did we have a bot ?
Re: Did we have a bot ?
- Subject: Re: Did we have a bot ?
- From: Owen McKerrow <email@hidden>
- Date: Tue, 21 Jun 2005 09:59:49 +1000
I checked the log files at it was a bot from www.archive.org so we
updated our robots.txt file, added in the <META> tag people had
suggested and even put a block on the fire wall for request from that
IP address. Suffice to say one of those worked :)
On 21/06/2005, at 3:01 AM, Chuck Hill wrote:
I think what you probably had was a poorly written page slurper (app
that saves pages locally). This has happened to us as well. If you
were logging out the headers you can check the user-agent to see which
application it was. I don't know what you can do about this other
than to ban certain user agents as you catch them.
Chuck
On Jun 19, 2005, at 4:38 PM, Owen McKerrow wrote:
Hi Guys,
Arrived at work this morning and discovered 235 error messages
waiting for us. And they were all a variation on ........
EXCEPTION:
<Application>: Unable to create page
'8CWzvTSKgnrbXl3tkKeTW0/www.uow.edu.au/educ/'.
DATE/TIME:
2005-06-19 23:05:19 Etc/GMT
STACK TRACE:
at pageWithName(WOApplication.java:1968)
at _dispatchWithPreparedSession(WOComponentRequestHandler.java:273)
at
_dispatchWithPreparedApplication(WOComponentRequestHandler.java:322)
at _handleRequest(WOComponentRequestHandler.java:358)
at handleRequest(WOComponentRequestHandler.java:432)
at dispatchRequest(WOApplication.java:1306)
at dispatchRequest(Application.java:164)
at runOnce(WOWorkerThread.java:173)
at run(WOWorkerThread.java:254)
at run(Thread.java:NA)
SESSION INFORMATION:
SessionID: htSjk9NnUPbIxdLw8Nydtw
and the URL grew from there......
EXCEPTION:
<Application>: Unable to create page
'8CWzvTSKgnrbXl3tkKeTW0/www.uow.edu.au/educ/www.uow.edu.au/its/'.
and grew......
EXCEPTION:
<Application>: Unable to create page
'8CWzvTSKgnrbXl3tkKeTW0/www.uow.edu.au/science/www.uow.edu.au/
commerce/,www.uow.edu.au/www.uow.edu.au/law/'.
Did we have a bot vist us ? What is the best way of dealing with Bots
and WebObjects ?
Owen McKerrow
WebMaster, emlab
http://emlab.uow.edu.au
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - -
'The test of a first-rate intelligence is the ability to hold two
opposed ideas in the mind at the same time and still be able to
function.'
-F.Scott Fitzgerald,
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
village.net
This email sent to email@hidden
--
Practical WebObjects - a book for intermediate WebObjects developers
who want to increase their overall knowledge of WebObjects, or those
who are trying to solve specific application development problems.
http://www.global-village.net/products/practical_webobjects
Owen McKerrow
WebMaster, emlab
http://emlab.uow.edu.au
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - -
People who prefer typing to pointing then seem to prefer acronyms to
save typing :-)
-Denis Stanton, On people using Command Line Interfaces
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden