Re: [Wonder-disc] EOF & Multithreading
Re: [Wonder-disc] EOF & Multithreading
- Subject: Re: [Wonder-disc] EOF & Multithreading
- From: Chuck Hill <email@hidden>
- Date: Fri, 11 Jul 2008 13:11:29 -0700
On Jul 11, 2008, at 12:51 PM, randy wigginton wrote:
Thanks for the suggestion. It turns out that a raw JDBC connection
with result set operates about 12x the speed of EOF for reading. I
don't need the assistance of EOF for these objects, so I'm going to
go straight to the DB.
In cases like this, I think that is the most appropriate solution.
Chuck
On Fri, Jul 11, 2008 at 10:51 AM, Chuck Hill <chill@global-
village.net> wrote:
On Jul 11, 2008, at 8:15 AM, Miguel Arroz wrote:
Hi!
I suppose you could use multiple EOF stacks, but I don't know if it
will perform better or not. It depends on what you call "waiting for
data".
If "waiting for data" is waiting for the data to be read from the
hard drive to memory, then you will make it even worse, because you
will cause more seek events, reducing the efective speed.
If "waiting for data" is waiting for Java to convert the raw data
to objects, then yes, you may gain a significative speed boost *if*
your machine is multi-core or multi-CPU, because you will be able to
do that conversion in parallel, using one core/CPU to each EOF stack.
Or consider using raw rows instead of EOs.
Chuck
Also, think is you can transfer your work for the DB itself. Like
if you are counting rows with certain characteristics, etc. If you
are working with so much data in a read-only fashion, it might be
interesting to push much of the work to the DB itself, to avoid all
the expensive overhead of fetching the data to WO, converting it to
objects, manage the memory usage, etc.
That is probably the best advice. EOF is not tuned for bulk
handling of massive data sets.
Chuck
On 2008/07/11, at 16:04, Randy Wigginton wrote:
I am working on an analysis application that reads tens of millions of
rows in order to analyze website behavior. Because the application is
so read-intensive, it currently spends 90% of its time waiting for
data. To address this, I am considering moving the DB reading into
one or more separate threads in order to more fully utilize the DB and
application servers. A search did not turn up anything that I found
useful.
Has anyone on this list done this previously? Are there any kind of
guidelines, best practices, etc?
Thanks for any guidance!
-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
Wonder-disc mailing list
email@hidden
https://lists.sourceforge.net/lists/listinfo/wonder-disc
http://www.survs.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden