Re: CoreData - large data set is slow to load on app launch - optimization tips?
Re: CoreData - large data set is slow to load on app launch - optimization tips?
- Subject: Re: CoreData - large data set is slow to load on app launch - optimization tips?
- From: Georg Tuparev <email@hidden>
- Date: Sun, 4 Dec 2005 18:40:03 +0100
Ruslan,
This thread is getting out of hand... Before replying it always make
sense to get your hands dirty with a technology. And please be
technical!
On Dec 4, 2005, at 5:26 PM, Ruslan Zasukhin wrote:
On 12/4/05 3:47 PM, "Georg Tuparev" <email@hidden> wrote:
Hi Georg,
I believe that few days ago I have ask on this list:
is CoreData supposed to be ONLY for simply things as
preferences and e.g. storing of windows positions?
Answer was -- CoreData can be used for something more complex.
And someone told you that you have no idea what you are talking about
but you preferred to ignore this. Position of windows and preferences
you store in NSUserDefaults.
I don't follow your logic here, Georg. I am agreeing above that
CoreData is
for something more complex. Yes or no?
I do not know about you, but I always told people that CoreData is
capable to be used in serious applications. I do not know how I can
say it more clearly. I also tell people that I had also occasionally
observed some slowness. But in all these cases it was my fault - just
suboptimal design of my object model. I love to tell people about my
mistakes, and I wonder why some of the cocoa developers always try to
blame the technology about their own mistakes - well probably it is
easier.
[cocoa irrelevant paragraphs removed] + I am still waiting for your
replacement of CoreData!
Here an example of what CoreData - just
doing this: a 20 GB large astronomical catalog spread in 50+ tables.
So far there is no query that takes me more then 1s - but I do not do
naive programming either :-)
I can easy show that your claims also can be taken negative.
* may be your db has 19.9 GB images?
I mention I am not a naive developer. Also if you thought a bit you
would know that 100M is not enough to catalogue even an amateur
astronomical catalogue :-)
No, images go to the filesystem...
* may be the most complex query you do is a simplest query as
select from T where ID = xx
How about this one: find 5 objects that need to be checked in the
coming hour, are close to the meridian, but during exposure time do
not cross it, seeing conditions are so that they have to be at least
45Deg away of a half moon, should not get close to any solar system
transit, the light curve could be taken without derotator, no filter
is required, the observations will have high scientific value
(calculated by a weighting function accessing few tables), are part
of an ongoing survey, and for their project there is a budget for at
least 10 hours observation time? Sounds good enough? Easily 30 tables
are touched. Oh, and the object tables are really huge, and the
coordinates are spherical.... and of course there is a time
conversion...
then FYI, FileMaker also is very fast on such operation :-)
* even if divide 20Gb to 50+ tables this is only 300-400MB per table
including indexes (again what about images?).
When I say developing intelligently, I mean it. The 20 GB is not 2TB
because the program has to run on an old laptop. So instead of
telling the client to connect to a RAID it is very simple (if one
things about it) to just have locally the data of the objects that
could be seen from my place during the next 3 days. This is called
proactive CPU-cycle avoiding :-)
* joins? group by ? Recursive joins ? No ?
How do you think, could the query above run without them?
Most importantly though, for client apps
it is the user experience that couts. If my UI can show 10 rows at a
time, why I need to fetch 100000000 and then complain that it is
slow?
Hmm, you looks to be experienced developer but say strange things.
Really?
May be you do not know that often/sometimes. to get 10 records of
result you
need make joins on several tables, then build GROUP BY, and so on ???
May be you do not know that even for 1-2 Gb it is possible to catch
query
which work minutes (this words of my ISP friend about mySQL)
I prefer others to talk about my work, but I will just mention that
my team designed and implemented arguably the largest data warehouse
project in europe (for a large chemical manufacturer), and for
several years this project was listed by apple as one of the WO-based
flagships. It is already few years old, but I believe I still
remember what is "group by" :-)
You happy with 1 second? Great. But somebody can wish 0.01 second.
My opinion is completely irrelevant. If the app is good for the
clients, then it is also good for me. BTW, could you please tell me
how one will (in terms of human physiology) notice the difference
between 0.01 and say 0.02s in application responsiveness? And in the
real world, everything has its price. If I tell my customers that I
can run the query 10 times faster but they need to cough 10k extra, I
am sure most of them will be happy with my lousy 1s :-)
------------
At last. It is interesting to listen people which use Windows, and
which
have spend zero time on MacOS, that they are happy with Windows. It
is fun
to see how changes their opinion after work with MacOS.
So I think its most valuable to hear the opinions of developers who
try and
use many databases...not just Valentina, SQLite, mySQL.
Well, I can sleep well, because I meet your criteria - what a
relief :-) Before we started with our astronomy projects we tested
many databases fully loaded with TBs of data, as well as some non-DB
based specialized solution (e.g. JPL's package to calculate orbital
elements for satellites). Combination of FrontBase and home grown in
memory indexing turned out to be the winner. Well, most of the
databases we tasted (including some prominent names) never managed to
even load these "astronomical" amounts of data... Some of the result
of our survey were published, some we are not allowed to disclose.
later...
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden