Re: Delete a table content
Re: Delete a table content
- Subject: Re: Delete a table content
- From: Chuck Hill <email@hidden>
- Date: Wed, 7 Sep 2005 09:50:58 -0700
Hi Pierre,
On Sep 6, 2005, at 11:35 PM, email@hidden wrote:
Hi!
When committing in batches, I found batch sizes of 200-300 changed
EOs to be the best. 5000 seems like too much.
If you do not need the undo manager, set it to null before starting
processing,
Have a look at the fetch spec batch iterator on my web site: http://
homepage.mac.com/i_love_my . It will not save you memory in this
precise case, but it might in similar set-ups.
In WO 5.2 or later, the editing context holds only weak references
to fetched EOs. Only modified EOs are strongly retained. Thus a
good evaluation of memory footprint is to sum the count of objects
you retain yourself with number of EOs with pending changes.
Have you tested this? My recollection is that there is a bug in
there and that the strong references to modified objects are never
released. That might be the undo manager, I'm not sure. I do recall
some discussions on the list that concluded that periodically
disposing the EC and creating a new one resulted in much lower memory
usage, contrary to what the docs say should happen.
Chuck
In your example, you retain up to 5000 InvoiceItem objects in a
list. It's the same 5000 objects that get modified. Upon
saveChanges the editing context releases its strong reference. With
the next loop iteration you do the same. Your maximum memory
footprint thus is the size of 5000 InvoiceItems. Reduce the batch
size and memory use will drop.
Had you mapped the too-many relationship from Invoice to
InvoiceItem, all InvoiceItems referenced would by strongly held
onto by the Invoice. Your memory footprint would thus depend on the
total number of InvoiceItems.
Pierre
webobjects-dev-bounces+pierre.bernard=email@hidden wrote
on 09/06/2005 10:23:00 AM:
> First of all I would like to thank you VERY MUCH for all the help
> you provided me : quick and clear each time :-)
>
> Concerning removing the inverse relationship, I will test that
> solution instead of setting the fid after I have finished that email
> (and I will remove any fid that is a class property in my model).
> I absolutely understand what the problem is. But I have to admit
> that I don't really understand what is going on behind the scene of
> EOs. And that leads me to stupid errors like the previous one.
> I had somewhere else in my code where I tried to do the same memory
> footprint improvement. But this time, I set the fid without having
> any relationship. Can that be a problem?
>
> And yes of course, that would be a very good idea to increase the
> JVM memory size but the hosting solution I will use don't give me
> access to such parameters :-(
>
> Concerning my needs, yes, I could post my model but I don't want to
> bother people with that.
> The only problem I had was to minimize the memory footprint of a
> portion of code like the following :
>
> I have thousands of Item I have to make an invoice from.
> I fetch them with a fetchLimit (5000 in my case)
>
>
> Invoice invoice = new Invoice();
> ec.insertObject(invoice);
> ...
> while (moreItems) {
> itemsList = ec.
> objectsWithFetchSpecification(fs); // with a fetch limit
> if (itemsList != null && itemsList.count
() > 0) {
> for (int i = 0; i < itemsList.count
(); i++) {
> // Update the item
> Item item = (Item)
itemsList.objectAtIndex(i);
> ...
>
> // Make the invoiceItem
> InvoiceItem item = new InvoiceItem
();
> ec.insertObject(item);
> ...
>
> }
> ec.saveChanges();
> ec.undoManager().removeAllActions();
> } else {
> moreItems = false;
> }
> }
>
> Now I will just add a item.setInvoice(invoice) instead of the
> invoice.addInvoiceItem(item)
> Does anyone has other ideas to minimize the footprint of a portion
> of code like this one?
>
> On Sep 5, 2005, at 9:21 PM, Ken Anderson wrote:
>
> Fabrice,
>
> If you're ending up with memory problems, it would be best to
> discuss how to solve them instead of solving the problem by
> crippling EOF. If you have a relationship that has thousands of
> records on the other side, why not consider eliminating the
> reflexive (reverse) relationship? It's a pretty common methodology
> employed to reduce the memory footprint of your app.
>
> If your invoice could have thousands of invoice items, maybe you
> should consider modeling the problem differently. Also, it's not
> uncommon to increase the JVM memory size to accommodate a typical
> webobjects application. It is considered extremely bad form to
> access the primary keys of your objects (they should, in fact, be
> non-class properties).
>
> The people on this list have built extremely large webobjects apps.
> If you would share your needs and your model, I'm sure we could come
> up with a good solution.
>
> Ken
>
> On Sep 5, 2005, at 11:48 AM, Fabrice Pipart wrote:
>
> I think you are absolutely right.
> I gave up the idea to change the rows with raw SQL. Loosing the sync
> was too bad for the rest of the application and it did not make
> sense to have to refetch all the objects !
>
> But...
>
> To avoid memory problems (performance and memory... I had lots of
> problems !), I found a solution to avoid OutOfMemory errors. And it
> seems that I end with almost the same problem as previously.
> Instead of setting the objects for a relationship, I set the fid :
> invoice.addToInvoiceItems(invoiceItem) became invoiceItem.
> setInvoiceFid(invoice.id())
> but again I loose sync... the invoice thinks it has no invoice items
> at the end :-(
>
> But now it is only ONE EO that I would have to refresh (instead of
> dealing with thousands of invoiceItems)
>
> Do you know how to do that (refresh the invoice EO)?
>
> Thanks
>
> Fabrice
>
> On Sep 5, 2005, at 2:59 PM, Ken Anderson wrote:
>
> Fabrice,
>
> Performance is only important if you need it. For deleting a lot of
> rows, it's certainly faster (and probably better), to delete the
> rows with database commands. You should, though, make sure that EOF
> is in sync with this change, which could be a lot of work. For
> updates, though, is that kind of performance worth the trade-off of
> not having everything in sync? Is it worth giving up the
> transaction and state management that EOF provides for you?
>
> It really all depends on your needs... both from a complexity point
> of view, and a performance point of view.
>
> To answer your question, any changes you make directly to the
> database through raw sql has the same issues as doing the update
> from some other application - EOF doesn't know about it. If you try
> to edit the EO that you modified, you will be presented with stale
> data, unless you refetch the EO AND tell it to refresh refetched
> objects. In the end, you'll lose a lot of the performance gain you
> got when going directly to the database, and you'll have more
> complicated software. Also, if you modify the stale EO and have a
> lock set on the attribute(s) you modified directly, you'll get an
> optimistic locking failure because the data in the snapshot doesn't
> match the data in the database.
>
> Ken
>
> On Sep 1, 2005, at 8:53 AM, Fabrice Pipart wrote:
>
> Finlly I used "delete from TABLE_NAME".
> And wow it IS 100 times faster than using EOs !
>
> Great !
> Thanks for the tip !
>
> And this gave me an idea, why not do raw SQL updates when I have to
> update one field on a large data set?
> And in fact it works, 100 times faster again.
>
> But is there any drawback doing it this way?
> Will the EOs I fetch after that, in sync with the database?
> It seems that it is not the case :-(
>
> Fabrice
>
> On Aug 30, 2005, at 6:10 PM, Nathan Walker wrote:
>
> I believe this is what you're looking for, Use this:
>
> sqlSelect = "truncate table yourtablename"; // NOTE: I use TRUNCATE
> here to delete all rows AND RESET Auto-Increment Fields
> EOUtilities.rawRowsForSQL(YourEC,"YourEOModelName",sqlSelect);
>
> You could also use "delete table..." there too, but that does not
> reset any auto-increment fields if you have any...
>
> That method is lightning FAST !
>
> On Aug 30, 2005, at 8:06 AM, Fabrice Pipart wrote:
>
> Hi list !
>
> I have a question that seems rather simple :
> is there a way to delete all the rows of a table without fetching
all of them?
> I have a table that is rather large and to avoid wasting a lot of
> memory I did not find a better way than using a fetchLimit and
> deleting each object until the table is empty...
>
> Any idea?
>
> Regards
>
> Fabrice Pipart
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
> 40visionworxpro.com
>
> This email sent to email@hidden
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
40anderhome.com
>
> This email sent to email@hidden
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
> [attachment "smime.p7s" deleted by Pierre Bernard/D3.4/BCL]
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
email@hidden
>
> This email sent to email@hidden
**********************************************************************
This email and any files transmitted with it are intended solely for
the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender
of this message. (email@hidden)
This email message has been checked for the presence of computer
viruses; however this protection does not ensure this message is
virus free.
Banque centrale du Luxembourg; Tel ++352-4774-1; http://www.bcl.lu
**********************************************************************
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40global-village.net
This email sent to email@hidden
--
Practical WebObjects - a book for intermediate WebObjects developers
who want to increase their overall knowledge of WebObjects, or those
who are trying to solve specific application development problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden