Re: Delete a table content
Re: Delete a table content
- Subject: Re: Delete a table content
- From: email@hidden
- Date: Wed, 7 Sep 2005 08:35:47 +0200
Hi!
When committing in batches, I found
batch sizes of 200-300 changed EOs to be the best. 5000 seems like too
much.
If you do not need the undo manager,
set it to null before starting processing,
Have a look at the fetch spec batch
iterator on my web site: http://homepage.mac.com/i_love_my . It will not
save you memory in this precise case, but it might in similar set-ups.
In WO 5.2 or later, the editing context
holds only weak references to fetched EOs. Only modified EOs are strongly
retained. Thus a good evaluation of memory footprint is to sum the count
of objects you retain yourself with number of EOs with pending changes.
In your example, you retain up to 5000
InvoiceItem objects in a list. It's the same 5000 objects that get modified.
Upon saveChanges the editing context releases its strong reference. With
the next loop iteration you do the same. Your maximum memory footprint
thus is the size of 5000 InvoiceItems. Reduce the batch size and memory
use will drop.
Had you mapped the too-many relationship
from Invoice to InvoiceItem, all InvoiceItems referenced would by strongly
held onto by the Invoice. Your memory footprint would thus depend on the
total number of InvoiceItems.
Pierre
webobjects-dev-bounces+pierre.bernard=email@hidden
wrote on 09/06/2005 10:23:00 AM:
> First of all I would like to thank you VERY MUCH for all the help
> you provided me : quick and clear each time :-)
>
> Concerning removing the inverse relationship, I will test that
> solution instead of setting the fid after I have finished that email
> (and I will remove any fid that is a class property in my model).
> I absolutely understand what the problem is.
But I have to admit
> that I don't really understand what is going on behind the scene of
> EOs. And that leads me to stupid errors like the previous one.
> I had somewhere else in my code where I tried
to do the same memory
> footprint improvement. But this time, I set the fid without having
> any relationship. Can that be a problem?
>
> And yes of course, that would be a very good idea to increase the
> JVM memory size but the hosting solution I will use don't give me
> access to such parameters :-(
>
> Concerning my needs, yes, I could post my model but I don't want to
> bother people with that.
> The only problem I had was to minimize the memory
footprint of a
> portion of code like the following :
>
> I have thousands of Item I have to make an invoice from.
> I fetch them with a fetchLimit (5000 in my case)
>
>
>
Invoice invoice = new Invoice();
>
ec.insertObject(invoice);
>
...
>
while (moreItems) {
>
itemsList = ec.
> objectsWithFetchSpecification(fs); // with a fetch limit
>
if (itemsList != null && itemsList.count()
> 0) {
>
for (int i = 0; i <
itemsList.count(); i++) {
>
// Update
the item
>
Item item = (Item) itemsList.objectAtIndex(i);
>
...
>
>
// Make
the invoiceItem
>
InvoiceItem
item = new InvoiceItem();
>
ec.insertObject(item);
>
...
>
>
}
>
ec.saveChanges();
>
ec.undoManager().removeAllActions();
>
} else {
>
moreItems =
false;
>
}
>
}
>
> Now I will just add a item.setInvoice(invoice) instead of the
> invoice.addInvoiceItem(item)
> Does anyone has other ideas to minimize the footprint
of a portion
> of code like this one?
>
> On Sep 5, 2005, at 9:21 PM, Ken Anderson wrote:
>
> Fabrice,
>
> If you're ending up with memory problems, it would be best to
> discuss how to solve them instead of solving the problem by
> crippling EOF. If you have a relationship that has thousands
of
> records on the other side, why not consider eliminating the
> reflexive (reverse) relationship? It's a pretty common methodology
> employed to reduce the memory footprint of your app.
>
> If your invoice could have thousands of invoice items, maybe you
> should consider modeling the problem differently. Also, it's
not
> uncommon to increase the JVM memory size to accommodate a typical
> webobjects application. It is considered extremely bad form
to
> access the primary keys of your objects (they should, in fact, be
> non-class properties).
>
> The people on this list have built extremely large webobjects apps.
> If you would share your needs and your model, I'm sure we could come
> up with a good solution.
>
> Ken
>
> On Sep 5, 2005, at 11:48 AM, Fabrice Pipart wrote:
>
> I think you are absolutely right.
> I gave up the idea to change the rows with raw
SQL. Loosing the sync
> was too bad for the rest of the application and it did not make
> sense to have to refetch all the objects !
>
> But...
>
> To avoid memory problems (performance and memory... I had lots of
> problems !), I found a solution to avoid OutOfMemory errors. And it
> seems that I end with almost the same problem as previously.
> Instead of setting the objects for a relationship,
I set the fid :
> invoice.addToInvoiceItems(invoiceItem) became
invoiceItem.
> setInvoiceFid(invoice.id())
> but again I loose sync... the invoice thinks
it has no invoice items
> at the end :-(
>
> But now it is only ONE EO that I would have to refresh (instead of
> dealing with thousands of invoiceItems)
>
> Do you know how to do that (refresh the invoice EO)?
>
> Thanks
>
> Fabrice
>
> On Sep 5, 2005, at 2:59 PM, Ken Anderson wrote:
>
> Fabrice,
>
> Performance is only important if you need it. For deleting a
lot of
> rows, it's certainly faster (and probably better), to delete the
> rows with database commands. You should, though, make sure that
EOF
> is in sync with this change, which could be a lot of work. For
> updates, though, is that kind of performance worth the trade-off of
> not having everything in sync? Is it worth giving up the
> transaction and state management that EOF provides for you?
>
> It really all depends on your needs... both from a complexity
point
> of view, and a performance point of view.
>
> To answer your question, any changes you make directly to the
> database through raw sql has the same issues as doing the update
> from some other application - EOF doesn't know about it. If
you try
> to edit the EO that you modified, you will be presented with stale
> data, unless you refetch the EO AND tell it to refresh refetched
> objects. In the end, you'll lose a lot of the performance gain
you
> got when going directly to the database, and you'll have more
> complicated software. Also, if you modify the stale EO and have
a
> lock set on the attribute(s) you modified directly, you'll get an
> optimistic locking failure because the data in the snapshot doesn't
> match the data in the database.
>
> Ken
>
> On Sep 1, 2005, at 8:53 AM, Fabrice Pipart wrote:
>
> Finlly I used "delete from TABLE_NAME".
> And wow it IS 100 times faster than using EOs
!
>
> Great !
> Thanks for the tip !
>
> And this gave me an idea, why not do raw SQL updates when I have to
> update one field on a large data set?
> And in fact it works, 100 times faster again.
>
> But is there any drawback doing it this way?
> Will the EOs I fetch after that, in sync with
the database?
> It seems that it is not the case :-(
>
> Fabrice
>
> On Aug 30, 2005, at 6:10 PM, Nathan Walker wrote:
>
> I believe this is what you're looking for, Use this:
>
> sqlSelect = "truncate table yourtablename"; // NOTE:
I use TRUNCATE
> here to delete all rows AND RESET Auto-Increment Fields
> EOUtilities.rawRowsForSQL(YourEC,"YourEOModelName",sqlSelect);
>
> You could also use "delete table..." there too, but that
does not
> reset any auto-increment fields if you have any...
>
> That method is lightning FAST !
>
> On Aug 30, 2005, at 8:06 AM, Fabrice Pipart wrote:
>
> Hi list !
>
> I have a question that seems rather simple :
> is there a way to delete all the rows of a table
without fetching all of them?
> I have a table that is rather large and to avoid
wasting a lot of
> memory I did not find a better way than using a fetchLimit and
> deleting each object until the table is empty...
>
> Any idea?
>
> Regards
>
> Fabrice Pipart
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> _______________________________________________
> Do not post admin requests to the list. They
will be ignored.
> Webobjects-dev mailing list
(email@hidden)
> Help/Unsubscribe/Update your Subscription:
> 40visionworxpro.com
>
> This email sent to email@hidden
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> _______________________________________________
> Do not post admin requests to the list. They
will be ignored.
> Webobjects-dev mailing list
(email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
>
> www.icconsulting.mc
>
> International Corporate Consulting
> 25, Boulevard de Belgique
> MC - 98000 Monaco
>
> T. +377 97982104 (direct)
> F. +377 97708807
> [attachment "smime.p7s" deleted by
Pierre Bernard/D3.4/BCL]
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (email@hidden)
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
**********************************************************************
This email and any files transmitted with it are intended solely for
the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender
of this message. (email@hidden)
This email message has been checked for the presence of computer
viruses; however this protection does not ensure this message is
virus free.
Banque centrale du Luxembourg; Tel ++352-4774-1; http://www.bcl.lu
**********************************************************************
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden