Re: Design pattern for bulk data handling
Re: Design pattern for bulk data handling
- Subject: Re: Design pattern for bulk data handling
- From: Tito Ciuro <email@hidden>
- Date: Sat, 08 Feb 2014 14:54:35 -0800
Also, consider dropping the table indexes before you start inserting. After the you're done inserting, rebuild the indexes. This should give you an an additional performance boost.
For even more speed, consider turning off the following setting:
http://www.sqlite.org/pragma.html#pragma_synchronous
*warning*: read the link carefully and understand what it does before using it!
-- Tito
> On Feb 8, 2014, at 14:25, Jens Alfke <email@hidden> wrote:
>
>
>> On Feb 8, 2014, at 10:20 AM, Ben <email@hidden> wrote:
>>
>> This is fine for most things, except that I sometimes need faster access to the underlying database - for example, when importing/exporting data. In these cases I'm after bulk data throughput without the overhead of creating/destroying many NSOperations with completion handlers since there can be in the order of millions of statements to handle.
>
> You can read arbitrarily large amounts of data with a single statement, so that's not an issue. But in general it takes many statements to insert a lot of rows. (And for performance reasons you really want to group all of those statements in a single transaction, or you'll lose an order of magnitude of performance.)
>
> What I'd do is provide a new operation type that does a bulk-insert or bulk-update. When run the operation performs a series of SQL statements starting with a "BEGIN" and ending with an "END".
>
> —Jens
> _______________________________________________
>
> Cocoa-dev mailing list (email@hidden)
>
> Please do not post admin requests or moderator comments to the list.
> Contact the moderators at cocoa-dev-admins(at)lists.apple.com
>
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden