Re: Design pattern for bulk data handling
Re: Design pattern for bulk data handling
- Subject: Re: Design pattern for bulk data handling
- From: Ben <email@hidden>
- Date: Sun, 09 Feb 2014 11:56:34 +0000
Thank you for the comments everyone, however I think I must have phrased the question badly. I am not looking for assistance on speeding up SQLite, more for designing a cleaner API that works well with this. I am already pretty familiar with SQLite and it's various options.
To try and clarify:
I have an database object that looks something like this:
@interface MyDB : NSObject
- (id)initWithFileAtURL:(NSURL *)fileURL;
- (id)executeQuery:(NSString *)sql completionHandler:(myBlock)handler;
@end
Using -executeQuery:completionHandler: passes the sql string to an NSOperation which does the database-touching work and passes results back through the completion block.
I'm now trying to work out a tidy API of executing (up to) millions of SQL statements without building up a huge queue of operations. After sleeping on the matter, my current idea is to add a method to the MyDB class like:
- (void)importFromURL:(NSURL *)fileURL withOptions:(NSDictionary *)options completionHandler:(myBlock)handler;
This would allow me to pass key/value pairs with options for table or column mappings and have all the database-specific API grouped in one place.
- Ben
On 8 Feb 2014, at 18:20, Ben <email@hidden> wrote:
> Hi list,
>
> I'm looking for the right design pattern for providing different types of access to an SQLite database.
>
> Currently I have a database object where queries are run on a serial NSOperationQueue and each operation has a completion block for reporting its results. The operation queue is an implementation detail behind a more basic API.
>
> This is fine for most things, except that I sometimes need faster access to the underlying database - for example, when importing/exporting data. In these cases I'm after bulk data throughput without the overhead of creating/destroying many NSOperations with completion handlers since there can be in the order of millions of statements to handle.
>
> In the past I had simply broken encapsulation by exposing the underlying database engine API. I'd rather find a better method than this for the future.
>
> Options considered so far:
>
> 1. Continue exposing underlying DB engine
> 2. Add a simpler synchronous API for use on a separate thread
> 3. Expose NSOperationQueue and allow custom import operations to be queued
> 4. Add importing functions to the database object itself
>
> Does anyone have any suggestions as to how best to approach this problem?
>
> - Ben
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden