Re: CSV parsing (large files)
Re: CSV parsing (large files)
- Subject: Re: CSV parsing (large files)
- From: Simone Tellini <email@hidden>
- Date: Wed, 30 Jul 2008 08:14:15 +0200
Il giorno 30/lug/08, alle ore 06:55, Jacob Bandes-Storch ha scritto:
I've got several large-size CSV files (a total of about 1.25 million
lines, and an average of maybe 10 or so columns) that I want to
parse into a 2D array. I found some parsing code that uses
NSScanner, and it works fine with small files, but it's very
resource-intensive and slow with large files. Should I try and
optimize it more, or try using C instead of Objective-C for parsing,
or what?
The ultimate goal here is to convert the CSV files into a SQLite
database. Should I even be doing it this way? Maybe just one row
into the database at a time? I'd like to hear people's opinions.
Thanks in advance!
for that kind of job, I'd stick to C, probably using strtok().
Keep a prepared SQLite insert statement and reuse it for all the
lines, binding the parameters for each line. Don't load the whole file
in memory: it's just a waste of memory and time to allocate it.
Instead parse a line at a time.
Also, you might want to issue "PRAGMA synchronous = NORMAL" or "PRAGMA
synchronous = OFF" before everything else (look at its meaning at http://www.sqlite.org/pragma.html)
--
Simone Tellini
http://tellini.info
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden