Re: Reading in dictionary from txt file: options for speed
Re: Reading in dictionary from txt file: options for speed
- Subject: Re: Reading in dictionary from txt file: options for speed
- From: Marcel Weiher <email@hidden>
- Date: Wed, 15 Apr 2009 15:35:27 -0700
On Apr 14, 2009, at 11:12 , Miles wrote:
I'm trying to find the best way to load in a 2MB text file of
dictionary
words and be able to do quick searches.
Simply loading the uncompressed txt file takes about 0.5 seconds
which I can
handle.
What do you do to load the txt file? 0.5 seconds for 2 MB seems quite
a bit.
[snip]
I'm not super concerned about the 2MB of disk space the txt file
takes up,
although I wouldn't be mad about decreasing it somehow. And once I
get the
whole dictionary in an array, the searches are basically fast enough
for my
purposes.
I would do the following:
1. map the file into memory using -[NSData
dataWithContentsOfMappedFile:] (or mmap() if you really want to)
2. Do not convert to individual objects for the words
3. get the pointer to the raw bytes
4. search using a little bit of plain old C (assuming you're OK with
encodings)
Memory mapping will be essentially instantaneous, with the I/O
performed on-demand when its actually needed (or you can pre-heat the
data, for example in a background thread). More importantly, you will
be doing good things for memory consumption, because the mapped memory
can be released to the OS without having to kill your app in low-
memory situations (without paging it out on Mac OS X, but the iPhone
doesn't page memory out).
I added an implementation of this approach to the testing program
provided by Wagner (thanks!) and it loads + counts the words in 0.084
seconds on the device. That's anywhere from around 50 - 100 times
faster than the other methods implemented in DictTest (plist / xml /
txt ). On the simulator, it runs in 0.043 seconds, so around 30-40
times faster than the other methods.
Download can be found here:
http://www.metaobject.com/downloads/Objective-C/DictTest.tgz
You mentioned that you were OK with search performance, so I won't go
into that.
Cheers,
Marcel
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden