Re: How to read files from disk directly?
Re: How to read files from disk directly?
- Subject: Re: How to read files from disk directly?
- From: Stan Sieler <email@hidden>
- Date: Tue, 05 Jul 2011 16:46:07 -0700
On Jul 4, 2011, at 4:07 PM, Eric Gorr wrote:
> What I know is that I can read a single 4gb file in about 3 minutes. It takes around 20-25 minutes to read 4gb spread among 190,000 files.
>
> I do not know for sure what the true bottleneck is, only that reading the same amount of data over a lot of files takes a lot longer.
I was curious about this, so I did a test...
Every night, my Mac creates a file listing all files in the system. This file,
/var/root/allfiles, currently has 1,114,268 filenames in it.
I timed how long it took to: (a) read the file, (b) read the file and open()/close() /dev/null
for each record, and (c) read the file and open()/close() each filename.
(Thus, I hope to have the data that allows me to find the per-disk-file cost of an open()/close() pair.)
Results (Mac Pro, 16 "core", 12 GB RAM):
Took 4,794 CPU milliseconds in 6 seconds (no open)
Took 15,648 CPU milliseconds in 19 seconds (open (/dev/null), close ())
Took 67,045 CPU milliseconds in 411 seconds (open (filename), close ())
# files opened: 1,142,518
# files failed to open: 1,405
Errno distribution:
2: 1,336
62: 4
102: 65
Looking only at the elapsed ("wall") times...
The cost of open()/close() of a normal disk file is about
354 microseconds of elapsed ("wall") time.
The cost of open()/close() of /dev/null is about 11 microseconds.
If we assume a short-cut path for /dev/null (an assumption that might not be justified), we see
that the "walk the HFS" and "allocate buffers" (to the extent a normal file has different
buffers than /dev/null) is about 343 microseconds.
(Interestingly, the app I use to create the list of 1,114,268 filenames (via opendir/readdir and qsort)
takes about 460 elapsed seconds to run through the entire HFS directory, writing the list to a disk file.)
(All files are on the boot disk, an internal SATA drive, HFS, test run as root)
Hope that's interesting :)
Stan
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden