Re: Get the underlying cache block size of a filesystem?
Re: Get the underlying cache block size of a filesystem?
- Subject: Re: Get the underlying cache block size of a filesystem?
- From: Toby Thain <email@hidden>
- Date: Thu, 11 Feb 2010 11:55:04 -0500
On 11-Feb-10, at 11:20 AM, James Bucanek wrote:
Greetings,
Disclaimer: I'm a big propionate of "don't fight the OS" and "don't
reinvent the wheel", but sometimes you just have to. My application
presents a pathological case that is discordant with the
filesystem's natural caching algorithm, and I'm trying to find a
more efficient solution.
Scenario: I have a very large index file (GBs) that has millions of
very small records (12 bytes each). Records are read stochastically
while parsing an incoming stream of data. You can guess what
happens: (1) an entire of block of data is read from the disk and
cached, (2) 12 bytes are extracted, (3) the block eventually gets
pushed out of the cache by other file activity, process repeats.
The result is that almost every record read incurs the overhead of
reading the 4K/8K/32K/whatever block of data from the physical
media. This effectively "overreads" hundreds of potentially
interesting records in the same block, discards them, and reads
them again next time.
So the standard wisdom of "let the OS cache the data" isn't working
here.
Did you explore using a [R]DBMS?
--Toby
I've *significantly* improved performance by creating my own in-
memory cache records. ...
Thanks!
--
James Bucanek
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40telegraphics.com.au
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Filesystem-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden