Re: 64-bit/shared memory performance
Re: 64-bit/shared memory performance
- Subject: Re: 64-bit/shared memory performance
- From: Andreas Fink <email@hidden>
- Date: Wed, 9 Apr 2008 08:22:58 +0000
I can't comment on how the kernel does mmap in that respect but
there's probably ways around this by instead of launching separate
processes every time to use threads instead so they share the same
memory by default.
Or use a daemon which holds the memory and then answers queries by
using a socket interface or the like. Of course it depends if it only
has to search in >4G of data and return a s mall part or if its also a
question of using large chunks of that data as the query answer.
Launching/Killing processes is an expensive task. The initial memory
mapping probably as well.
On 09.04.2008, at 07:22, Jamil J. Weatherbee wrote:
I have grown extremely frustrated with the shm_open()/mmap()
interface.
In order to work with a shared memory object greater than 2GB I have
had to "chain" together multiple segments of an object created with
shm_open().
Now, here is my dilemma. I am specifically trying to avoid a
multithreaded client/server architecture as a very much (in theory
at this point) lower performance alternative to this.
I need to:
1) Create a LARGE contiguous shared memory area (i.e. 4GB+) which is
intended to actually be entirely resident in the physical RAM of the
machine.
2) Access the shared memory area for read only by many client
hundreds of processes in a speedy fashion.
Speedy meaning that at present I have to shm_open() and mmap()
together segments of 1GB in order to produce one large 4GB object in
the individual client processes. This works fine
but as soon as I try and READ the object it appears that the
kernel is doing some kind of "COPY ON READ" as you can see in "top"
that it takes about 6 seconds for RSIZE to grow to VSIZE.
I am unclear on why this is happening as I am no VM expert ---
especially since I am using the MAP_FIXED directive in mmap() so
that the addresses are all the same in every processes
virtual address space --- actually this is what makes the
chaining work. I can only imagine that since it is the VM system
writing some kind of kernel mappings.
3) What I need is the fastest, most overhead free manner of directly
accessing a segment of shared memory from multiple processes. These
processes are spawned and die often and quickly. Actually, they
are the result of .php scripts execing a small compiled C program on
the local machine. This would normally be the domain of a SQL
server or the like but this type of data access cannot be
efficiently handled inside of a standard RDBMS architecture. I need
top performance so the data query must be written in C and must have
read access to the data in memory (the 4GB of shared memory is
actually only an "index" to the real data which exists in secondary
storage).
Any help/suggestions would be greatly appreciated.
-Jamil
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden