site_archiver@lists.apple.com Delivered-To: darwin-kernel@lists.apple.com On 09.04.2008, at 07:22, Jamil J. Weatherbee wrote: Any help/suggestions would be greatly appreciated. -Jamil _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/afink%40list.fink.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel@lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.a... I can't comment on how the kernel does mmap in that respect but there's probably ways around this by instead of launching separate processes every time to use threads instead so they share the same memory by default. Or use a daemon which holds the memory and then answers queries by using a socket interface or the like. Of course it depends if it only has to search in >4G of data and return a s mall part or if its also a question of using large chunks of that data as the query answer. Launching/Killing processes is an expensive task. The initial memory mapping probably as well. I have grown extremely frustrated with the shm_open()/mmap() interface. In order to work with a shared memory object greater than 2GB I have had to "chain" together multiple segments of an object created with shm_open(). Now, here is my dilemma. I am specifically trying to avoid a multithreaded client/server architecture as a very much (in theory at this point) lower performance alternative to this. I need to: 1) Create a LARGE contiguous shared memory area (i.e. 4GB+) which is intended to actually be entirely resident in the physical RAM of the machine. 2) Access the shared memory area for read only by many client hundreds of processes in a speedy fashion. Speedy meaning that at present I have to shm_open() and mmap() together segments of 1GB in order to produce one large 4GB object in the individual client processes. This works fine but as soon as I try and READ the object it appears that the kernel is doing some kind of "COPY ON READ" as you can see in "top" that it takes about 6 seconds for RSIZE to grow to VSIZE. I am unclear on why this is happening as I am no VM expert --- especially since I am using the MAP_FIXED directive in mmap() so that the addresses are all the same in every processes virtual address space --- actually this is what makes the chaining work. I can only imagine that since it is the VM system writing some kind of kernel mappings. 3) What I need is the fastest, most overhead free manner of directly accessing a segment of shared memory from multiple processes. These processes are spawned and die often and quickly. Actually, they are the result of .php scripts execing a small compiled C program on the local machine. This would normally be the domain of a SQL server or the like but this type of data access cannot be efficiently handled inside of a standard RDBMS architecture. I need top performance so the data query must be written in C and must have read access to the data in memory (the 4GB of shared memory is actually only an "index" to the real data which exists in secondary storage). This email sent to afink@list.fink.org This email sent to site_archiver@lists.apple.com