April 2007 - Darwin-kernel - lists.apple.com

Info needed for File System Development
by Shailesh Parulekar 26 Apr '07

26 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello, It shall be of great help to me. Thanks, --shailesh _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I need to develop a Filter driver to plug-in into the MAC File System in order to 'filter' the read/writes to the files and process the same as per the need of our product. I have been doing kernel programming since long but I have worked only on Unix/ Linux. I am completely new to MAC OS. Also I have never worked in the File System Area. Can anyone suggest me links/ documents/ white-papers which explain: 1. kernel programming on MAC (essentially for File System Filter Driver) and 2. File system Internals in detail (BSD File System) _________________________________________________________________ The average US Credit Score is 675. The cost to see yours: $0 by Experian. http://www.freecreditreport.com/pm/default.aspx?sc=660600&bcd=EMAILFOOTERAV… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything wrong with x86_64 and SYSV shared memory allocation?
by Kevin Murphy 24 Apr '07

24 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Terry, Regards, Kevin Murphy _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 24, 2007, at 8:00 AM, Terry Lambert wrote: On Apr 23, 2007, at 7:33 AM, Kevin Murphy wrote: What I'm actually trying to do is compile 64-bit PostgreSQL, which compiles fine using "-arch x86_64" in CFLAGS, but it fails its regression tests because it can't allocate memory with shmget (ENOMEM is returned). The 32-bit PostgreSQL compiles and works fine. It's trying to allocate the memory in the lower 4G of the process virtual address space, and unless you turn off the 4G page 0 mapping (or munamp() it after you're running but before you attempt to allocate shared memory), it's not going to see any free address space available. Thanks very much. I've forward the information to the PostgreSQL community. Since I'm not a hacker, my role at this point will be that of interested bystander. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything wrong with x86_64 and SYSV shared memory allocation?
by Jonas Maebe 24 Apr '07

24 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 24 apr 2007, at 14:47, Terry Lambert wrote: On 24 apr 2007, at 14:00, Terry Lambert wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 24, 2007, at 5:13 AM, Jonas Maebe wrote: Oh, and Chris also suggests using System V semaphores instead of POSIX semaphores in Postgres, to avoid the fd pressure But then you get the problem of leaking semaphores in case a program crashes or is abnormally terminated, no? (requiring manual cleanup with ipcs) Or you just use the same key file with ftok(), which will get you the same key each time, and always attempt to delete them at program startup, ignoring the error if they weren't there to delete because you didn't crash/weren't aborted. Of course, POSIX semaphores can be forced to be resource tracked using sem_unlink() after the last sem_open(), but once they are unlinked, subsequent sem_open()'s fail until everyone has drained out their references through either _exit() or sem_close(), so they are more or less in the same boat. Whether the cleanup happens because of a sem_unlink() at startup by path, or an ftok() by path to get a key, and a semctl() call, you are explicitly cleaning things up, either way, with either interface. I'm mainly interested in this from a compiler run time library point of view, which offers a generic synchronisation object. Currently I'm indeed using sem_open/sem_unlink. The name is constructed on the fly based on a.o. the process id and thread id, so it should never conflict with another process/thread. In this scenario, cleaning up in case of collisions is not really an option, since you may run out of available semaphores before you get a conflict to clean up. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything wrong with x86_64 and SYSV shared memory allocation?
by Terry Lambert 24 Apr '07

24 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 24 apr 2007, at 14:00, Terry Lambert wrote: -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 24, 2007, at 5:13 AM, Jonas Maebe wrote: Oh, and Chris also suggests using System V semaphores instead of POSIX semaphores in Postgres, to avoid the fd pressure But then you get the problem of leaking semaphores in case a program crashes or is abnormally terminated, no? (requiring manual cleanup with ipcs) Or you just use the same key file with ftok(), which will get you the same key each time, and always attempt to delete them at program startup, ignoring the error if they weren't there to delete because you didn't crash/weren't aborted. Of course, POSIX semaphores can be forced to be resource tracked using sem_unlink() after the last sem_open(), but once they are unlinked, subsequent sem_open()'s fail until everyone has drained out their references through either _exit() or sem_close(), so they are more or less in the same boat. Whether the cleanup happens because of a sem_unlink() at startup by path, or an ftok() by path to get a key, and a semctl() call, you are explicitly cleaning things up, either way, with either interface. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything wrong with x86_64 and SYSV shared memory allocation?
by Jonas Maebe 24 Apr '07

24 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 24 apr 2007, at 14:00, Terry Lambert wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Oh, and Chris also suggests using System V semaphores instead of POSIX semaphores in Postgres, to avoid the fd pressure But then you get the problem of leaking semaphores in case a program crashes or is abnormally terminated, no? (requiring manual cleanup with ipcs) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Anything wrong with x86_64 and SYSV shared memory allocation?
by Terry Lambert 24 Apr '07

24 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Anything I need to know? I double-checked the result with a SYSV shared memory demo program (google for shared-mem.c). *** /Users/chrismarcellino/postgresql-8.2.2/src/template/darwin Fri Mar 10 20:38:40 2006 --- /Users/chrismarcellino/darwin Wed Feb 7 13:56:01 2007 *************** *** 4,8 **** # (Note: on OS X before 10.2, you might need -traditional-cpp instead) CC="$CC -no-cpp-precomp" ! # Select appropriate semaphore support ! USE_NAMED_POSIX_SEMAPHORES=1 --- 4,16 ---- # (Note: on OS X before 10.2, you might need -traditional-cpp instead) CC="$CC -no-cpp-precomp" ! # Select appropriate semaphore support. Darwin 6.0 (Mac OS X 10.2) and higher ! # support System V semaphores for better peformance. ! case $host_os in ! darwin[015].*) ! USE_NAMED_POSIX_SEMAPHORES=1 ! ;; ! *) ! USE_SYSV_SEMAPHORES=1 ! ;; ! esac -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 23, 2007, at 7:33 AM, Kevin Murphy wrote: I've previously posted to darwin-dev and darwin-x86 lists with no response. Whenever I try to use shmem to allocate SYSV shared memory on an XServe Quad Xeon (OS X 10.4.9, latest XCode) using the "-arch x86_64" compiler switch, I get ENOMEM returned. There's plenty of SYSV memory allocated via sysctl.conf, and the same code works fine with "-arch i386" (or "-arch ppc64" on a G5). This box has 16GB RAM, although that is presumably irrelevant. What I'm actually trying to do is compile 64-bit PostgreSQL, which compiles fine using "-arch x86_64" in CFLAGS, but it fails its regression tests because it can't allocate memory with shmget (ENOMEM is returned). The 32-bit PostgreSQL compiles and works fine. It's trying to allocate the memory in the lower 4G of the process virtual address space, and unless you turn off the 4G page 0 mapping (or munamp() it after you're running but before you attempt to allocate shared memory), it's not going to see any free address space available. This is a consequence of a tools change that made that memory unavailable under normal circumstances, to avoid the TLB flushes on a CR3 reload that's necesswary if program data and the kernel occupy the same virtual address ranges. This is a difference between the PPC 64 bit and Intel 64 bit implementations (but PPC will likely follow eventually). You should also consider using shm_open() instead; the most recent patches (February of this year) for this seem to be here: <http://www.mail-archive.com/pgsql-patches@postgresql.org/ msg14808.html> And since POSIX shared memory isn't as limited as System V shared memory, there's a nice performance win, as well (you will still need to map it below the 4G mark, however). Note that due to limitations in the Postgres implementation, you will also not be able to mix 32 and 64 bit clients of the shared memory segment, given that Size is typedef'ed to size_t, and therefore that's 8 bytes on LP64 and 4 bytes on ILP32 (this is one reason why using sized types is just a good idea). Oh, and Chris also suggests using System V semaphores instead of POSIX semaphores in Postgres, to avoid the fd pressure (binary backward compatibility is a bear in some places): This email sent to site_archiver(a)lists.apple.com

1 0

Anything wrong with x86_64 and SYSV shared memory allocation?
by Kevin Murphy 23 Apr '07

23 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Thunderbird 1.5.0.10 (Windows/20070221) I've previously posted to darwin-dev and darwin-x86 lists with no response. Anything I need to know? I double-checked the result with a SYSV shared memory demo program (google for shared-mem.c). Thanks, Kevin Murphy _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Whenever I try to use shmem to allocate SYSV shared memory on an XServe Quad Xeon (OS X 10.4.9, latest XCode) using the "-arch x86_64" compiler switch, I get ENOMEM returned. There's plenty of SYSV memory allocated via sysctl.conf, and the same code works fine with "-arch i386" (or "-arch ppc64" on a G5). This box has 16GB RAM, although that is presumably irrelevant. What I'm actually trying to do is compile 64-bit PostgreSQL, which compiles fine using "-arch x86_64" in CFLAGS, but it fails its regression tests because it can't allocate memory with shmget (ENOMEM is returned). The 32-bit PostgreSQL compiles and works fine. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Speedstep AppleTV kernel
by Terry Lambert 21 Apr '07

21 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Apr 20, 2007, at 3:07 PM, Dan Penney wrote: I've been looking to implement the Speedstep portion of the FreeBSD kernel into the mach kernel. I know how I want to do this, I just can't figure out where I should add the code for it. Thanks. The processor advanced power management features are already supported. The FreeBSD code is actually less featureful than what's already in there. This email sent to site_archiver(a)lists.apple.com

1 0

Speedstep AppleTV kernel
by Dan Penney 20 Apr '07

20 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=TBE6gy0FInPWsPo6jiITHq2CsJx0OBSMyuSnf3DSohYecMRkS3HJOhcPLdBbdEWeqVcnzqPnROabHvCMFkeXyBhtnA6BU96vPJoZhhYXFpMB6SFdI1LS7XSBSgKhyIs7r3wiWPAzyEe9o+8OuxE5h91UFSo6csCTjbUPPcn8YGE= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=fyWM5kAwnJSYeSWi0YHFfHN4bN0x8rX6kCcLEJZ9aIMxjSko9yutopLSpzbrNycuscjLpRuHfggLAq/kPYmSqRpv1V3SzXXbjtWHAqUbVd0mJD7zwGHUDhnwvVUqF8tw/TrABScr+bsxwOjDdwAR6m2Klu+YY5fdP9lViaKPRKY= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Problem with EDID Data Retrival
by amber maheshwari 19 Apr '07

19 Apr '07

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=E0Kfbe6vliFxC55gQRr36/5Y8dtLN1DrNN5bHqhcR5a+HqVRmeLYJP2hpKlBQkRLp+UwR4G3SpPcR4X6i3Ujvt9RkuhfFph8u/iD3wWrvoEVmT6KuQ9CZkoQfn25QYSDFUP31Ld71/1eg1o7CtJIt/XKZ4odKq9hqlvSiIT+lcw= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=tNTXej4OYUsArgjngCsvjKARUCV8o+IQ6h/3tj2vzLonSVOXaHdb7TW8GruXYLj25Vfe14bBjM62r41dhTb0HJZ1ccbJc+rr4jxtbmPIdPx9LWMAkqKRnnAUFAM+JCLXwe3aJL5dTf38UciHOquQ5y/uMi80OqT9zsvwl4ptz+0= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0