November 2003 - Darwin-kernel

Re: Can a task on each of 2 processors share 7 GB of physical m emory
by Shawn Erickson 13 Nov '03

13 Nov '03

On Nov 13, 2003, at 8:58 AM, Miller, Larry wrote: Quinn, By wired, I mean the data that I put in the memory remains at a fixed place in physical memory - it is not paged out. This would probably be equivalent to a fixed mapping from VM to physical memory. Does it need to be fixed in a physical address location? or just prevented from being paged out? I heard that OS-X Panther was in some ways a 64-bit OS. In other words, I thought tasks could execute 64-bit addressing load and store machine instructions. Maybe I am mistaken. Panther (and 10.2.7/8) have some level of support for 64b physical addressing but it is not exposed at the process level yet. Aspects of this ability is exposed in the kernel for dealing with taking 64b physical addresses and mapping things such that you can DMA using 32b addresses for PCI, etc. The VM sub-system supports the use of 64b physical addresses (to use the 8GB available in G5s for example) for mapping to process but processes are still limited to 32b. Also, I maybe mistakenly thought that you could compile a C/C++ program in a way such that you could have 64-bit pointers that are dereferenced. You could get the compiler to do that but 10.3 (or 10.2.7/8) do not support 64b addressing for processes nor does any of user-land OS frameworks/libraries/APIs support 64b addresses. If these things are not available, does anyone one have an idea of when they will become available in the future? Any such future information is under NDA (just about as it always is for Apple). If you have a strong business need to know this information consider filing a DTS incident with Apple (http://developer.apple.com/products/techsupport/index.html) By doing so you may get the information you need (under NDA of course). You will not get such information on this email list. I believe on the "drivers" list you may have gotten a suggestion on a possible short term solution (not reviewed the solution yet myself). -Shawn _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Capturing file system events
by Brian Bergstrand 13 Nov '03

13 Nov '03

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ken, You should take a look at KQueues. These allow you to be notified of file system events (and other types) in close to real time. No need for a KEXT either. KQueues are only available in Darwin 7 (Panther) though. Check out /usr/include/sys/event.h. HTH. On Thursday, November 13, 2003, at 08:56 AM, Ken Tozier wrote: > A project I'm working on needs to be able to capture the following > three file system events in order to write those changes to a log > file: > - Container changed events (when file is moved from one directory to > another on the same volume) > - Name changed events > - File add events (when files are added to watch folders or any of the > watch folder's sub folders from a different volume) > > AppleScript's folder actions are in the ball park conceptually, but > their inability to operate on closed folders and the fact that if > something in a deeply nested folder hierarchy changes, those changes > don't propagate to higher level directories, make folder actions > unsuitable for my needs. > > Would a Kext be the place to capture these types of file events in > real time? > > Ken Tozier > Brian Bergstrand <http://www.classicalguitar.net/brian/> PGP Key: <http://www.classicalguitar.net/brian/misc/public_key.txt> The United States is a nation of laws: badly written and randomly enforced. - Frank Zappa As of 04:20:57 PM, iTunes is playing "Head Down" from "Superunknown" by "Soundgarden" -----BEGIN PGP SIGNATURE----- Version: PGP 8.0.2 iQA/AwUBP7P213nR2Fu2x7aiEQI/KwCfanjNrZehZAtu5dF9Cn8DgSZxdeoAn0nK MXIDZsIe0H5H17TLnCOOkLh4 =4L70 -----END PGP SIGNATURE----- _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Kernel Extensions
by Matt Jaffa 13 Nov '03

13 Nov '03

In a Kernel Extension, I am trying to do file reading and writing, how do i go about this, I included fstream but I dont think that this is supported. Any suggestions? Thanks, Matt _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Capturing file system events
by Jim Magee 13 Nov '03

13 Nov '03

On Nov 13, 2003, at 9:56 AM, Ken Tozier wrote: A project I'm working on needs to be able to capture the following three file system events in order to write those changes to a log file: - Container changed events (when file is moved from one directory to another on the same volume) - Name changed events - File add events (when files are added to watch folders or any of the watch folder's sub folders from a different volume) AppleScript's folder actions are in the ball park conceptually, but their inability to operate on closed folders and the fact that if something in a deeply nested folder hierarchy changes, those changes don't propagate to higher level directories, make folder actions unsuitable for my needs. Would a Kext be the place to capture these types of file events in real time? There is no need for a kext. Depending upon the filesystem type in question (I'll assume local HFS+ or UFS), you can use kqueue(2) and kevent(2) syscalls to watch folder/directory changes like these. Now, nothing in those APIs does the automatic recursion to sub-folders/directories, but you can do that yourself. The directory files must be held open (so limits on filedescriptors per process may have to be raised). There is also a new open(2) flag (O_EVTONLY) that won't hold up volume unmounts while watching for changes. If you are writing a Carbon application, and only need to see changes done at the Finder/application level (not backdoor changes by command-line applications) you can use higher level notifications like FNNotify(). --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

RE: Can a task on each of 2 processors share 7 GB of physical m emory
by Miller, Larry 13 Nov '03

13 Nov '03

Quinn, By wired, I mean the data that I put in the memory remains at a fixed place in physical memory - it is not paged out. This would probably be equivalent to a fixed mapping from VM to physical memory. I heard that OS-X Panther was in some ways a 64-bit OS. In other words, I thought tasks could execute 64-bit addressing load and store machine instructions. Maybe I am mistaken. Also, I maybe mistakenly thought that you could compile a C/C++ program in a way such that you could have 64-bit pointers that are dereferenced. If these things are not available, does anyone one have an idea of when they will become available in the future? Thanks, Larry -----Original Message----- From: Quinn To: 'darwin-kernel(a)lists.apple.com' Sent: 11/13/2003 1:58 AM Subject: Re: Can a task on each of 2 processors share 7 GB of physical memory At 18:37 -0800 12/11/03, Miller, Larry wrote: >I would like to be able to wire these blocks. The G5 has two different types of wired memory. There's the traditional definition of wired, that is that the virtual to physical memory translation can't change, and then there's a new concept of "wired for I/O". Wiring memory for I/O enters it into a memory management unit that translates 32-bit PCI addresses into > 32-bit processor physical addresses (the DART). Which of these forms of wiring are you interested in? >And I would like to have a >task on each of the two processors have direct access (thru pointer >dereferencing) to this 7 GB worth of memory. This indicates that you want to access > 4 GB of memory from a single process via direct pointer access. Do I understand you correctly? If so, this can't be done on G5 systems running current versions of Mac OS X. Currently all processes have a 32-bit address space and run in 32-bit mode. Therefore no process can directly access more than 4 GB of memory (the limit is less if you consider the address space consumed for frameworks etc). That's not to say that the G5's large physical memory is useless. It allows you to run multiple processes, each with a huge working set. You can also indirectly access large memory spaces by changing the memory mappings within your process. However, both of these are outlawed by your problem statement. Apple is aware of the need for supporting 64-bit address spaces. However, I'm not authorised to say anything about our plans in this area. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Negative vm_stat results...
by William Kucharski 13 Nov '03

13 Nov '03

>I've been evaluating a dual processor G5, working it pretty hard for >several weeks (2 x ~100% CPU jobs running all the time.) I just >noticed the following virtual memory statistics: > >dberry% vm_stat >Mach Virtual Memory Statistics: (page size of 4096 bytes) >Pages free: 4799. >Pages active: 140469. >Pages inactive: 90789. >Pages wired down: 26087. >"Translation faults": -1283718751. >Pages copy-on-write: 599196. >Pages zero filled: -1293420378. >Pages reactivated: 1004467. >Pageins: 29024. >Pageouts: 2675. >Object cache: 325107 hits of 618313 lookups (52% hit rate) No big deal here; the statistics above are captured to an "integer_t," rather than a "natural_t," and the code that updates the statistics makes no effort to make sure the results do not exceed INT_MAX. So yes, it's just a 32-bit signed int that's "rolling over" into negative values. William Kucharski kucharsk(a)mac.com _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Capturing file system events
by Ken Tozier 13 Nov '03

13 Nov '03

A project I'm working on needs to be able to capture the following three file system events in order to write those changes to a log file: - Container changed events (when file is moved from one directory to another on the same volume) - Name changed events - File add events (when files are added to watch folders or any of the watch folder's sub folders from a different volume) AppleScript's folder actions are in the ball park conceptually, but their inability to operate on closed folders and the fact that if something in a deeply nested folder hierarchy changes, those changes don't propagate to higher level directories, make folder actions unsuitable for my needs. Would a Kext be the place to capture these types of file events in real time? Ken Tozier _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Virtual memory weirdness?
by Quinn 13 Nov '03

13 Nov '03

At 17:38 -0500 12/11/03, Alex Reynolds wrote: Pages zero filled: -1293420378. [...] Any thoughts? I believe that the negative numbers are simply a case of the counters wrapping. Looking through the code I don't think anything sinister is going on. The kernel stores these counters as signed 32-bit integers and just increments them each time an event happens. vm_stat gets the counters (using host_statistics HOST_VM_INFO) and prints them using printf %d, which interprets them as signed and hence you see a negative number. AFAICT the kernel does not base any runtime decisions on these counters, so I don't think they're the cause of your instability. Feel free to file a bug about this. On the hardware available when Mach was designed, wrapping a 32-bit integer was really hard. On current hardware it's pretty easy. We probably should increase the width of the counters. <http://developer.apple.com/bugreporter/> S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Can a task on each of 2 processors share 7 GB of physical memory
by Quinn 13 Nov '03

13 Nov '03

At 18:37 -0800 12/11/03, Miller, Larry wrote: I would like to be able to wire these blocks. The G5 has two different types of wired memory. There's the traditional definition of wired, that is that the virtual to physical memory translation can't change, and then there's a new concept of "wired for I/O". Wiring memory for I/O enters it into a memory management unit that translates 32-bit PCI addresses into > 32-bit processor physical addresses (the DART). Which of these forms of wiring are you interested in? And I would like to have a task on each of the two processors have direct access (thru pointer dereferencing) to this 7 GB worth of memory. This indicates that you want to access > 4 GB of memory from a single process via direct pointer access. Do I understand you correctly? If so, this can't be done on G5 systems running current versions of Mac OS X. Currently all processes have a 32-bit address space and run in 32-bit mode. Therefore no process can directly access more than 4 GB of memory (the limit is less if you consider the address space consumed for frameworks etc). That's not to say that the G5's large physical memory is useless. It allows you to run multiple processes, each with a huge working set. You can also indirectly access large memory spaces by changing the memory mappings within your process. However, both of these are outlawed by your problem statement. Apple is aware of the need for supporting 64-bit address spaces. However, I'm not authorised to say anything about our plans in this area. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Can a task on each of 2 processors share 7 GB of physical memory
by Miller, Larry 13 Nov '03

13 Nov '03

On an 8 GB Dual G5 Power Mac with OS-X, I would like to allocate a set of blocks of physical memory totalling 7 GB (roughly). The blocks themselves might be say 64 KB to 256 MB each, preferably on the higher end. I am flexible on this. I would like to be able to wire these blocks. And I would like to have a task on each of the two processors have direct access (thru pointer dereferencing) to this 7 GB worth of memory. Is this possible? (I asked this question in an email to the drivers list, but then I realized this might be a better list to ask it in). Thanks for any help. Larry _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0