November 2003 - Darwin-kernel

RE: Can a task on each of 2 processors share 7 GB of physical m emory
by Miller, Larry 15 Nov '03

15 Nov '03

It is not clear to me that mmap(2) can let a process get a pointer to a block of physical memory which has been previously wired and contains data that has been loaded oby a PCI-X card. It would be good if after suitable initialization, a file descriptor could refer to say a 7 GB file that is mapped entirely into pages in physical memory. The PCI-X card could load the data into those pages. And then a process on each of the two processord could access whatever part of it they wanted using mmap. Is this possible??? Larry -----Original Message----- From: Jim Magee [mailto:jmagee@apple.com] Sent: Friday, November 14, 2003 4:17 PM To: Miller, Larry Cc: '''darwin-kernel(a)lists.apple.com' ' ' Subject: Re: Can a task on each of 2 processors share 7 GB of physical m emory mmap(2)? On Nov 13, 2003, at 11:22 PM, Miller, Larry wrote: > Jim, > > What would be an efficient remapping technique so a given task can > quickly > get a window into a desired part of the physical memory? > > Larry > > > > -----Original Message----- > From: Jim Magee > To: Miller, Larry > Cc: ''darwin-kernel(a)lists.apple.com' ' > Sent: 11/13/2003 5:37 PM > Subject: Re: Can a task on each of 2 processors share 7 GB of > physical m > emory > > On Nov 13, 2003, at 11:58 AM, Miller, Larry wrote: >> By wired, I mean the data that I put in the memory remains at a fixed >> place >> in physical memory - it is not paged out. This would probably be >> equivalent >> to a fixed mapping from VM to physical memory. > > Then you probably could simultaneously wire 7GB out of 8GB physical as > you request. > >> I heard that OS-X Panther was in some ways a 64-bit OS. In other >> words, I >> thought tasks could execute 64-bit addressing load and store machine >> instructions. Maybe I am mistaken. >> >> Also, I maybe mistakenly thought that you could compile a C/C++ >> program in a >> way such that you could have 64-bit pointers that are dereferenced. > > There are lots of things that define a 64-bit support. Using 64-bit > addresses to access >4GB of mapped virtual memory in a single task is > only one (maybe two - depends upon how you count) such feature (which > happens to be absent in Panther/Darwin 7.x). The ability to access >> 4GB of physical of physical memory in the first place is also one. An > > additional feature is the ability to access and use 64-bit wide > registers in critical functions (and have the kernel preserve that > state correctly on interrupts/signals, etc...). These last two ARE in > Darwin 7.x. > >> If these things are not available, does anyone one have an idea of >> when they >> will become available in the future? > > As always, we can't make any statements about future Mac OS X plans > here or elsewhere. But from a technology point of view, it is > down-right inevitable that 64-bit address space support will make it > into Darwin - eventually. > > In the meantime, using multiple address spaces to wire the memory down > in, and (re)mapping techniques to gain access to some percentage of the > memory at any given time, should work. For most applications, this > should be acceptable (performance-wise). It's the cost of getting stuff > in and out of physical memory that is the real bottleneck. At first > reading, this seems to fit well with the "up to 256MB chunk" model of > memory allocation you suggested earlier. > > --Jim > _______________________________________________ > darwin-kernel mailing list | darwin-kernel(a)lists.apple.com > Help/Unsubscribe/Archives: > http://www.lists.apple.com/mailman/listinfo/darwin-kernel > Do not post admin requests to the list. They will be ignored. > _______________________________________________ > darwin-kernel mailing list | darwin-kernel(a)lists.apple.com > Help/Unsubscribe/Archives: > http://www.lists.apple.com/mailman/listinfo/darwin-kernel > Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Can a task on each of 2 processors share 7 GB of physical m emory
by Jim Magee 15 Nov '03

15 Nov '03

mmap(2)? On Nov 13, 2003, at 11:22 PM, Miller, Larry wrote: Jim, What would be an efficient remapping technique so a given task can quickly get a window into a desired part of the physical memory? Larry -----Original Message----- From: Jim Magee To: Miller, Larry Cc: ''darwin-kernel(a)lists.apple.com' ' Sent: 11/13/2003 5:37 PM Subject: Re: Can a task on each of 2 processors share 7 GB of physical m emory On Nov 13, 2003, at 11:58 AM, Miller, Larry wrote: By wired, I mean the data that I put in the memory remains at a fixed place in physical memory - it is not paged out. This would probably be equivalent to a fixed mapping from VM to physical memory. Then you probably could simultaneously wire 7GB out of 8GB physical as you request. I heard that OS-X Panther was in some ways a 64-bit OS. In other words, I thought tasks could execute 64-bit addressing load and store machine instructions. Maybe I am mistaken. Also, I maybe mistakenly thought that you could compile a C/C++ program in a way such that you could have 64-bit pointers that are dereferenced. There are lots of things that define a 64-bit support. Using 64-bit addresses to access >4GB of mapped virtual memory in a single task is only one (maybe two - depends upon how you count) such feature (which happens to be absent in Panther/Darwin 7.x). The ability to access 4GB of physical of physical memory in the first place is also one. An additional feature is the ability to access and use 64-bit wide registers in critical functions (and have the kernel preserve that state correctly on interrupts/signals, etc...). These last two ARE in Darwin 7.x. If these things are not available, does anyone one have an idea of when they will become available in the future? As always, we can't make any statements about future Mac OS X plans here or elsewhere. But from a technology point of view, it is down-right inevitable that 64-bit address space support will make it into Darwin - eventually. In the meantime, using multiple address spaces to wire the memory down in, and (re)mapping techniques to gain access to some percentage of the memory at any given time, should work. For most applications, this should be acceptable (performance-wise). It's the cost of getting stuff in and out of physical memory that is the real bottleneck. At first reading, this seems to fit well with the "up to 256MB chunk" model of memory allocation you suggested earlier. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Disk Image Plugin
by Thorsten Sprenger 14 Nov '03

14 Nov '03

Hi, is it possible do write some sort of Disk Image Plugin ? I want to develop a special image format which uses some sort of redundancy. I would prefer to use a plugin - if possible - instead of writing a KEXT which handles my special partition type. Thanks in advance ! /Thorsten _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel Extensions
by Mike Vannorsdel 14 Nov '03

14 Nov '03

You'll have to use the VFS functions (vn_open, vn_rdwr, vn_close, ect). This also requires a good grasp on vnodes, vnode reference counting, and vnode locking. You'll find doing file I/O from within the kernel is very different, and more difficult, than from userland. Your best bet is to find a book or online tutorials on using VFS. On Nov 13, 2003, at 12:34 PM, Matt Jaffa wrote: In a Kernel Extension, I am trying to do file reading and writing, how do i go about this, I included fstream but I dont think that this is supported. Any suggestions? _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel Extension
by Shawn Erickson 14 Nov '03

14 Nov '03

On Nov 11, 2003, at 7:05 PM, Matt Jaffa wrote: Do I really need to use Sockets and everything, All I want to do is have it where someone types in an internet address in any web browser on the computer, my app that is running behind the scenes will intercept any of those requests, and then if that address is in this database we have it will allow them to go there. At work we are just making this app to restrict others only to work related sites, so I quickly just want to make this thing. I just need to know how little snitch, intercepts the http(port 80) request and then after allowing it, the web browser shows it. On Nov 11, 2003, at 10:04 PM, Matt Jaffa wrote: The Problem with a web proxy is all you have to change the web proxy settings around and disable the blocks on the other ips and ports. Yes the people who use these computers will have sudo access, but if I design my app, then it will be able to have a non-sudo password associated with it, that they will create when installing my app. Just re-reading things... your application will have a non-sudo password associated with it? I don't follow this. Regardless your application, kext, etc. can be removed using sudo if the user has "admin" privileges on the system. You cannot prevent that. It would be trivial to bypass this. Again the only real solution to this is to firewall the internal network from the external world, blocking all ports (or at least the ones you care about). Then only allow access to websites via a web proxy that has a privileged way through the firewall. Using filtering in web proxy as needed. -Shawn _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel Extensions
by Shawn Erickson 14 Nov '03

14 Nov '03

On Nov 13, 2003, at 11:34 AM, Matt Jaffa wrote: In a Kernel Extension, I am trying to do file reading and writing, how do i go about this, I included fstream but I dont think that this is supported. Any suggestions? Don't? It is strongly discouraged attempting to do that. What are you doing that requires file access from inside the kernel? It is likely that a few alternate solutions exist. I assume this is in relation to the in kernel web site access filtering you are trying to do? -Shawn _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Kernel Extensions
by Justin Walker 14 Nov '03

14 Nov '03

On Thursday, November 13, 2003, at 11:34 AM, Matt Jaffa wrote: In a Kernel Extension, I am trying to do file reading and writing, how do i go about this, I included fstream but I dont think that this is supported. I'm surprised that you even succeeded in compiling the KEXT. This is not only "not supported", it can't work. You cannot use "user mode" libc in the kernel. I would suggest that you look at two sources of information: - the Apple developer website - the darwin-development/darwin-kernel archives (http://lists.apple.com) There should be plenty of discussion there about the advisability of doing file I/O from the kernel (executive summary: don't). Mac OS X provides other means for doing this; you should pursue them. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | Men are from Earth. | Women are from Earth. | Deal with it. *--------------------------------------*-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

RE: Can a task on each of 2 processors share 7 GB of physical m emory
by Miller, Larry 14 Nov '03

14 Nov '03

Jim, What would be an efficient remapping technique so a given task can quickly get a window into a desired part of the physical memory? Larry -----Original Message----- From: Jim Magee To: Miller, Larry Cc: ''darwin-kernel(a)lists.apple.com' ' Sent: 11/13/2003 5:37 PM Subject: Re: Can a task on each of 2 processors share 7 GB of physical m emory On Nov 13, 2003, at 11:58 AM, Miller, Larry wrote: > By wired, I mean the data that I put in the memory remains at a fixed > place > in physical memory - it is not paged out. This would probably be > equivalent > to a fixed mapping from VM to physical memory. Then you probably could simultaneously wire 7GB out of 8GB physical as you request. > I heard that OS-X Panther was in some ways a 64-bit OS. In other > words, I > thought tasks could execute 64-bit addressing load and store machine > instructions. Maybe I am mistaken. > > Also, I maybe mistakenly thought that you could compile a C/C++ > program in a > way such that you could have 64-bit pointers that are dereferenced. There are lots of things that define a 64-bit support. Using 64-bit addresses to access >4GB of mapped virtual memory in a single task is only one (maybe two - depends upon how you count) such feature (which happens to be absent in Panther/Darwin 7.x). The ability to access >4GB of physical of physical memory in the first place is also one. An additional feature is the ability to access and use 64-bit wide registers in critical functions (and have the kernel preserve that state correctly on interrupts/signals, etc...). These last two ARE in Darwin 7.x. > If these things are not available, does anyone one have an idea of > when they > will become available in the future? As always, we can't make any statements about future Mac OS X plans here or elsewhere. But from a technology point of view, it is down-right inevitable that 64-bit address space support will make it into Darwin - eventually. In the meantime, using multiple address spaces to wire the memory down in, and (re)mapping techniques to gain access to some percentage of the memory at any given time, should work. For most applications, this should be acceptable (performance-wise). It's the cost of getting stuff in and out of physical memory that is the real bottleneck. At first reading, this seems to fit well with the "up to 256MB chunk" model of memory allocation you suggested earlier. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Can a task on each of 2 processors share 7 GB of physical m emory
by Jim Magee 14 Nov '03

14 Nov '03

On Nov 13, 2003, at 11:58 AM, Miller, Larry wrote: By wired, I mean the data that I put in the memory remains at a fixed place in physical memory - it is not paged out. This would probably be equivalent to a fixed mapping from VM to physical memory. Then you probably could simultaneously wire 7GB out of 8GB physical as you request. I heard that OS-X Panther was in some ways a 64-bit OS. In other words, I thought tasks could execute 64-bit addressing load and store machine instructions. Maybe I am mistaken. Also, I maybe mistakenly thought that you could compile a C/C++ program in a way such that you could have 64-bit pointers that are dereferenced. There are lots of things that define a 64-bit support. Using 64-bit addresses to access >4GB of mapped virtual memory in a single task is only one (maybe two - depends upon how you count) such feature (which happens to be absent in Panther/Darwin 7.x). The ability to access >4GB of physical of physical memory in the first place is also one. An additional feature is the ability to access and use 64-bit wide registers in critical functions (and have the kernel preserve that state correctly on interrupts/signals, etc...). These last two ARE in Darwin 7.x. If these things are not available, does anyone one have an idea of when they will become available in the future? As always, we can't make any statements about future Mac OS X plans here or elsewhere. But from a technology point of view, it is down-right inevitable that 64-bit address space support will make it into Darwin - eventually. In the meantime, using multiple address spaces to wire the memory down in, and (re)mapping techniques to gain access to some percentage of the memory at any given time, should work. For most applications, this should be acceptable (performance-wise). It's the cost of getting stuff in and out of physical memory that is the real bottleneck. At first reading, this seems to fit well with the "up to 256MB chunk" model of memory allocation you suggested earlier. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

RE: Can a task on each of 2 processors share 7 GB of physical m emory
by Quinn 14 Nov '03

14 Nov '03

At 8:58 -0800 13/11/03, Miller, Larry wrote: By wired, I mean the data that I put in the memory remains at a fixed place in physical memory - it is not paged out. This would probably be equivalent to a fixed mapping from VM to physical memory. OK, there's no limit to the amount of memory that you can wire in this manner. I heard that OS-X Panther was in some ways a 64-bit OS. In other words, I thought tasks could execute 64-bit addressing load and store machine instructions. Maybe I am mistaken. Also, I maybe mistakenly thought that you could compile a C/C++ program in a way such that you could have 64-bit pointers that are dereferenced. On current versions of Mac OS X all processes have a 32-bit address space and run in 32-bit mode. sizeof(char *) is always 4. If these things are not available, does anyone one have an idea of when they will become available in the future? At 9:58 +0000 13/11/03, Quinn wrote: Apple is aware of the need to support 64-bit address spaces. However, I'm not authorised to say anything about our plans in this area. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0