September 2002 - Darwin-kernel

Re: What should we do when headers are missing in 10.2?
by Michael Cashwell 11 Sep '02

11 Sep '02

On Wednesday, September 11, 2002, at 11:30 AM, Jim Magee wrote: It should be in /usr/include/netinet/ip_var.h. It should also be in /System/Library/Frameworks/Kernel.framework/Headers/netinet/ip_var.h. That's because you're building a kernel extension which cannot use the headers in /usr/include. My 10.2 system, installed with only the contents of the released CDs on a new volume, seems to have it. I remember there was a problem with them being missing in earlier seeds of the developer tools, but I thought for sure that was addressed. I'm sure someone more knowledgeable than I will chime in with exactly which versions of things have the fix. Disturbing. I did a completely clean install of the retail 10.2, its dev tools and the August patch to the dev tools and searching all local disks yields no files with "ip_var" in the name. If anyone learns what's up with this please let us know. Thanks. Cheers, Mike _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: What should we do when headers are missing in 10.2?
by Andrew Gallatin 11 Sep '02

11 Sep '02

Jim Magee writes: > > That's just the way BSD headers do things. To change it would break > BSD [in-kernel] compatibility. But having BSD [in-kernel] > compatibility also implies something else. As you will see, the > definition of struct ipovly [the type for the ti_i field] is marked > __APPLE_API_PRIVATE. That's you indicator that you have to be prepared > to have your software track the OS on a release-for-release basis. I see your point, but ipovly is not a good example. Unless you're planning to reimpliment IP so that you can't talk to anybody else, I doubt that ipovly will be changing much ;) > If that's not your cup of tea (and it's certainly not the typical > business model of any long-time Mac developer), we are in the process > of creating a set of more sustainable APIs for such kernel extensions. > But as good as we are at guessing your needs, we aren't psychic. You > [all] need to let us know what services you need sustainable interfaces > for. Well do our best to include those [or comparable equivalents]. I could make all uses APPLE_API_PRIVATE headers go away in my driver (OS-Bypass cluster interconnect -- http://www.myri.com/scs/macosx.html) if somebody would just fix the VM system so that it wouldn't block a user process indefinatly if said process attempts to free memory locked via IOMemoryDescriptor's prepare function. But I couldn't seem to get anybody interested in changing this behaviour in the past... Regards, Drew _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: What should we do when headers are missing in 10.2?
by Jim Magee 11 Sep '02

11 Sep '02

On Wednesday, September 11, 2002, at 11:09 AM, Stiphane Sudre wrote: In fact, my problem is not with the need to #include any file before or after. The real issue is that the files are missing from /usr/include in a complete Mac OS X 10.2 install (by complete, I mean everything checked, last dev Tools, etc...). Currently, I'm using the header files from a 10.1 backup but it's far from being a good solution. So I'm asking for a better/recommended/official solution. It should be in /usr/include/netinet/ip_var.h. It should also be in /System/Library/Frameworks/Kernel.framework/Headers/netinet/ip_var.h. That's because you're building a kernel extension which cannot use the headers in /usr/include. My 10.2 system, installed with only the contents of the released CDs on a new volume, seems to have it. I remember there was a problem with them being missing in earlier seeds of the developer tools, but I thought for sure that was addressed. I'm sure someone more knowledgeable than I will chime in with exactly which versions of things have the fix. If that's not your cup of tea (and it's certainly not the typical business model of any long-time Mac developer), we are in the process of creating a set of more sustainable APIs for such kernel extensions. But as good as we are at guessing your needs, we aren't psychic. You [all] need to let us know what services you need sustainable interfaces for. Well do our best to include those [or comparable equivalents]. This makes me think I'm currently wondering why memcmp is said not to be defined when strings.h is included in a C source of a Kernel extensions. This is where that distinction between /usr/include and Kernel.framework really kicks in. The kernel runtime environment is [purposely] MUCH smaller than what you have available at user space. As such, /System/Library/Frameworks/Kernel.framework/Headers/string.h is NOT the identical file you have available at user space through /usr/include/string.h. It is a greatly limited subset for what is available in-kernel. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Darwin & SMT?
by Bernie Zenis 11 Sep '02

11 Sep '02

On Wednesday, September 11, 2002, at 10:05 AM, Jim Magee wrote: On Tuesday, September 10, 2002, at 02:05 PM, Bernie Zenis wrote: I see that real SMT processors are finally being built which makes me happy. I just didn't know if Darwin had its foot in any of it. I'm thinking that for an SMT processor to become a real possibility for replacing (S)MP designs that the "thread" in SMT would have to refer to a full fledged process (eg, own address space, etc.). Would it be easier to port Darwin to a SMT processor in which its "threads" were light weight or full fledged? Or both equally well? The best ones are the ones that look like full-fledged second processors (full register set, including address space registers, etc...) even though they are just two sets of registers for a shared set of execution units. That's because, in the common case, the second thread that could benefit from getting started early (or that would benefit the current thread by not preempting it) runs in a different address space. In Mac OS X/Darwin, a common example is the IOKit workloop threads that run much of the code previously run in interrupt context. Being able to schedule these so that they interleave with the current thread, rather than preempting it (or waiting for it), is a big win (latency-wise). Normally, the stumbling point in doing this on true SMP machines is the overhead of kicking the other processor to make sure the top N (two, in our case) threads (priority-wise) are running at any given time. SMT processors can potentially reduce this overhead significantly (no need for bus transactions). The higher CPU clock rates go, the more "starved for real work" the execution units become. We have just crossed the threshold where things like wildly speculative execution are not filling those voids and tend to be counter-productive (cache-thrash-wise). If you are going to thrash the cache a bit, it would be better to know that it is being done on behalf of code that really *needs* to run. And if you happen to have two threads that are running in the same cache working set, it should really fly. That's because it is still quite hard to arrange for all the execution units to be busy with a single cache-resident thread of execution. These are the reasons why you are seeing a resurgence in the SMT notion. I, personally, can't wait to play with them to see if the implementation lives up to the theory. As far as I am aware, there has been no work done on the possibility of removing speculative instruction execution, out of order instruction execution, nor branch prediction in SMT processors. Some/All of those capabilities are complex and expensive. Removing them would create an easier to design & test processor with a small die. However, you'd need a SMT processor with enough thread contexts to keep the execution units busy. And adding more thread contexts would increase the design complexity and cost. It is a trade off and, not being a processor designer myself, I don't know which would be better. I also don't know if a typical Mac OS X/Darwin system has enough "busy" threads to keep a high context count SMT processor filled with work. Then there are the times when a single thread becomes dominant (and my CPU Monitor graph fills). You might want speculative instruction execution, out of order instruction execution, and branch prediction back in that case. However, are there really that many tasks that have to be run as a single thread? Can a majority (to darn near all) be split up? But will any of those tasks, split into multiple threads, still have the same characteristics as a single thread? On the plus side, Apple has seen the value of multiple processors for a while now. Developers are starting to see that breaking up tasks can have a real effect. Last I heard, the SMT simulator is still Alpha only. Anyone been working on a PowerPC version? Thanks, Bernie _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: What should we do when headers are missing in 10.2?
by Stéphane Sudre 11 Sep '02

11 Sep '02

On Wednesday, September 11, 2002, at 04:59 PM, Jim Magee wrote: On Wednesday, September 11, 2002, at 06:42 AM, Stiphane Sudre wrote: In 10.2, there are some headers that vanished. This leads to the impossibility to compile some code which was compiling fine on 10.1. A (maybe bad) example is this kind of compilation error: /System/Library/Frameworks/Kernel.framework/Headers/netinet/ tcpip.h:66: field `ti_i' has incomplete type What is the recommended solution in this situation? You have to include <netinet/ip_var.h> before including <netinet/tcpip.h>. You may also have to include something else before including <netinet/ip_var.h>. I didn't look to see if that was end of the ordering dependencies. In fact, my problem is not with the need to #include any file before or after. The real issue is that the files are missing from /usr/include in a complete Mac OS X 10.2 install (by complete, I mean everything checked, last dev Tools, etc...). Currently, I'm using the header files from a 10.1 backup but it's far from being a good solution. So I'm asking for a better/recommended/official solution. That's just the way BSD headers do things. To change it would break BSD [in-kernel] compatibility. But having BSD [in-kernel] compatibility also implies something else. As you will see, the definition of struct ipovly [the type for the ti_i field] is marked __APPLE_API_PRIVATE. That's you indicator that you have to be prepared to have your software track the OS on a release-for-release basis. I swear I'm not using it directly, it's just the header that needs it. I would not dare using APPLE_API_PRIVATE stuff. If that's not your cup of tea (and it's certainly not the typical business model of any long-time Mac developer), we are in the process of creating a set of more sustainable APIs for such kernel extensions. But as good as we are at guessing your needs, we aren't psychic. You [all] need to let us know what services you need sustainable interfaces for. Well do our best to include those [or comparable equivalents]. This makes me think I'm currently wondering why memcmp is said not to be defined when strings.h is included in a C source of a Kernel extensions. -- Stephane _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: What should we do when headers are missing in 10.2?
by Jim Magee 11 Sep '02

11 Sep '02

On Wednesday, September 11, 2002, at 06:42 AM, Stiphane Sudre wrote: In 10.2, there are some headers that vanished. This leads to the impossibility to compile some code which was compiling fine on 10.1. A (maybe bad) example is this kind of compilation error: /System/Library/Frameworks/Kernel.framework/Headers/netinet/ tcpip.h:66: field `ti_i' has incomplete type What is the recommended solution in this situation? You have to include <netinet/ip_var.h> before including <netinet/tcpip.h>. You may also have to include something else before including <netinet/ip_var.h>. I didn't look to see if that was end of the ordering dependencies. That's just the way BSD headers do things. To change it would break BSD [in-kernel] compatibility. But having BSD [in-kernel] compatibility also implies something else. As you will see, the definition of struct ipovly [the type for the ti_i field] is marked __APPLE_API_PRIVATE. That's you indicator that you have to be prepared to have your software track the OS on a release-for-release basis. If that's not your cup of tea (and it's certainly not the typical business model of any long-time Mac developer), we are in the process of creating a set of more sustainable APIs for such kernel extensions. But as good as we are at guessing your needs, we aren't psychic. You [all] need to let us know what services you need sustainable interfaces for. Well do our best to include those [or comparable equivalents]. Otherwise, every release you continue to use __APPLE_API_PRIVATE services, you will have to be prepared to do this form of discovery, patch, re-release. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Darwin & SMT?
by Jim Magee 11 Sep '02

11 Sep '02

On Tuesday, September 10, 2002, at 02:05 PM, Bernie Zenis wrote: I see that real SMT processors are finally being built which makes me happy. I just didn't know if Darwin had its foot in any of it. I'm thinking that for an SMT processor to become a real possibility for replacing (S)MP designs that the "thread" in SMT would have to refer to a full fledged process (eg, own address space, etc.). Would it be easier to port Darwin to a SMT processor in which its "threads" were light weight or full fledged? Or both equally well? The best ones are the ones that look like full-fledged second processors (full register set, including address space registers, etc...) even though they are just two sets of registers for a shared set of execution units. That's because, in the common case, the second thread that could benefit from getting started early (or that would benefit the current thread by not preempting it) runs in a different address space. In Mac OS X/Darwin, a common example is the IOKit workloop threads that run much of the code previously run in interrupt context. Being able to schedule these so that they interleave with the current thread, rather than preempting it (or waiting for it), is a big win (latency-wise). Normally, the stumbling point in doing this on true SMP machines is the overhead of kicking the other processor to make sure the top N (two, in our case) threads (priority-wise) are running at any given time. SMT processors can potentially reduce this overhead significantly (no need for bus transactions). The higher CPU clock rates go, the more "starved for real work" the execution units become. We have just crossed the threshold where things like wildly speculative execution are not filling those voids and tend to be counter-productive (cache-thrash-wise). If you are going to thrash the cache a bit, it would be better to know that it is being done on behalf of code that really *needs* to run. And if you happen to have two threads that are running in the same cache working set, it should really fly. That's because it is still quite hard to arrange for all the execution units to be busy with a single cache-resident thread of execution. These are the reasons why you are seeing a resurgence in the SMT notion. I, personally, can't wait to play with them to see if the implementation lives up to the theory. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

What should we do when headers are missing in 10.2?
by Stéphane Sudre 11 Sep '02

11 Sep '02

In 10.2, there are some headers that vanished. This leads to the impossibility to compile some code which was compiling fine on 10.1. A (maybe bad) example is this kind of compilation error: /System/Library/Frameworks/Kernel.framework/Headers/netinet/tcpip.h:66: field `ti_i' has incomplete type What is the recommended solution in this situation? Is there a darwin patch we can apply to get back these headers in a clean way? _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

cacheInit() patch and submitting
by Terry Greeniaus 10 Sep '02

10 Sep '02

Hey, I have a patch to the cacheInit() routine in machine_routines_asm.s. I am unsure of who to submit it to. Unfortunately, this patch performs operations that are also under NDA to us from Motorola, thus I can't discuss it directly on this list either. If someone could email me, or post to the list who I might contact to get this patch into the kernel, it would be much appreciated. Thanks! TG _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Darwin & SMT?
by Bernie Zenis 10 Sep '02

10 Sep '02

On Tuesday, September 10, 2002, at 01:41 PM, Jim Magee wrote: On Tuesday, September 10, 2002, at 01:16 PM, Bernie Zenis wrote: Is anyone here familiar with Simultaneous Multithreading (http://www.cs.washington.edu/research/smt/)? Does anyone know if Darwin would lend itself well to SMT? For example, if a SMT PowerPC chip came out, would it be easy to port Darwin to it? I realize that might depend on what kind of SMT architecture was implemented. I have always felt that Darwin works best on an SMP machine. There are enough system threads, and multi-threaded applications to want two CPUs in many "peak" CPU utilization situations (my Mac OS X system typically has ~200 threads). I think the real world anecdotal evidence bears this out as well (most people with dual-ies are quite happy with the responsiveness of their machines, even when compared to single processor machines of quite higher clock rates). All of this is one of the reasons Apple ships all their PowerMacs as dual processor machines right now. The CPI numbers under most of those situations indicate that an SMT (others call it Hyper-threading) processor might do reasonably well compared to an SMP machine. But as you say, it all depends on the particular SMT design and other architectural factors on any given processor. I've always know that modern computers are doing many things at once; but, I didn't really realize (because I lived in Mac OS 8/9 land :-) how many things that was until I got X installed and looked at outputs of programs like top. Obviously, most of those threads are not demanding a lot of the machine during much of the time (otherwise my CPU Monitor load would be much higher). I had thought many years ago that computers would benefit from a processor design like SMT, I just didn't know that it had a name and that people we're working on it until a few years ago. (You would think that a college professor teaching an MP class would know.) I see that real SMT processors are finally being built which makes me happy. I just didn't know if Darwin had its foot in any of it. I'm thinking that for an SMT processor to become a real possibility for replacing (S)MP designs that the "thread" in SMT would have to refer to a full fledged process (eg, own address space, etc.). Would it be easier to port Darwin to a SMT processor in which its "threads" were light weight or full fledged? Or both equally well? Thanks, Bernie PS I'm really happy Apple decided to finally follow the dual processor route. (It is about time! :-) _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0