August 2005 - Darwin-kernel

Got boot device
by Herbert 16 Aug '05

16 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Does anyone know how/where the "Got boot device = etc" line is fetched? I need to know which is the boot device (in my driver), and so far as I can see there is a way, because Apple's doing it, but I can't track it down. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: mbufs and cluster
by Mike Smith 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 15, 2005, at 1:30 PM, Michael Tuexen wrote: Are you really saying, that one should reboot the system after a program was run which used a lot of mbufs and freed them? No. = Mike Best regards Michael On Aug 15, 2005, at 22:12 Uhr, Josh Graessley wrote: Hello Michael, -josh On Aug 15, 2005, at 12:50 PM, Michael Tuexen wrote: Hi Josh, once again thank you very much! Best regards Michael On Aug 15, 2005, at 17:24 Uhr, Josh Graessley wrote: -josh On Aug 15, 2005, at 7:46 AM, Michael Tuexen wrote: Dear all, I have a simple question: But netstat -m gives for example: [PowerMac:/] tuexen% netstat -m 330 mbufs in use: 325 mbufs allocated to data 1 mbufs allocated to packet headers 3 mbufs allocated to socket names and addresses 1 mbufs allocated to Appletalk data blocks 376/848 mbuf clusters in use 1778 Kbytes allocated to network (46% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Thank you very much for your help. Best regards Michael This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/drivers%40mu.org This email sent to drivers(a)mu.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… the principal problem I see is, that an Mac OS X system uses more and more mbuf clusters over the time and there is no chance to get them back. From my understanding the state of the system before and after running a user program should be the same (in terms of used kernel buffers and so on). Your understanding is wrong. Not only in this regard; the state of the VM/buffer cache and many other parts of the system is mutated by such an event. Mbuf clusters may be converted into mbufs; this is in fact a necessary part of the operation of the allocator. It doesn't matter whether free memory is "mbufs" or "mbuf clusters", since the system needed that many mbufs at one point, it's likely to again. Converting them back would just be a waste of time. The problem came up when testing the SCTP KAME implementation which does not have this problem on FreeBSD 4.* or 5.*. I believe that once a cluster is used for mbufs, those mbufs are never again collecting in to a cluster. Again, it has been some time since I've mucked with that stuff, so I might be wrong. It sounds like you're using too many mbufs. You might want to look in to ways to reduce the number of mbufs in use. Using your information I was able to find (after a one week search) a 'mbuf cluster leak' in the SCTP KAME implementation when the code runs on Mac OS X. The point is, that it is no leak. What happens is that when a program runs it needs some mbufs. So some clusters are converted to mbufs and the number of free mbuf clusters shrinks. After the program has finished the number of free mbuf clusters is reduced but the number of free mbufs is incremented. During tests this resulted in a scenario where you ran out of mbuf clusters. So the question is: How/when are free mbufs put together to free mbuf clusters again? If I remember correctly, mbufs themselves are stored in clusters. A cluster can be used as a cluster on it's own or it can be used to store mbufs. An mbuf is a fraction of a cluster. If the system runs out of mbufs, it may convert a cluster in to something like 8 mbufs. Normally a used cluster element is one where there exist an mbuf which has a pointer to the cluster element. This happens at least when you get it with m_mclget or the corresponding macro. This would than mean that the number of uses mbufs is not larger than the number of used mbuf clusters. Why can it happen that the number of mbuf clusters is larger than the number of mbufs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Is ctl_enqueuembuf destructive?
by Vincent Lubet 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Aug 15, 2005, at 2:17 PM, Allan Hoeltje wrote: This is correct, ctl_enqueuembuf consumes the mbuf If so then it seems that ctl_enqueuembuf is no more efficient than ctl_enqueuedata. I don't believe one is much more efficient than the other. Vincent _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I want to send IP packets from my kext to a client app. Up to now I've been making a copy of the mbuf data with mbuf_copydata and then sending the data to the client with ctl_enqueuedata. Today I looked at the source for ctl_enqueuedata and saw that it takes my flat data and copies it into a newly allocated mbuf which is then passed to sbappend and sbcompress. So, it seemed to me that I could be more efficient if I just used ctl_enqueuembuf to send the original mbuf to the client. However, when I do that I get the Grey Screen of Death. I noticed that ctl_enqueuedata allocates a new mbuf but does not free it. Should I assume that sbappend causes the mbuf to be freed? If so, then calling ctl_enqueuembuf would also cause the original mbuf to be freed, right? It would certainly explain the GSofD. The source for sbcompress seems to indicate that it does free the mbuf space but I am not sure if I am reading m_free correctly. Do I need to do an mbuf_dup on my original mbuf and ctl_enqueuembuf the duplicate? Yes you need to duplicate the mbuf if you plan to use it after calling ctl_enqueuembuf. This email sent to site_archiver(a)lists.apple.com

1 0

Re: change boot partition from command line
by David Höppner 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Is ctl_enqueuembuf destructive?
by Allan Hoeltje 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Microsoft-Entourage/10.1.4.030702.0 I want to send IP packets from my kext to a client app. Up to now I've been making a copy of the mbuf data with mbuf_copydata and then sending the data to the client with ctl_enqueuedata. Today I looked at the source for ctl_enqueuedata and saw that it takes my flat data and copies it into a newly allocated mbuf which is then passed to sbappend and sbcompress. So, it seemed to me that I could be more efficient if I just used ctl_enqueuembuf to send the original mbuf to the client. However, when I do that I get the Grey Screen of Death. I noticed that ctl_enqueuedata allocates a new mbuf but does not free it. Should I assume that sbappend causes the mbuf to be freed? If so, then calling ctl_enqueuembuf would also cause the original mbuf to be freed, right? It would certainly explain the GSofD. The source for sbcompress seems to indicate that it does free the mbuf space but I am not sure if I am reading m_free correctly. Do I need to do an mbuf_dup on my original mbuf and ctl_enqueuembuf the duplicate? If so then it seems that ctl_enqueuembuf is no more efficient than ctl_enqueuedata. -Allan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: mbufs and cluster
by Michael Tuexen 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Josh, Are you really saying, that one should reboot the system after a program was run which used a lot of mbufs and freed them? Best regards Michael On Aug 15, 2005, at 22:12 Uhr, Josh Graessley wrote: Hello Michael, -josh On Aug 15, 2005, at 12:50 PM, Michael Tuexen wrote: Hi Josh, once again thank you very much! Best regards Michael On Aug 15, 2005, at 17:24 Uhr, Josh Graessley wrote: -josh On Aug 15, 2005, at 7:46 AM, Michael Tuexen wrote: Dear all, I have a simple question: But netstat -m gives for example: [PowerMac:/] tuexen% netstat -m 330 mbufs in use: 325 mbufs allocated to data 1 mbufs allocated to packet headers 3 mbufs allocated to socket names and addresses 1 mbufs allocated to Appletalk data blocks 376/848 mbuf clusters in use 1778 Kbytes allocated to network (46% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Thank you very much for your help. Best regards Michael This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… the principal problem I see is, that an Mac OS X system uses more and more mbuf clusters over the time and there is no chance to get them back. From my understanding the state of the system before and after running a user program should be the same (in terms of used kernel buffers and so on). The problem came up when testing the SCTP KAME implementation which does not have this problem on FreeBSD 4.* or 5.*. I believe that once a cluster is used for mbufs, those mbufs are never again collecting in to a cluster. Again, it has been some time since I've mucked with that stuff, so I might be wrong. It sounds like you're using too many mbufs. You might want to look in to ways to reduce the number of mbufs in use. Using your information I was able to find (after a one week search) a 'mbuf cluster leak' in the SCTP KAME implementation when the code runs on Mac OS X. The point is, that it is no leak. What happens is that when a program runs it needs some mbufs. So some clusters are converted to mbufs and the number of free mbuf clusters shrinks. After the program has finished the number of free mbuf clusters is reduced but the number of free mbufs is incremented. During tests this resulted in a scenario where you ran out of mbuf clusters. So the question is: How/when are free mbufs put together to free mbuf clusters again? If I remember correctly, mbufs themselves are stored in clusters. A cluster can be used as a cluster on it's own or it can be used to store mbufs. An mbuf is a fraction of a cluster. If the system runs out of mbufs, it may convert a cluster in to something like 8 mbufs. Normally a used cluster element is one where there exist an mbuf which has a pointer to the cluster element. This happens at least when you get it with m_mclget or the corresponding macro. This would than mean that the number of uses mbufs is not larger than the number of used mbuf clusters. Why can it happen that the number of mbuf clusters is larger than the number of mbufs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/ jgraessley%40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: mbufs and cluster
by Josh Graessley 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hello Michael, -josh On Aug 15, 2005, at 12:50 PM, Michael Tuexen wrote: Hi Josh, once again thank you very much! Best regards Michael On Aug 15, 2005, at 17:24 Uhr, Josh Graessley wrote: -josh On Aug 15, 2005, at 7:46 AM, Michael Tuexen wrote: Dear all, I have a simple question: But netstat -m gives for example: [PowerMac:/] tuexen% netstat -m 330 mbufs in use: 325 mbufs allocated to data 1 mbufs allocated to packet headers 3 mbufs allocated to socket names and addresses 1 mbufs allocated to Appletalk data blocks 376/848 mbuf clusters in use 1778 Kbytes allocated to network (46% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Thank you very much for your help. Best regards Michael This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com I believe that once a cluster is used for mbufs, those mbufs are never again collecting in to a cluster. Again, it has been some time since I've mucked with that stuff, so I might be wrong. It sounds like you're using too many mbufs. You might want to look in to ways to reduce the number of mbufs in use. Using your information I was able to find (after a one week search) a 'mbuf cluster leak' in the SCTP KAME implementation when the code runs on Mac OS X. The point is, that it is no leak. What happens is that when a program runs it needs some mbufs. So some clusters are converted to mbufs and the number of free mbuf clusters shrinks. After the program has finished the number of free mbuf clusters is reduced but the number of free mbufs is incremented. During tests this resulted in a scenario where you ran out of mbuf clusters. So the question is: How/when are free mbufs put together to free mbuf clusters again? If I remember correctly, mbufs themselves are stored in clusters. A cluster can be used as a cluster on it's own or it can be used to store mbufs. An mbuf is a fraction of a cluster. If the system runs out of mbufs, it may convert a cluster in to something like 8 mbufs. Normally a used cluster element is one where there exist an mbuf which has a pointer to the cluster element. This happens at least when you get it with m_mclget or the corresponding macro. This would than mean that the number of uses mbufs is not larger than the number of used mbuf clusters. Why can it happen that the number of mbuf clusters is larger than the number of mbufs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/jgraessley% 40apple.com smime.p7s

1 0

Re: mbufs and cluster
by Michael Tuexen 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hi Josh, once again thank you very much! Best regards Michael On Aug 15, 2005, at 17:24 Uhr, Josh Graessley wrote: -josh On Aug 15, 2005, at 7:46 AM, Michael Tuexen wrote: Dear all, I have a simple question: But netstat -m gives for example: [PowerMac:/] tuexen% netstat -m 330 mbufs in use: 325 mbufs allocated to data 1 mbufs allocated to packet headers 3 mbufs allocated to socket names and addresses 1 mbufs allocated to Appletalk data blocks 376/848 mbuf clusters in use 1778 Kbytes allocated to network (46% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Thank you very much for your help. Best regards Michael This email sent to jgraessley(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Using your information I was able to find (after a one week search) a 'mbuf cluster leak' in the SCTP KAME implementation when the code runs on Mac OS X. The point is, that it is no leak. What happens is that when a program runs it needs some mbufs. So some clusters are converted to mbufs and the number of free mbuf clusters shrinks. After the program has finished the number of free mbuf clusters is reduced but the number of free mbufs is incremented. During tests this resulted in a scenario where you ran out of mbuf clusters. So the question is: How/when are free mbufs put together to free mbuf clusters again? If I remember correctly, mbufs themselves are stored in clusters. A cluster can be used as a cluster on it's own or it can be used to store mbufs. An mbuf is a fraction of a cluster. If the system runs out of mbufs, it may convert a cluster in to something like 8 mbufs. Normally a used cluster element is one where there exist an mbuf which has a pointer to the cluster element. This happens at least when you get it with m_mclget or the corresponding macro. This would than mean that the number of uses mbufs is not larger than the number of used mbuf clusters. Why can it happen that the number of mbuf clusters is larger than the number of mbufs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/ jgraessley%40apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: fsync()
by Jonas Maebe 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On 15 Aug 2005, at 20:55, Steve Dekorte wrote: See http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… I've been told that the docs for fync() are incorrect and that OSX does not actually wait for the data to be written to the disk before returning. Is this true and if so, how can I ensure the data is written before continuing? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Carbon File Manger VFS caching
by Brian Bergstrand 15 Aug '05

15 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks for the detailed explanation Quinn. I'll look into it more. On Aug 15, 2005, at 4:28 AM, Quinn wrote: At 18:37 -0500 12/8/05, Brian Bergstrand wrote: <http://developer.apple.com/technotes/tn2004/tn2124.html> Brian Bergstrand <http://www.bergstrand.org/brian/> PGP Key ID: 0xB6C7B6A2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFDAOZEedHYW7bHtqIRAuz7AKC3HboLbvtF9rSRvfbqFc41KRBfswCgpDPp DHFxpKnfnV+ohZXneXmf2vE= =I94z -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… We've had problems with the Carbon File Mgr caching dir entries for too long in Panther and Jaguar and the problem persists in Tiger. Both the Finder and the File Manager cache directory contents. You need to distinguish between the two. The way to do that is to write a simple tool that iterates a directory using File Manager calls. If you're seeing stale data, the File Manager cache is the problem. OTOH, if the data is up-to-date, you need to look to the Finder. I believe that the enumeration cache is per-process. If so, your tool will to have to keep running between successive enumerations. You can dump the enumeration cache for a volume using PrintEnumerationCache, as described in Technote 2124 "Debugging Magic". As far as the File Manager is concerned, looking at the source, it seems that it caches directory contents based the modification time of the directory. To improve performance, it only rechecks the directory's modification time when a client starts an enumeration. You should make sure you're bumping the mod date of the directory when you make changes to the directory. The above applies only to Posix volumes; IIRC that's what you're implementing. If the Finder is the problem, I believe that, in 10.4 and later, the Finder will use kqueues to pick up changes to its windows. To see an example of (roughly) what the Finder does, take a look at the FileNotification sample. <http://developer.apple.com/samplecode/FileNotification/ FileNotification.html> In theory, if your file system notifies the system of changes, this will automatically bubble up through kqueues to the Finder. AFAIK you can do this will public KPIs. For an example of how to do this, search the HFS [Plus] source for "HFS_KNOTE". In practice, this isn't going to work because, looking at the Finder source, it seems that it contains a special case list of file systems that support kqueues. *grrrr* <rdar://problem/4216129> This email sent to site_archiver(a)lists.apple.com

1 0