July 2003 - Darwin-kernel - lists.apple.com

Re: I/O in my Filesystem causing application crashes.
by Ron Aldrich 29 Jul '03

29 Jul '03

On Monday, July 28, 2003, at 03:01 PM, Quinn wrote: At 16:42 -0700 21/7/03, raldrich(a)mac.com wrote: I would, however, like to find some documentation for "cluster_pagein" and "cluster_pageout" - an overview of the unified buffer cache would be nice as well... If your file system supports caching, the cluster_xxx calls are the way to go. All you have to do is direct your VOP_READ/WRITE/PAGEIN/PAGEOUT calls to the corresponding cluster_xxx calls, implement VOP_CMAP and VOP_STRATEGY and all the hard work is done. This is how both HFS Plus and AppleShare work (at least, for AppleShare, in the non-cached case) and it makes the read/write path very easy. I've gotten cluster I/O working, and it works quite well - really simplified the read/write path, although I suspect that it's less optimal than my previous code (it did make debugging memory mapped I/O a lot easier). Thanks for the pointers, Ron Aldrich Software Architects, Inc. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: [OT] Dynamic Stack Allocation
by Andrew Pinski 29 Jul '03

29 Jul '03

On Monday, Jul 28, 2003, at 23:58 US/Eastern, Sam Vaughan wrote: This is a little off-topic, but perhaps of interest to the curious... I was amazed to find this (summarised) code in the Darwin DiskArbitration source (FSParticular.c line 352): void foo() { int j = bar(); CFDictionaryRef dicts[j]; CFStringRef keys[j]; } It is part of C99 and also a GCC extension which is turned on by default. To my surprise, this compiles with no warnings and works. I would have expected a compiler error since it doesn't know at compile time how large to make the two arrays. I then wrote a test app called foo.c: #include <stdio.h> int main(int argc, char** argv) { if (argc < 2) return 0; int items = atoi(argv[1]); int fred[items]; printf("allocated fred[%d], items in fred: %d\n", items, sizeof(fred) / sizeof(fred[0])); return 0; } And this is what I got: % gcc -o foo foo.c % ./foo 1 allocated fred[1], items in fred: 1 % ./foo 1000 allocated fred[1000], items in fred: 1000 % ./foo 1000000 Segmentation fault The problem is that the default max stack size is only 512K. You can increase it by using limit or ulimit depending on you shell. Looking at the disassembly it appears that the stack is dynamically grown to accommodate the requested memory. Gauging by the dismayed response from others I've shown this to, I'm guessing it's not very portable! Interesting nonetheless. It is as long as you have a C99 compiler or just GCC and have enough stack size. Thanks, Andrew Pinski _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

[OT] Dynamic Stack Allocation
by Sam Vaughan 29 Jul '03

29 Jul '03

This is a little off-topic, but perhaps of interest to the curious... I was amazed to find this (summarised) code in the Darwin DiskArbitration source (FSParticular.c line 352): void foo() { int j = bar(); CFDictionaryRef dicts[j]; CFStringRef keys[j]; } To my surprise, this compiles with no warnings and works. I would have expected a compiler error since it doesn't know at compile time how large to make the two arrays. I then wrote a test app called foo.c: #include <stdio.h> int main(int argc, char** argv) { if (argc < 2) return 0; int items = atoi(argv[1]); int fred[items]; printf("allocated fred[%d], items in fred: %d\n", items, sizeof(fred) / sizeof(fred[0])); return 0; } And this is what I got: % gcc -o foo foo.c % ./foo 1 allocated fred[1], items in fred: 1 % ./foo 1000 allocated fred[1000], items in fred: 1000 % ./foo 1000000 Segmentation fault Looking at the disassembly it appears that the stack is dynamically grown to accommodate the requested memory. Gauging by the dismayed response from others I've shown this to, I'm guessing it's not very portable! Interesting nonetheless. Sam _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: I/O in my Filesystem causing application crashes.
by Quinn 28 Jul '03

28 Jul '03

At 16:42 -0700 21/7/03, raldrich(a)mac.com wrote: I would, however, like to find some documentation for "cluster_pagein" and "cluster_pageout" - an overview of the unified buffer cache would be nice as well... If your file system supports caching, the cluster_xxx calls are the way to go. All you have to do is direct your VOP_READ/WRITE/PAGEIN/PAGEOUT calls to the corresponding cluster_xxx calls, implement VOP_CMAP and VOP_STRATEGY and all the hard work is done. This is how both HFS Plus and AppleShare work (at least, for AppleShare, in the non-cached case) and it makes the read/write path very easy. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Waking a socket call from an alternate thread.
by Jim Magee 28 Jul '03

28 Jul '03

On Monday, July 28, 2003, at 3:39 AM, Wally Crooze wrote: Our filesystem requires a mechanism to wake up these calls and give an indication that the thread was interrupted/awoken. This indicator could be through an error or alternatively handle by our own code. But the mechanism to wake up the calls is MacOSX specific and even if we play with internal structures to mimic this... the outcome will be an OS version specific hack that can only be expected to handle the *obvious* conditions. Could somebody give me the supported mechanisms to wakeup/interrupt socket calls which will support all (hopefully) the cases that they could sleep. This is very important due to our filesystem being in a clustered network that must handle failover and redistribution. I assume you are trying to do this all from within the kernel? In that case, there is no SUPPORTED mechanisms to do this. At user-space, you could use pthread_kill() to send signals to your "worker" thread. You would have to check for EINTR and do the appropriate thing. But there is no pthread library, and therefore no pthread_kill() API. within the kernel. Within the kernel, the mechanism underlying all of these interruption mechanisms are represented by the Mach SPIs thread_abort() and thread_abort_safely(). They both interrupt a thread out of interruptible kernel operations. The former may leave the current operation in a non-restartable state. It is usually restricted to "kill -9" type operations where you don't care about restarting the current operation.The latter will only interrupt threads that can safely restart their current operations after handling the interruption. However, there is a "trick" to using these thread_abort() mechanisms within the kernel. Usually, the code that implements the return-to-user-space handling resets these conditions. For pure kernel threads, that path is not executed (and there is no published API to simulate it either). You could, however, work with a "borrowed/captured" user thread and return it user-space when EINTR is detected (to clear the interrupted condition). But as I said, use of these thread_abort() SPIs should be considered "at your own risk." No guarantees are made about their correctness (or even existence) from release to release. But enough code uses them (including Java runtime, Carbon thread manager, and BSD signals) at their core, that they are not likely to become invalid anytime soon. --Jim _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: Duplicate entries in a file system...
by Quinn 28 Jul '03

28 Jul '03

At 13:33 -0400 9/7/03, Marek Kozubal wrote: So does anyone know what would cause Carbon/Finder to create this duplicate entry in the Finder window? Just for closure, I thought I'd post the resolution to this. Marek opened a DTS incident and we managed to track down the root cause of the problem. Marek is working on a file system that pretends to be HFS Plus in all ways. It turns out that the Finder contains a special case for HFS Plus. If the files are coming from an HFS Plus file system, the Finder assumes that it knows the order in which items will be returned by FSGetCatalogInfoBulk (which is layered on top of the BSD getdirentriesattr call). If the entries come back in a different order, the Finder gets *very* confused, and displays the symptoms Marek described. The obvious resolution is that, if you're going to pretend to be HFS Plus, you have to do so in all ways, including the sort order of directory entries. I've also file a bug report to investigate <rdar://problem/3338538> Apple's response to this. S+E -- Quinn "The Eskimo!" <http://www.apple.com/developer/> Apple Developer Technical Support * Networking, Communications, Hardware _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Waking a socket call from an alternate thread.
by Wally Crooze 28 Jul '03

28 Jul '03

Hello, During the course of execution of our filesystem there will be times when socket calls will be made that will sleep. The simple ones are soaccept on connections and soreceive on buffer, but there ares also times when the socket call will sleep due to memory acquisition and other unexpected reasons. Our filesystem requires a mechanism to wake up these calls and give an indication that the thread was interrupted/awoken. This indicator could be through an error or alternatively handle by our own code. But the mechanism to wake up the calls is MacOSX specific and even if we play with internal structures to mimic this... the outcome will be an OS version specific hack that can only be expected to handle the *obvious* conditions. Could somebody give me the supported mechanisms to wakeup/interrupt socket calls which will support all (hopefully) the cases that they could sleep. This is very important due to our filesystem being in a clustered network that must handle failover and redistribution. Thanks. -- Wally Crooze +-------------------------------------------------------+ | ,-_|\ Wally Crooze <wallycrooze(a)pobox.com> | | / \ | | \_,-._/ | | v | +-------------------------------------------------------+ _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: How to translate a mach_timespec ...
by Peter Dyballa 26 Jul '03

26 Jul '03

file:///Developer/Documentation/Darwin/General/KernelProgramming/Mach/_Time_Management.html states: The mach_timespec_t API is deprecated in Mac OS X. The newer and preferred API is based on timer objects that in turn use AbsoluteTime as the basic data type. AbsoluteTime is a machine-dependent type, typically based on the platform-native time base. Routines are provided to convert AbsoluteTime values to and from other data types, such as nanoseconds. Timer objects support asynchronous, drift-free notification, cancellation, and premature alarms. They are more efficient and permit higher resolution than clocks. file:///Developer/Documentation/Darwin/General/KernelProgramming/services/_Obtaining__Information.html states: The remaining functions return type mach_timespec_t, which is similar to the traditional BSD struct timespec, except that fractions of a second are measured in nanoseconds instead of microseconds: |struct mach_timespec {| | unsigned int tv_sec;| | clock_res_t tv_nsec;| |};| |typedef struct mach_timespec *mach_timespec_t;| -- Greetings Pete _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

How to translate a mach_timespec to "yyyy.mm.dd hh:mm:ss"
by Timothy Standing 24 Jul '03

24 Jul '03

I am trying to find a way to generate a string which contains the current time in the "yyyy.mm.dd hh:mm:ss" format. It seems like I should just be able to call clock_get_calendar_value() to get a mach_timespec_t and then call some function to translate this into a posix struct tm. Unfortunately, I can't find a function in the kernel which performs this second step. Does anyone know of a way to perform the conversion from a mach_timespec_t to a structure which contains the values for year, month, day etc? Does anyone know how IOLog() generates this information? Thanks in advance, Tim Standing SoftRAID, LLC _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: I/O in my Filesystem causing application crashes.
by raldrich＠mac.com 21 Jul '03

21 Jul '03

On Saturday, July 19, 2003, at 09:16 AM, Quinn wrote: At 17:35 -0700 16/7/03, raldrich(a)mac.com wrote: Any ideas? My guess is that this has something to do with how your VOP_PAGEIN entry point is implemented. I did some poking around in Preview with GDB and it seems that it memory maps the image file (using mmap). [NSBMPImageReader getByte] then reads the file by reading the mapped memory. Unfortunately I don't have a recommendation for how you might implement VOP_PAGEIN given your current I/O design. I'd have to study the problem in more detail than I have time for right now. I think you're right - at least in one case, I am seeing a call to pagein before the crash - pagein and pageout are currently unimplemented, and return an error (damm, I guess I'm going to have to figure out how to implement them). I found an overview discussion of the vnode pager in "The design and implementation of the 4.4 BSD operating system" - conceptually, it seems like it should work with my I/O model - I would, however, like to find some documentation for "cluster_pagein" and "cluster_pageout" - an overview of the unified buffer cache would be nice as well... One other oddity, is that if I simply double-click a .gif file on the disc, Preview is launched, but then does nothing - it doesn't open the file, doesn't memory map it, doesn't try to read it - nothing. If, however, I launch Preview, and select "open" to open the .gif file, then Preview crashes after my pagein call returns an error. Thanks, Ron Aldrich Software Architects, Inc. _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0