June 2004 - Darwin-kernel - lists.apple.com

File Monitoring under Mac OS X
by Krishna Monian 16 Jun '04

16 Jun '04

Hi All, I am currently trying to write a file logger, which logs every single file access (read, write, delete) on a specified volume. Could someone brief me on what would the best way be to go about doing this. Would it have to be in kernel mode or can it be done in user mode? Is it in anyway similar to the SampleFilterScheme example posted on Apple's website. http://developer.apple.com/samplecode/SampleFilterScheme/SampleFilterScheme… Thanks Krishna Monian __________________________________ Do you Yahoo!? New and Improved Yahoo! Mail - 100MB free storage! http://promotions.yahoo.com/new_mail _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Jay A. Kreibich 15 Jun '04

15 Jun '04

On Mon, Jun 14, 2004 at 09:16:20PM -0400, Marek Kozubal scratched on the wall: > On Mon, 14 Jun 2004, Jay A. Kreibich wrote: > > > To repeat a third time: UPDATE(8) WON'T BE RUNNING. I CANNOT CALL > > SYNC(8). > > Of course you can call sync(). updated mearly calls sync every 30 No, I can't. Or to be more specific, my script cannot. As I pointed out in the original post, I'm trying to do this from a script. When I send a SIGTERM to init it will kill *everything* including the script I use to send the SIGTERM. Therefore no matter what my script looks like, the last command executed will very likely be the "kill -s TERM 1". My script cannot call sync(8) once the system is in single user mode because my script has been killed by that point. However, there is no need to call sync(8) if close(2) calls fsync(2) or some similar function, which is why I asked my original question about the behavior of close(2). -j -- Jay A. Kreibich | Integration & Software Eng. jak(a)uiuc.edu | Campus IT & Edu. Svcs. <http://www.uiuc.edu/~jak> | University of Illinois at U/C _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Marek Kozubal 15 Jun '04

15 Jun '04

On Mon, 14 Jun 2004, Jay A. Kreibich wrote: To repeat a third time: UPDATE(8) WON'T BE RUNNING. I CANNOT CALL SYNC(8). Of course you can call sync(). updated mearly calls sync every 30 seconds. man 8 update SYNOPSIS update DESCRIPTION The update command helps protect the integrity of disk volumes by flush- ing volatile cached filesystem data to disk at thirty second intervals. Update uses the sync(2) function call to do the task. and man 2 sync SYNOPSIS #include <unistd.h> void sync(void); DESCRIPTION The sync() function forces a write of dirty (modified) buffers in the block buffer cache out to disk. The kernel keeps this information in core to reduce the number of disk I/O transfers required by the system. As information in the cache is lost after a system crash a sync() call is issued frequently by the user process update(8) (about every 30 seconds). The function fsync(2) may be used to synchronize individual file descrip- tor attributes. It seems like sync() should even work in single-user mode according to these man pages... --- Marek Kozubal marek(a)portents.com _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Jay A. Kreibich 14 Jun '04

14 Jun '04

On Mon, Jun 14, 2004 at 11:31:43AM -0700, Justin Walker scratched on the wall: > On Jun 14, 2004, at 11:01, Jay A. Kreibich wrote: > > [snip] > >>The point of being in single-user mode is that nothing happens > >>without your executing it directly. There should be no 'background' > >>jobs running, including 'update'. > > > > But the way you get into single user mode is that init issues a > > kill(2) with a PID of -1. According to the kill(2) man page this > > causes "the signal is sent to all processes excluding system processes > > and the process sending the signal." In truth, it kills EVERYTHING > > except init (the process sending the signal). That includes somewhat > > useful processes like mach_init (PID=2), without which the OS is, > > shall we say, somewhat limited. > > man init What about it? I've read both that and most of the source code to init. If you're looking for a philosophical definition of "single user mode" it isn't there. The init source code shows a kill with PID=-1. The kill man page implies this shouldn't really kill everything except init. It does. Booting to single user mode starts more processes than just init and a shell. > > The "single user mode" you get on boot, and the "single user mode" > > you get via a SIGTERM to init(1) are two different things. > > Not really. I think most people would call mach_init a pretty core part of how OS X is /supposed/ to work. The fact that it is running in one "single user mode" and not the other implies heavily that one of these "isn't right." I don't care which one you call wrong, but they aren't the same. > > Hence, the quesiton on if close(2) will take care of the dirty work. > > To repeat: 'close' makes sure that the pages are marked for update; What do you mean by "marked for update"? That they are queued for writing? That's been the whole question all along, which you've been saying isn't true. Or are you just saying that the dirty pages are marked as dirty (well, duh...) for something else to later come along and queue them for writing? > update makes sure (via 'sync') that the dirty pages are written to disk. Actually, all it does is queue them. It does not "make sure" of anything. If you call sync(2) and then cut the power as soon as the call returns, you can still loose data. This is the traditional reason for the old-school practice of running sync(8) three times. It provides (usually) enough time for the page system to process the queue. > You can also call 'sync' yourself. To repeat a third time: UPDATE(8) WON'T BE RUNNING. I CANNOT CALL SYNC(8). > If you make transition to single-user mode correctly, all should work. Define "should work." From the looks of the init(8) source code, I'll end up in single-user mode with a whole lot of dirty pages in memory. If update(8) were running (which it isn't) or if I call sync(1) (which I can't) then clearly everything would shortly end up on disk, and the minimal activity in single user mode would make additional dirty pages unlikely. But since those are not options, the question remains: does close(2) call fsync(2) (or something similar) on the file it is closing so that by the time all the processes are shut down and init decides it is in single user mode, there are no dirty pages left in memory? -j -- Jay A. Kreibich | Integration & Software Eng. jak(a)uiuc.edu | Campus IT & Edu. Svcs. <http://www.uiuc.edu/~jak> | University of Illinois at U/C _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Justin Walker 14 Jun '04

14 Jun '04

On Jun 14, 2004, at 11:01, Jay A. Kreibich wrote: On Mon, Jun 14, 2004 at 09:35:24AM -0700, Justin Walker scratched on the wall: On Jun 14, 2004, at 8:11, Jay A. Kreibich wrote: [snip] The point of being in single-user mode is that nothing happens without your executing it directly. There should be no 'background' jobs running, including 'update'. But the way you get into single user mode is that init issues a kill(2) with a PID of -1. According to the kill(2) man page this causes "the signal is sent to all processes excluding system processes and the process sending the signal." In truth, it kills EVERYTHING except init (the process sending the signal). That includes somewhat useful processes like mach_init (PID=2), without which the OS is, shall we say, somewhat limited. man init The "single user mode" you get on boot, and the "single user mode" you get via a SIGTERM to init(1) are two different things. Not really. I also can't sync(1) manually after the system is in single user mode because init will kill my script, as well as everything else. Can you make your process sensitive to the shutdown protocol (catch SIGTERM) and do the close at that point? Yes. init(2) sends SIGHUPs, SIGTERMs, and finally SIGKILLs to each process before the transition, but it does not call sync(2) itself. I'm assuming the stuff running is written well enough to catch the HUP or TERM correctly and shut themselves down. Even if they aren't, the kernel should clean up after the processes and close(2) all the active file descriptors for each process. The kernel does exactly that. Reacting to the SIGHUP is likely to generate dirty buffer pages that need to be written to disk, however, only now update(8) isn't running. For example, if the software is smart enough to check-point, it is very likley to generate I/O, which will dirty pages that need to get written out before I remove power. Hence, the quesiton on if close(2) will take care of the dirty work. To repeat: 'close' makes sure that the pages are marked for update; update makes sure (via 'sync') that the dirty pages are written to disk. You can also call 'sync' yourself. The kernel is not the place where policy decisions are made; hence the separation between the kernel and 'sync'. The decision to keep writing dirty pages to the backing store is made in user mode. If you make transition to single-user mode correctly, all should work. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | Men are from Earth. | Women are from Earth. | Deal with it. *--------------------------------------*-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Jay A. Kreibich 14 Jun '04

14 Jun '04

On Mon, Jun 14, 2004 at 09:35:24AM -0700, Justin Walker scratched on the wall: > On Jun 14, 2004, at 8:11, Jay A. Kreibich wrote: > >>>On Jun 7, 2004, at 21:56, Jay A. Kreibich wrote: > >>> > >>>> I have a fairly simple question, but I need a very sure answer. > >>>> If close(2) is called on the last file descriptor that references > >>>> a standard file, are dirty pages associated with that file's buffers > >>>> automatically queued for writing, in a similar manor to sync(2)? > > As I re-read the question, I realize it might not be clear, but my > > question is explicitly about the behavior of close(2). I can't > > depend on update(8), as it is killed by init when it drops the system > > into single user mode. Yes, I know the docs imply it shouldn't be, > > but it is. So I need to depend on the close(2) call to queue the > > dirty pages for writing. > > The point of being in single-user mode is that nothing happens > without your executing it directly. There should be no 'background' > jobs running, including 'update'. But the way you get into single user mode is that init issues a kill(2) with a PID of -1. According to the kill(2) man page this causes "the signal is sent to all processes excluding system processes and the process sending the signal." In truth, it kills EVERYTHING except init (the process sending the signal). That includes somewhat useful processes like mach_init (PID=2), without which the OS is, shall we say, somewhat limited. The "single user mode" you get on boot, and the "single user mode" you get via a SIGTERM to init(1) are two different things. > > I also can't sync(1) manually after the system is in single user > > mode because init will kill my script, as well as everything else. > > Can you make your process sensitive to the shutdown protocol (catch > SIGTERM) and do the close at that point? Yes. init(2) sends SIGHUPs, SIGTERMs, and finally SIGKILLs to each process before the transition, but it does not call sync(2) itself. I'm assuming the stuff running is written well enough to catch the HUP or TERM correctly and shut themselves down. Even if they aren't, the kernel should clean up after the processes and close(2) all the active file descriptors for each process. Reacting to the SIGHUP is likely to generate dirty buffer pages that need to be written to disk, however, only now update(8) isn't running. For example, if the software is smart enough to check-point, it is very likley to generate I/O, which will dirty pages that need to get written out before I remove power. Hence, the quesiton on if close(2) will take care of the dirty work. -j -- Jay A. Kreibich | Integration & Software Eng. jak(a)uiuc.edu | Campus IT & Edu. Svcs. <http://www.uiuc.edu/~jak> | University of Illinois at U/C _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Justin Walker 14 Jun '04

14 Jun '04

On Jun 14, 2004, at 8:11, Jay A. Kreibich wrote: On Fri, Jun 11, 2004 at 10:42:50PM -0700, Jim Magee scratched on the wall: On Jun 8, 2004, at 9:48 AM, Justin Walker wrote: On Jun 7, 2004, at 21:56, Jay A. Kreibich wrote: I have a fairly simple question, but I need a very sure answer. If close(2) is called on the last file descriptor that references a standard file, are dirty pages associated with that file's buffers automatically queued for writing, in a similar manor to sync(2)? Yup. I would modify Justin's answer slightly to say that "if the data was written with the write(2), etc... set of programmatic interfaces, the data will be sync'ed automatically by the update(8) daemon." That's quite a modification, given that it turns his "yup" into a "nope." It doesn't. Jim gave you a bit more information, is all. The answer is still 'yup'. For details see the lengthy 'udpated(8)' man page. But this is not really tied to the close(2) call. As I re-read the question, I realize it might not be clear, but my question is explicitly about the behavior of close(2). I can't depend on update(8), as it is killed by init when it drops the system into single user mode. Yes, I know the docs imply it shouldn't be, but it is. So I need to depend on the close(2) call to queue the dirty pages for writing. That is, as you say, a much different question, to which the answer is 'nope'. The point of being in single-user mode is that nothing happens without your executing it directly. There should be no 'background' jobs running, including 'update'. I also can't sync(1) manually after the system is in single user mode because init will kill my script, as well as everything else. Can you make your process sensitive to the shutdown protocol (catch SIGTERM) and do the close at that point? Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large * Institute for General Semantics | If you're not confused, | You're not paying attention *--------------------------------------*-------------------------------* _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Jay A. Kreibich 14 Jun '04

14 Jun '04

On Fri, Jun 11, 2004 at 10:42:50PM -0700, Jim Magee scratched on the wall: > > On Jun 8, 2004, at 9:48 AM, Justin Walker wrote: > > >On Jun 7, 2004, at 21:56, Jay A. Kreibich wrote: > > > >> I have a fairly simple question, but I need a very sure answer. > >> > >> If close(2) is called on the last file descriptor that references > >> a standard file, are dirty pages associated with that file's buffers > >> automatically queued for writing, in a similar manor to sync(2)? > > > >Yup. > > I would modify Justin's answer slightly to say that "if the data was > written with the write(2), etc... set of programmatic interfaces, the > data will be sync'ed automatically by the update(8) daemon." That's quite a modification, given that it turns his "yup" into a "nope." > But this is not really tied to the close(2) call. As I re-read the question, I realize it might not be clear, but my question is explicitly about the behavior of close(2). I can't depend on update(8), as it is killed by init when it drops the system into single user mode. Yes, I know the docs imply it shouldn't be, but it is. So I need to depend on the close(2) call to queue the dirty pages for writing. I also can't sync(1) manually after the system is in single user mode because init will kill my script, as well as everything else. > If the data was written into > the memory of a mapped file, the data would only by sync'ed with an > explicit call to msync(2). Otherwise, it stays modified in memory > until the system needs to recycle the memory (even an explicit sync(8) > call will not push it out). That's a good caveat to know. Thanks. -j -- Jay A. Kreibich | Integration & Software Eng. jak(a)uiuc.edu | Campus IT & Edu. Svcs. <http://www.uiuc.edu/~jak> | University of Illinois at U/C _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: file sync on close(2)
by Jim Magee 12 Jun '04

12 Jun '04

On Jun 8, 2004, at 9:48 AM, Justin Walker wrote: > On Jun 7, 2004, at 21:56, Jay A. Kreibich wrote: > >> I have a fairly simple question, but I need a very sure answer. >> >> If close(2) is called on the last file descriptor that references >> a standard file, are dirty pages associated with that file's buffers >> automatically queued for writing, in a similar manor to sync(2)? > > Yup. I would modify Justin's answer slightly to say that "if the data was written with the write(2), etc... set of programmatic interfaces, the data will be sync'ed automatically by the update(8) daemon." But this is not really tied to the close(2) call. If the data was written into the memory of a mapped file, the data would only by sync'ed with an explicit call to msync(2). Otherwise, it stays modified in memory until the system needs to recycle the memory (even an explicit sync(8) call will not push it out). --Jim [demime 0.98b removed an attachment of type application/pkcs7-signature which had a name of smime.p7s] _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0

Re: ?? how to get SystemStarter/diskarbitrationd to wait for drives to mount ??
by Dan Markarian 11 Jun '04

11 Jun '04

Hi Brian, "noauto" works. "" is default. On Jun 11, 2004, at 12:03 PM, Brian Bergstrand wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dan, The UUID/fstab setup has worked for me since 10.3.x. I have four volumes setup this way to mount on /Users, /usr/local and too disable auto-mounting for the other two UUID=xxxxxxxxxxxxxxxx /usr/local hfs rw 0 2 UUID= xxxxxxxxxxxxxxxx /Users hfs rw 0 2 UUID= xxxxxxxxxxxxxxxx /Volumes/Data hfs rw,noauto 0 2 UUID= xxxxxxxxxxxxxxxx /Volumes/Users hfs rw,noauto 0 2 The trick is, that you have to duplicate this info in NetInfo /mounts for it to actually work. Then use nidump to create the fstab file: sudo nidump fstab / > /etc/fstab Now that the info is in both fstab and NetInfo things work fine on 10.3.x (and 10.2.x). It would be nice if NetInfo didn't need a copy, but that's no big deal. Is that the oversight you were talking about? On Jun 11, 2004, at 8:18 AM, Dan Markarian wrote: Hi Richard, It is not that startup is not waiting for that volume, rather it is that we chose we not to mount it until you log in. The volume is on a disk marked external, which on a Client install, does not get mounted until someone logs in. You should be seeing it mount at boot on a Server install just fine, by contrast. There are two ways to have it mount at boot. The first one, also the ideal one, is unfortunately not working in 10.3.x due to an oversight on our part. I mention it only for future reference. It would be to list the volume in question by UUID in /etc/fstab along with the "auto" option. This would override any default. Brian Bergstrand <http://www.bergstrand.org/brian/>, AIM: triryche206 PGP Key: <http://www.bergstrand.org/brian/misc/public_key.txt> If all the world's a stage, I want to operate the trap door. - Paul Beatty As of 10:55:00 AM, iTunes is playing "Ballerina" from "Passion And Warfare" by "Steve Vai" -----BEGIN PGP SIGNATURE----- Version: PGP 8.0.3 iQA/AwUBQMnJunnR2Fu2x7aiEQJTrACfXPEojq+aaV4/kgN11UMHJAL8tHQAnA+d 9FLSn+sqP6dxVMETLTqLGnu4 =rDJJ -----END PGP SIGNATURE----- _______________________________________________ darwin-kernel mailing list | darwin-kernel(a)lists.apple.com Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-kernel Do not post admin requests to the list. They will be ignored.

1 0