October 2006 - Darwin-dev - lists.apple.com

Re: Limit on kevent entries?
by Michael Smith 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dear darwin-dev list, is there a limit on the number of files one can simultaneously monitor with the kevent mechanism? Basically, I want to monitor if a file has changed, for potentially hundreds or even thousands of files. kevent is not suitable for this. There are other mechanisms in the pipeline that improve the developer experience, but this is your best option for now. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Oct 12, 2006, at 12:02 PM, maurits <listpost(a)plasticsfuture.org wrote: Apart from the technique discussed in Amit's book, you might also want to consider using the kauth FILEOP scope. This will require that you write some kernel code and come up with a matching and queueing mechanism for notifications that is efficient (because you are called on the hot path), but it's a precise mechanism that is available in 10.4, and which was developed for this specific reason. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by maurits 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com d'uh. Good point. I used O_RDONLY... Thanks again for the great help, Moritz -- http://www.plasticsfuture.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Thu, 12 Oct 2006, maurits wrote: OK, I'd highly appreciate any input. Portability is not an issue for me. But I have the feeling that kevents doesn't really scale too well, and I agree with you that I'm probably abusing the mechanism... In general kqueues do scale well, and that's part of the purpose of them. We use them to multiplex I/O over many thousands of sockets. I haven't pushed them terribly hard on Darwin (just enough to hit some bugs in older versions) but I have no reason to believe they shouldn't scale well there too, relative to Darwin's level of low level performance in general. cool, that's good to hear. I guess I didn't mean that kqueues don't scale well (in fact they do, it seems), but I'm just not sure how good it is to crank up the number of open files to really large numbers. However, I've now used setrlimit() to set it up to 5000, and my machine is still alive, and the code works just great now. Guess I can live with a limit in that region. P.S. if you continue to use kqueue, make sure you open the files with O_EVTONLY, if you aren't already. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by maurits 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Amit Singh's got a sample app which uses the same mechanism that Spotlight uses for spotting filesystem events, i.e. reading from /dev/fsevents. This is not officially documented of course, but you can look at its implementation in the Darwin sources. neat. This looks very interesting. Thanks to all of you. Cheers, maurits -- http://www.plasticsfuture.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… <http://www.kernelthread.com/software/fslogger/> Unfortunately, I guess there are a number of reasons that preclude a deployment of similar techniques in end-user apps, at least the requirement to run as root, the undocumented API, and the possible interference with Spotlight. But interesting nevertheless. So I guess I'll stick with my kqueues solution, with cranked-up RLIMIT_NOFILE. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by Chris Ridd 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: AcbtxiI3YQkf4Fm5EduZGgARJHlZbA== Thread-topic: Limit on kevent entries? User-agent: Microsoft-Entourage/11.2.5.060620 On 12/10/06 5:57, Ian Lister <list-darwin-dev(a)lister.dnsalias.net> wrote: > On Thu, 12 Oct 2006, maurits wrote: >> is there a limit on the number of files one can simultaneously monitor with >> the kevent mechanism? > > There are several limits on the number of files you can have open, which > they have to be to be able to watch them with EVFILT_VNODE. > > $ ulimit -n > 256 > $ sysctl kern.maxfiles > kern.maxfiles = 12288 > $ sysctl kern.maxfilesperproc > kern.maxfilesperproc = 10240 > [etc] > >> Basically, I want to monitor if a file has changed, for >> potentially hundreds or even thousands of files. The code I have (based on >> Michael McCracken's stakeout) seems to be able to monitor no more than 251 >> files (weird number, I don't have a clue why). > > Because the default resource limit is 256 open files, and you have a few > things such as stdin/out/err and the kqueue open, as well as all the files > you're trying to monitor. > > You can increase the resource limit to kern.maxfilesperproc, which might > be "good enough" for your purposes. > >> Is there a limitation? Is it >> even sensible to monitor so many files simultaneously using the kevent >> mechanism? > > I don't think so, mostly because it's not sensible to have so many files > open in order to be able to monitor them. I think there are non-portable > ways to do this at higher levels (e.g. the way Spotlight and Time Machine > watch huge chunks of the filesystem), but I don't know them. Try some > filesystem doc, or wait for somebody more knowledgeable to provide a > better answer :-) Amit Singh's got a sample app which uses the same mechanism that Spotlight uses for spotting filesystem events, i.e. reading from /dev/fsevents. This is not officially documented of course, but you can look at its implementation in the Darwin sources. <http://www.kernelthread.com/software/fslogger/> Cheers, Chris _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by Ian Lister 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Cheers, Ian On Thu, 12 Oct 2006, maurits wrote: OK, I'd highly appreciate any input. Portability is not an issue for me. But I have the feeling that kevents doesn't really scale too well, and I agree with you that I'm probably abusing the mechanism... In general kqueues do scale well, and that's part of the purpose of them. We use them to multiplex I/O over many thousands of sockets. I haven't pushed them terribly hard on Darwin (just enough to hit some bugs in older versions) but I have no reason to believe they shouldn't scale well there too, relative to Darwin's level of low level performance in general. P.S. if you continue to use kqueue, make sure you open the files with O_EVTONLY, if you aren't already. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by maurits 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Ian, Thanks again, maurits -- http://www.plasticsfuture.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Thu, 12 Oct 2006, maurits wrote: is there a limit on the number of files one can simultaneously monitor with the kevent mechanism? There are several limits on the number of files you can have open, which they have to be to be able to watch them with EVFILT_VNODE. boy, of course I forgot to realize that I'd have all those files open... Thanks for your answer. That makes a lot of sense now. Even the 251 :) I think there are non-portable ways to do this at higher levels (e.g. the way Spotlight and Time Machine watch huge chunks of the filesystem), but I don't know them. Try some filesystem doc, or wait for somebody more knowledgeable to provide a better answer :-) OK, I'd highly appreciate any input. Portability is not an issue for me. But I have the feeling that kevents doesn't really scale too well, and I agree with you that I'm probably abusing the mechanism... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Limit on kevent entries?
by Ian Lister 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com $ ulimit -n 256 $ sysctl kern.maxfiles kern.maxfiles = 12288 $ sysctl kern.maxfilesperproc kern.maxfilesperproc = 10240 [etc] Cheers, Ian _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Thu, 12 Oct 2006, maurits wrote: is there a limit on the number of files one can simultaneously monitor with the kevent mechanism? There are several limits on the number of files you can have open, which they have to be to be able to watch them with EVFILT_VNODE. Basically, I want to monitor if a file has changed, for potentially hundreds or even thousands of files. The code I have (based on Michael McCracken's stakeout) seems to be able to monitor no more than 251 files (weird number, I don't have a clue why). Because the default resource limit is 256 open files, and you have a few things such as stdin/out/err and the kqueue open, as well as all the files you're trying to monitor. You can increase the resource limit to kern.maxfilesperproc, which might be "good enough" for your purposes. Is there a limitation? Is it even sensible to monitor so many files simultaneously using the kevent mechanism? I don't think so, mostly because it's not sensible to have so many files open in order to be able to monitor them. I think there are non-portable ways to do this at higher levels (e.g. the way Spotlight and Time Machine watch huge chunks of the filesystem), but I don't know them. Try some filesystem doc, or wait for somebody more knowledgeable to provide a better answer :-) This email sent to site_archiver(a)lists.apple.com

1 0

Limit on kevent entries?
by maurits 12 Oct '06

12 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dear darwin-dev list, For your reference, my code looks somewhat like this: int kq = kqueue(); err = kevent(kq, &ev, 1, NULL, 0, &nullts); } EV_SET(&ev, SIGIO, EVFILT_SIGNAL, EV_ADD | EV_ENABLE, 0, 0, 0); err = kevent(kq, &ev, 1, NULL, 0, &nullts); and then I poll using n = kevent(kq, NULL, 0, &ev, 1, NULL); Thanks, maurits -- http://www.plasticsfuture.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… is there a limit on the number of files one can simultaneously monitor with the kevent mechanism? Basically, I want to monitor if a file has changed, for potentially hundreds or even thousands of files. The code I have (based on Michael McCracken's stakeout) seems to be able to monitor no more than 251 files (weird number, I don't have a clue why). Is there a limitation? Is it even sensible to monitor so many files simultaneously using the kevent mechanism? for (every file...) { EV_SET(ev, fd, EVFILT_VNODE, EV_ADD | EV_ENABLE | EV_CLEAR, NOTE_RENAME | NOTE_WRITE | NOTE_DELETE | NOTE_EXTEND, 0, (void *) path); This email sent to site_archiver(a)lists.apple.com

1 0

Re: [APPL:DarwinDev] Re: cp -f option
by Jay A. Kreibich 11 Oct '06

11 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mutt/1.4.2.1i On Wed, Oct 11, 2006 at 10:15:40PM +0100, Graham J Lee scratched on the wall: > On 11 Oct 2006, at 21:29, Jay A. Kreibich wrote: > > > lacks write permission. "-f" will overwrite such a file, assuming > > the user has permission to grant themselves write permission. > > Presumably it actually will go ahead iff the user has write > permissions in the containing directory and (the file is owned by the > user or sticky is unset). You don't need write permissions on a file > to delete the file, and cp could just delete the in-the-way file in > order to create its new file. Good point. I assume that's correct. -j -- Jay A. Kreibich | CommTech, Emrg Net Tech Svcs jak(a)uiuc.edu | Campus IT & Edu Svcs <http://www.uiuc.edu/~jak> | University of Illinois at U/C _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: [APPL:DarwinDev] Re: cp -f option
by Graham J Lee 11 Oct '06

11 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 11 Oct 2006, at 21:29, Jay A. Kreibich wrote: lacks write permission. "-f" will overwrite such a file, assuming the user has permission to grant themselves write permission. -- Graham J Lee http://www.thaesofereode.info/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Presumably it actually will go ahead iff the user has write permissions in the containing directory and (the file is owned by the user or sticky is unset). You don't need write permissions on a file to delete the file, and cp could just delete the in-the-way file in order to create its new file. This email sent to site_archiver(a)lists.apple.com

1 0