July 2008 - Darwin-kernel - lists.apple.com

Determining specific file sessions
by Jernej Azarija 22 Jul '08

22 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=EDkao/UdkDeeTTBL4rS8aGqM8fYX/5pYcOyfVkptY+0=; b=rQwQmZkI2EGH4g8DWqzblMD5O5E1HOF+CqOhGHTJGVWzxjGhcXWH6gks8gWgnRrr5l 5s1gfCr++4GVHrylrmP+L1ctDVmdmnDoMJKi6CtUxwTnEaWKI0C8VYqXLN4u3hfUwmgY D8+VUpaexOoo0883WLklj8dJGJtvpI8Hnu34g= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=e2c1M//Tz4wF8HFSADk6+zp8lcxQsOXX9runYx+U0q1+ppc3PHbf5RRH5zdbt2GSFF 6ThtJlWm+WfGBKnwizP50Yw1YOc5kQgrNBzGe4kCnF6QxJo1YSY5FrJsFvyIzxEjEN2q B6XNK60q5M85BRSacW5nOqMVF7WVuSqm6TRW4= Hello, in my current project I'm using Kauth for a security application. Part of the application deals with file operation checking and therefore uses KAUTH_SCOPE_VNODE. It is required for the application to exactly know when a specific process open/closed a file. For example, the file `foo.bar' may be accessed by many processes each holding more than one file descriptor associated with the given file. It is therefore not enough to know the time a file was opened/closed as it may relate to different processes and/or file descriptors within the process. The only valid idea I could think of to distinguish between various instances of opened files is to relate each open session with the process ID and process FD associated with the file. As to my knowledge, this is impossible within the VNODE scope, and I'm afraid there is no legal way to do it either (am I right?). I'd be really thankful is someone could tell me if there is a way to achieve what I've specified, or alternatively anoher way to get the same result (either in kernel or user mode). Thanks, Jernej Azarija _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: A question about the filter scheme
by XiongJia Le 14 Jul '08

14 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=paYYAAeRrC1muUf6bwUoZ3OBmfNMGxpHXsn+sMv+mvk=; b=cM2OJmEpELoF4f4s0swrX4kpl/XNBM2jd6WFJhoQoMZF8mEFTWy388Agirfa0L85sJ dc1LKS1p+TitYE33YU0i879wpiT4nrfACD938ipJmZX1yXXHfoSE214fdP782Ekcmuyx lTjKb220Nmb2tXSiqj6ibipFoRpkTOi/3+rVo= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=USNsfa/26AIWkeDPktmCaQLQ5ZJokx5mKhE51pMWuQ4TYPT+ETqAyWAjznXAIRLZwb 1nyRxjFwSMSRXddT3swDz8CAb3YOWEji1Wh+hPmKO3xM1XXfgCWgwZHDdr1tLdOuda7S HSWTGRJq4MOJ+kGpcphd1TnDzuUJl2bH5o//k= Thanks. But I am not understand, Is only the "Content hint" can control the driver match action ? Because think the AppleRAID Driver is base on IOFilter. And AppleRAID can filter all local disk. So,I read the AppleRAID Driver from Open Darwin. That was got from the AppleRAID Driver's Info.plist file. <key>IOPropertyMatch</key> - <array> - <dict> <key>Content Hint</key> <string>Apple_Label</string> </dict> - <dict> <key>Content Hint</key> <string>4C616265-6C00-11AA-AA11-00306543ECAC</string> </dict> </array> Should I need add this to my driver's Info.plist file? I am trying this. If you know some info about this tell me thanks. Thanks, Xiongjia Le On Mon, Jul 14, 2008 at 11:33 AM, Tony Wong <masterofmanythings(a)hotmail.com> wrote: > I am not sure I am right but at least this is what I experienced. > > Content hint is difficult to use because you will have to specify > the condition for each partition. If you write the driver for > yoruself, it is fine but it is not easy if you want to send it to > many customers. > > I have worked around this by checking the properties of the > IO chain. For example, disks on the Fiber Channel have different > properties like UNIT_SERIAL_NUMBER, etc and on board local disks > do not have (not sure this is the correct example but U get my point). > By checking on these properties, you can distinguish if a drive is > local or not. > > Tony Wong > >> Date: Sun, 13 Jul 2008 22:17:28 +0800 >> From: lexiongjia(a)gmail.com >> To: darwin-drivers(a)lists.apple.com; darwin-kernel(a)lists.apple.com >> CC: >> Subject: A question about the filter scheme >> >> Hey folks, >> >> I am an newbie in the KExt programing. But I need write a local disk >> I/O filter Driver in OS X. >> I have read the filter scheme sample at the apple document site. But I >> found I/O filter scheme need a content hint to matched attach >> partition. >> >> Is there any way to filter all local hard disk by one filter scheme >> driver? >> >> Thanks, >> Xiongjia Le >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Darwin-drivers mailing list (Darwin-drivers(a)lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> >> http://lists.apple.com/mailman/options/darwin-drivers/masterofmanythings%40… >> >> This email sent to masterofmanythings(a)hotmail.com > > > ________________________________ > The i'm Talkaton. Can 30-days of conversation change the world? Find out > now. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

A question about the filter scheme
by XiongJia Le 13 Jul '08

13 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=8T9vd+V7YAQwOVcuhpbl6rufgctV5WWkUGt4wqT3E7s=; b=LXTQTLkce+hkxf+br3qiWjcaT00BUBUzW3rYcALfJDrKqYvOKJs9/vnMOxPvf95tE6 mOLhR3BrU83rTwTEqfxUjhiX5foMCWk+EicqxC1m898B22Vac8s810YU1HjNp+HsEH8z T/02P+LRf418dYynjzAaAlQhydt1I7/6CMBEQ= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=w7tgB0qUtdA92/dU5idhezt9VEbeDgmQYzuxh62iKV+H3Y5YNLvCCo8UGVzSd0/xZn 6wElfF1XNlYMlbYpwYkPqjLc1d7FVC0dY+UYRzwq6ZnpvJ7LBD3CM9HRggAH2Jft77RE 8NvdJkQH3g1AieflX42rVwaH6+2CE1C5TJsQg= Hey folks, I am an newbie in the KExt programing. But I need write a local disk I/O filter Driver in OS X. I have read the filter scheme sample at the apple document site. But I found I/O filter scheme need a content hint to matched attach partition. Is there any way to filter all local hard disk by one filter scheme driver? Thanks, Xiongjia Le _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by Terry Lambert 13 Jul '08

13 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by Terry Lambert 13 Jul '08

13 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by shailesh jain 13 Jul '08

13 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=2+ufRnv3XC8eD+LPwxD6P+CT6P4uDULjMGzXNBsUKGY=; b=SKbFX7NoHwvmTbypiFMYh/QthjIMMXLdo1ilihwKelkZG4+DI1r0+jEcQnH7A2q98D L6eb1jwdHZds6dmnN5Lr+DNhJ+yg0xd6/azNmg2Yt2iW8TAoqPErFNN0ZfX8qSlGoQ79 0zkaFH4LTNivDQ6nvbxAHAgOgXyemQ80GeB38= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=g+8JHyuR0C0mhV8LJwawdAltKwC2VcodOnVjoIYJ5wLOp7L3UwXSwdGtXtkXCU0b3s hdpC3BriD6zoVQLSzFZpMrQd86D2SBbxZz0BxbAfoKXo0LdOVe+Nfco4re/gNPP6GJqz wfYBJzRySxJy+MrZPOuirjXVXd8c6zKRl4wPE= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by Michael Smith 12 Jul '08

12 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Jul 11, 2008, at 3:56 PM, shailesh jain wrote: It's not clear what you mean by "flushing mmaped pages". = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… 1) If the user mmap's a file and makes some changes to it and then calls read() system call (without msync'ing), is it the responsibility of a filesystem to flush the changes that could have been made by mmap system call before actually reading the file ? No. The Darwin VM and buffer cache are fully coherent. The read(2) call may race with another thread that is currently updating the mapped region, however. 2) Also, while referring to source for smbfs, I came across a comment that said " we shall maintain synchronization between mmap and read/write by using UPL". Isn't just flushing mmaped pages enough ? The smbfs code uses the UPL as its co-ordination with the VM; by passing file I/O through the buffer writes update the VM. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by Terry Lambert 12 Jul '08

12 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Hard reality time: Therefore, no vendor implements memory change notification like this. -- Terry /Shail On Tue, Jul 8, 2008 at 10:46 PM, Michael Smith <drivers(a)mu.org> wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… On Jul 11, 2008, at 3:56 PM, shailesh jain wrote: 1) If the user mmap's a file and makes some changes to it and then calls read() system call (without msync'ing), is it the responsibility of a filesystem to flush the changes that could have been made by mmap system call before actually reading the file ? There is no such thing as an uncached mmap(). This is because you will write to the mapped version of the page in the buffer cache, and it will then be marked dirty, so that multiple writes in the same page can be "gathered" and written simultaneously. The only way to implement that completely uncached would be to map the page read-only and set a page attribute flag to indicate that "this is really an uncached write page". The net result of this would be a page fault each time you wrote a byte in the page, and then in the page-fault fixup handler in the VM, you'd see the flag and "fix the fault" with a write of the entire page to the underlying device. This is because processors only provide page protections on a page boundary and not on a byte boundary, for (I hope) obvious reasons. This would be incredibly, incredibly expensive, and, if this were, for example, a flash device, you would exceed the spec.'ed number of available write cycles very quickly, after which no more writes would work because the device was "used up". Even something like a disk would end up "used up" pretty quickly by this kind of behaviour, which is one reason there are disk caches. Typically, if someone does something insane like trying to implement a totally uncached device, then rather than permitting mmap at all, they just disable it and say "sorry, no mmap for this device", or they modify their hardware design to include something like static column DRAM, so that writes to the device are cached by the device (and they lose their non-cached implementation that way, instead). E.g. treat as incoherent VM and buffer cache. IF you are asking about cache coherency in general, instead, and have given up on trying to cut the ubc out of the picture, THEN the answer is that ubc is an implementation of a coherent VM and buffer cache. The msync() system call was invented to support the concept of explicit application-based coherency, when the application chooses to mix read/write and mmap based access. The msync is intended as a barrier following memory writes, which causes a sync from the VM to the buffer cache in systems where the VM and buffer cache are decoherent (systems such as SVR3, BSD 4.3, Solaris pre-Solaris 5 and post Solaris 9, etc.). You will generally only see it used in applications that mix I/O styles, usually because they were written and modified over long periods periods of time by different people, and grew organically instead of being designed. An example is the classic "netnews", which actually gets its barriers and msync's right. 2) Also, while referring to source for smbfs, I came across a comment that said " we shall maintain synchronization between mmap and read/write by using UPL". Isn't just flushing mmaped pages enough ? Not unless there are serialization barriers between the calls to write and modifications via mmap. An msync will only sync out the range specified, and that will turn into writes if the dirty bits are set on the page(s) in that range. There is no guarantee that there will not be a memory modification in a page, followed by a write, followed by another memory modification, followed by an msync, other than the application author knowing what they are doing and instituting order of operation guarantees in their code (this is how "netnews" did it). Failure to order your memory and write operations will result in a stale version of data in the cached page that is later msync'ed, which means that any writes you did between memory modification and msync end up being lost, as the VM page is copied over the buffer cache page (in the case of a decoherent VM and buffer cache), or the cached page contents overwriting the uncached write (in the unified VM and buffer cache case, for systems which support application-forced uncached writes, like MacOS X). The moral of this story is to not mix memory and file based I/O in decoherent systems, and to not mix cached and uncached I/O even in coherent systems, unless you know what you are doing, and barrier appropriately. Since you can't actually control what an application author does on your system, typically the best strategy is to by default unify your VM and buffer cache and always do cached I/O (implicit coherency). Then give the people who know what they are doing an "escape hatch" from the default behaviour. On Jul 8, 2008, at 2:25 AM, Terry Lambert wrote: On Jul 7, 2008, at 7:21 PM, shailesh jain <coolworldofshail(a)gmail.com> wrote: I am writing a file system that does not support caching. I wanted to know, If by just passing NO_CACHE and CANT_CACHE option, I can disable entire caching ? As of my understanding, the options are just to disable name lookup etc caching and it cannot disable file content caching i.e where UBC comes into play. Can I disable file content caching i.e UBC ? (set it to UBC_INFO_NULL ?). Also, apple docs says that UBC for apple is different from FreeBSD. I wanted to know in what respect ? Read the sources. I put massive block comments in the UBC code for just such an emergency. 8-). Also, look at the sources for older versions of smbfs, which had similar issues. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Read/Write call interaction w/ UBC
by shailesh jain 11 Jul '08

11 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=FCdA0V3XJW5qFzVPpmk0y1aLLeAnNkg3z+sVQuBL+hU=; b=GF4KBtKo7ICWC+GgT1dmzH0tiI2yivV4nOB2dz8ZqstzclewJ92Nxo9z771kqSPUiB 97tURS82ooHvv35F2FNFc5Vb9ZeAwgfts4v5wF21CucMGthq78n2JxmYL/gub5ZFmP2p PMzCsgN9+SsTX+h7xHep41ZALT3L+xxb5SSXI= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=x6/Z5IHWj6eBqoFT1a1dYrNN+pSQthlK09MrDW872F6F00wBIqeAOEv/g0GBHw4jnf ZG63xLstVeJQAVGgPJ3oTGsnha4eBcdk/aIRL0oOK8SBLJPrNxpyxfVOnytWZxVBgCZ3 wsqSSFftisVyrS7q6D4zZMbqv6ZLfvJGLaZ9M= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: kauth FILEOP_CLOSE_MODIFIED semantics for unexpectedly terminated processes.
by Maxim Zhuravlev 11 Jul '08

11 Jul '08

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=BAeMaMOSb4vgwPsSNweH1n2F8MtuQFoVv5IxBRTI+9w=; b=Tj9C01iZL64RhIznsUpBRIko08X7PHwVr9CRAXDb6tKSsxulPEu5jROiEuoDEt9Kl2 ui8YShICZqKOle7NKDj/cvAj9UIJYcQRt3zWVXWE5BVzsZb0NWreex/T3AIZ0KZ/rW3T v/eHT8rFrpyXM8sRa7rtYWvyVWPrV/25RU3II= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=O49+mhPtPOCA/jH0olLGqmhikkNORWn2mXG84kDNbNV1UICCeOlT2HzE+bFPQIoPAF 5zSIxKmHZP2Wx6iLQ04QGbztEk3nTx+8lvmx8wPb55KxuglBqv3zBHLdkKBg/aNAKQXW cpV+vuEamdnlbk6SUlmRYOGB+IfE1OAmRe3xs= 2008/7/11 Quinn <eskimo1(a)apple.com>: > At 12:13 +0400 11/7/08, Maxim Zhuravlev wrote: >> >> I've been explorering the kauth FILEOP scope and found out, that for >> unexpectedly terminated or exit()'d process, which hadn't explicitly closed >> a file handle it was writing to, a KAUTH_FILEOP_CLOSE notification with >> KAUTH_FILEOP_CLOSE_MODIFIED flag set isn't issued. This happens, cause >> exit() closes all handles bypassing close() syscall, which actually checks >> flags and issues that kind of notifications. > > I would classify that as a bug. I encourage you to file it. Please post > the bug number so that I can keep an eye on it. > > <http://developer.apple.com/bugreporter/> Bug ID: 6069147 _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0