March 2006 - Darwin-dev - lists.apple.com

Re: Why no work on rsync?
by Peter O'Gorman 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla Thunderbird 1.0.2 (Macintosh/20050317) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jordan K. Hubbard wrote: | It's all just a bit messy, which is probably why nobody's even wanted | to touch user space. They just do the filesystem support, declare | victory and run away. :-) I'm not quite sure how hard you guys looked. There are patches available for GNU coreutils (cp etc), rsync and others. I'm assuming that they are supported by the distros that use them. Example: http://cvs.samba.org/cgi-bin/cvsweb/rsync/patches/#dirlist Peter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) iQCVAwUBRApnQ7iDAg3OZTLPAQLXygP+MOOgz/X5+Sf2IysY0GkRUmUM/jJGq8P8 IrBiBDnpr4y0hMfFGBAP4UWpKmEehdRKByxrwgQ3sjC1yEC+4gJ65VnqY+ZzLbiH gCUaI3WesiJQLHQ9QY4jIS4aBzdd+jFjm2HqIQExck9U0XZIpI3SoCQfO1sK3E9y kkBxiFALsVk= =zR+x -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no work on rsync?
by Finlay Dobbie 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=sIBJZzLk1j48EoqiDgS+KIwoopvcaIgBoyORQPWcTbXe1dlx5RCsLS5JVsAR8xcU5G+a1kuZVz3eDX5nMVOkhDv+j3YCw76143BwAAwvJFf+IbdMkNohs4LzciA+IDed6cOzlcYkX01eV4ShBJ+C7FzALSk/WtVgdqlwTxWP8Y4= On 3/4/06, Martin Costabel <costabel(a)wanadoo.fr> wrote: > Finlay Dobbie wrote: > > Not to mention, I believe the implementation uses SPI and as a result > > isn't even buildable out of the box (needs private Darwin headers, > > builds with darwinbuild fine though). > > There is only one private header file needed: copyfile.h How many private headers is irrelevant: Apple's patches rely on private headers, if upstream rsync was to merge their changes then they would break the ability to build rsync out of the box on Mac OS X. Period. -- Finlay _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: missing task_name_t
by stephen joseph butler 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=QNfwlpQW/oACkffXGhc7JKGrRwU0P3Kp7y/P0XaOpa/GIoAbijMMy69Ue87j/knbza5aTH6ES3OD1huXl41pmWOsg0MGCuHzPZHAaU/RzWDBYH6jv7s4NN6gK7ar36iFpattlYj1KlKfwiZzICBamTOu0a1UqxJrIharte62J+0= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com DirectoryServiceDaemon-352.1.log~3

1 0

Re: missing task_name_t
by stephen joseph butler 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=JqbzT7/sIqoyBho5gvoqyu/0vDDPZAAJftqb5pfRI8APHLKg+0Xg2saMZWW7GkiNFUVsEiFX4jvQnXiU/RtCa20C7LeatZMDlQ2OCRZvt5Q1ukU0Gxshgu+/n1hA4KB9a3Upx1Tx4QH4vwmyyJyWbjvny1XWykHiOaW3+8x0Q84= _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: SQLite on Tiger - THREADSAFE on or off ?
by Eric Albert 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Mar 4, 2006, at 4:42 PM, Mark Gilbert wrote: I am using LibSQLite3.0.dylib in a multi thread app. Hope this helps, Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I have encountered problems where access to the database from a second thread is showing the database as locked. After some research it would appear that libsqlite must be compiled with the THREADSAFE flag set to 1 in order to support multithread access. It also implied that UNIX distributions may have this set to 0 by default. So... first - is the library included with Tiger THREADSAFE on or off ? My results suggest not. The version included with Tiger was built with --enable-threadsafe. You can see this at <http://www.opensource.apple.com/darwinsource/10.4.5.ppc/SQLite-28/ Makefile>. This email sent to site_archiver(a)lists.apple.com

1 0

SQLite on Tiger - THREADSAFE on or off ?
by Mark Gilbert 05 Mar '06

05 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi There,. I am using LibSQLite3.0.dylib in a multi thread app. I am a little stuck here, probably down to my ignorance in 'roll your own'. Can anyone advise ? All I want is a thread safe libsqlite3.0.dylib Thanks Mark Gilbert. -- mark(a)gallery.co.uk Tel: +44 208 340 5677 fax: +44 870 055 7790 http://www.gallery.co.uk New Product ! - MetaCorder Location Sound Field Recorder http://www.metacorder.info _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I have encountered problems where access to the database from a second thread is showing the database as locked. After some research it would appear that libsqlite must be compiled with the THREADSAFE flag set to 1 in order to support multithread access. It also implied that UNIX distributions may have this set to 0 by default. So... first - is the library included with Tiger THREADSAFE on or off ? My results suggest not. Also, I have downloaded the sqlite source and piled into an unfamiliar manual rebuild of the library (I am not a Unix hacker...). I have configured and make'd the sqlite stuff, but it doesn't turn out a .dylib file (just lots of other files...). I did find a 'make install' command, but this seemed to be largely interested in creating extensions for TCL (it did make a libsqlitetcl dylib, however). This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no work on rsync?
by John Siracusa 04 Mar '06

04 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thread-index: AcY/5t45HNzSuKvaEdq41gAKlbpFBg== Thread-topic: Why no work on rsync? User-agent: Microsoft-Entourage/11.2.1.051004 On 3/4/06 6:13 PM, Jordan K. Hubbard wrote: > On Mar 4, 2006, at 2:37 PM, Boyd Waters wrote: >> Hmm. Actually you are correct: no standard tools support these >> various approaches. Filesystem architects have long grappled with >> this stuff, but no one has made user-space work well. > > And, just for the record, I'm not arguing that we did either. On this tangent, whenever I do "make dist" to build a Perl module, it creates ._ files and sticks them in the tar archive. (This is on an HFS+ disk.) They're not there before the tar because I delete them all prior to the tar step. On another, seemingly identical machine (same version of everything, same Perl source synced form SVN) it *doesn't* do this when I "make dist." And it's not tar itself that's doing it because if I use gnutar the same thing happens on the one machine (but again, not on the other). Mysterious. Anyway, I'm glad you guys are trying to make it work. Someone should :) > It's all just a bit messy, which is probably why nobody's even wanted > to touch user space. They just do the filesystem support, declare > victory and run away. :-) Wimps! (...like the Spotlight guys not using EAs...for shame! ;) -John _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no work on rsync?
by Jordan K. Hubbard 04 Mar '06

04 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Mar 4, 2006, at 2:37 PM, Boyd Waters wrote: - Jordan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… What? I find it difficult to believe that some research into EA preservation did not yield prior work. But that would be heavily dependent upon the timing, as this seems to be an active area of research. Alas. Well, the timing would have been very early Tiger, so I'm not saying our knowledge is entirely up to date here. It's always worth another look, but at the time, there wasn't much (well, ANYTHING) in the CLI space. ReiserFS version 4 has extensive support for per-file metadata. They still argue about the best way to represent such metadata, and the pseudo-subdir approach that I've seen some POSIX tools take with Macintosh metadata (e.g. filename/.resourcefork) is quite similar. Or ._filename, in Tiger's case, but yeah - you've always got the non- EA-aware filesystem interop case to deal with. Dealing with serialization/deserialization with different source/destination filesystems can be amusing too. Hmm. Actually you are correct: no standard tools support these various approaches. Filesystem architects have long grappled with this stuff, but no one has made user-space work well. And, just for the record, I'm not arguing that we did either. When I said we were at the "end of the spear" before (a phrase that Rob took some objection to), I wasn't trying to suggest that we had absolutely solved everything the right way - a lot of our command-line strategy was driven more by expediency ("Oh crap, we have EAs in HFS now - we need to quickly figure out how to preserve them in some number of common cases"). Just to lay the problem out for everyone else, you have to at least assume the following when analyzing approaches in this space: 1. That the source and the target for any operation may or may not support EAs, yet heterogeneous operations can't lose data. 2. That creating, editing, renaming, copying and deleting files all need to be able to keep data and metadata together, fully cognizant of the limitations posed by item #1. For us, that meant making sure that all the basic file primitives (cp, rm, mv, etc) did the right thing for a file / file pair (again, see #1) as an obvious first step, but transport requirements also meant modifying rsync, scp, tar, pax, cpio, etc so that all of this would also work over the network or in archival scenarios. Then we remembered that editors like vim and emacs did temporary file gymnastics in order to accomplish edits on files and that merely editing a file wouldn't be expected by most people to result in a loss of all its EA information, so those utilities had to be taught to transfer the EAs over and that's just two utilities which do that - there are more, including 3rd party editing tools, which are quite likely going to cause EA data loss until they start employing the same strategies. It's all just a bit messy, which is probably why nobody's even wanted to touch user space. They just do the filesystem support, declare victory and run away. :-) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why no work on rsync?
by Boyd Waters 04 Mar '06

04 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Mar 4, 2006, at 1:42 PM, Jordan K. Hubbard wrote: - boyd _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com (BIBA, Port ACLs, whatever) and dealing with EAs/ACLs on a daily basis, but it does beg the question: What problems have they solved along the way and why do the various GNU command-line tools they must be using not support EAs or ACLs yet? We did a search for support like this before hacking the CLI tools ourselves, and we found no apparent attempts to grapple with the problems of EA preservation / transfer. What? I find it difficult to believe that some research into EA preservation did not yield prior work. But that would be heavily dependent upon the timing, as this seems to be an active area of research. Alas. I've attempted to use SELinux for many years, but can't manage the policies in practice, so it was never more than a lab exercise for me. That said, I've long been actively interested in filesystem support for metadata. NTFS allows a large number of "named forks" per file, and they actually have used (well, supported) per-file rich metadata in the Office applications for years now (although for a long time they were bundling the metadata in a FAT-like mini-filesystem inside Office binary files, in order to support the metadata forks on non- NTFS filesystems, so that is rather moot). Any file operations on an NTFS volume would need to preserve this. Although it is not at all clear to me that any attempt is made to do so for POSIX tools like rsync running on Windows on Cygwin. ReiserFS version 4 has extensive support for per-file metadata. They still argue about the best way to represent such metadata, and the pseudo-subdir approach that I've seen some POSIX tools take with Macintosh metadata (e.g. filename/.resourcefork) is quite similar. Hmm. Actually you are correct: no standard tools support these various approaches. Filesystem architects have long grappled with this stuff, but no one has made user-space work well. smime.p7s

1 0

Re: Why no work on rsync?
by Jordan K. Hubbard 04 Mar '06

04 Mar '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Mar 4, 2006, at 12:18 PM, Rob Braun wrote: - Jordan _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… It's true there is no complete standard, but there are similarities in approach. It appears to be a sort of newtonian approximation process to find the common middle ground. Apple appears to be trying the same newtonian approximation process to a completely different function. The treatment of the resource fork as an EA just doesn't fit with Apple's or anyone else's EA model. The resource fork is like a stream that is kinda sorta treated like an EA. TAdditionally, Apple has deviated in the treatment of EA namespace and external representation of the EAs. I don't think I ever argued that Resource Forks mapped cleanly and completely into the EA model - to be sure, they predated EAs by many years, so any feature overlap is purely coincidental. That said, I think Apple made the right decision to map them into the EA space rather than promoting two different first-class metadata access APIs going forward. We can't escape the ResourceFork legacy (too many things still assume it), nor can we deprecate the Carbon APIs for dealing with resource forks directly, but we can at least try to subsume one into the other in the POSIX namespace so as to slowly deprecate the "fork" (pun intended) going forward. Redhat Enterprise Linux and Fedora have both had SELinux enabled by default for quite a while now. SELinux uses EAs and ACLs rather extensively and encountered and solved many problems along the way. There are still many problems and Apple would do well to pay attention to avoid the pitfalls they have encountered. Interesting point. II'm still not sure about the size of the SELinux installed base, that is to say the number of people actually running SELinux with one or more active policies (BIBA, Port ACLs, whatever) and dealing with EAs/ACLs on a daily basis, but it does beg the question: What problems have they solved along the way and why do the various GNU command-line tools they must be using not support EAs or ACLs yet? We did a search for support like this before hacking the CLI tools ourselves, and we found no apparent attempts to grapple with the problems of EA preservation / transfer. This email sent to site_archiver(a)lists.apple.com

1 0