August 2009 - Darwin-dev - lists.apple.com

Re: where does initialization time go?
by Eric Gouriou 28 Aug '09

28 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Startup as measured by DTrace --- ./cold.sh startup.d Total: 10001.723521ms --- It only takes 0.3s to run the initializers according to dyld above. Where did the other 9.7s go? cold.sh is rather simple --- #!/bin/bash Eric and the startup.d script doesn't do much either --- #pragma D option quiet BEGIN { start = timestamp; } /* stop tracing here */ mozilla$target:::main-entry { exit(0); } Just in case, here are the warm startup stats for Firefox --- fastest mac firefox! http://wagerlabs.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Aug 28, 2009, at 12:50 AM, Joel Reymont wrote: My DTrace script measures time to reach a static probe that fires as soon as main is entered. I thought that the only thing that happens before main is the fixup of shared libraries and the running of static initializers. cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote - foreground -P 2" sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" -wZs $1 --- I am afraid that the data collected using this methodology is highly suspect. - firefox-bin gets run as root This is not a normal environment for an application. There are plenty of high-level frameworks which are not quite happy running in that environment, in good part because the root user will not have a typical home directory setup and due to differences in environment variables. This is a known limitation of dtrace -c . Feel free to add your voice via bugreporter.apple.com - A typical launch is not just fork/exec or posix_spawn Even if dtrace -c could magically launch using a regular user, it still isn't a typical launch. dtrace uses fork/exec or posix_spawn (I didn't check which one), while a typical launch (Dock, Finder, open) uses LaunchServices. It isn't all that different but you might as well address both issues together. I'd recommend reworking your script to watch the exec-success of firefox-bin and use a static probe (no need for $target then) to mark the end time. Then you can use a more normal launch method, for example open .../Minefiled.app . - purge(8) is not the way to get a cold launch purge leaves the system in a state that is highly unrealistic. A recently booted system will have a great deal of useful content in the buffer cache that purge throws away. You'll be measuring a launch which involves a lot of page-ins which will not occur on a typical user configuration. In some cases this can point to issues which also occur in a real cold launch, but it is very likely to waste your time. To get a real cold launch, there is no workaround, you have to reboot. It gets worse due to the BootCache capturing IOs during a boot and replaying those IOs (optimized) during the next boot. This recording will last about 60s from the time launchd starts executing, although this detail is subject to change without notice. So you really have to: - boot your system, wait >60s, launch - repeat Discard the first measurement unless you are certain that the initial boot cache playlist did not contain your launch IOs. And yes, it is a pain :-( - Using a SSD to measure cold launch is not representative of a typical user. A cold launch using a SSD is much closer to a warm launch using a good old mechanical hard drive. Optimizing this scenario will not necessarily improve the experience of your users. END { this->total = timestamp - start; printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000); } --- total time: 25.9 milliseconds (100.0%) total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups) total segments mapped: 49, into 5903 pages with 684 pages pre-fetched total images loading time: 5.6 milliseconds (21.7%) total rebase fixups: 149,011 total rebase fixups time: 3.0 milliseconds (11.6%) total binding fixups: 24,932 total binding symbol lookups: 797, average images searched per symbol: 2.3 total binding fixups time: 3.0 milliseconds (11.6%) total bindings lazily fixed up: 45 of 19,109 total init time time: 14.3 milliseconds (55.0%) total images with weak exports: 3 This email sent to site_archiver(a)lists.apple.com

1 0

where does initialization time go?
by Joel Reymont 28 Aug '09

28 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Aug 28, 2009, at 12:33 AM, Jonas Maebe wrote: Do these results mean that I should be digging somewhere else? Here are the gory details... cold startup for Safari cold startup for Firefox Startup as measured by DTrace --- ./cold.sh startup.d Total: 10001.723521ms --- It only takes 0.3s to run the initializers according to dyld above. Where did the other 9.7s go? cold.sh is rather simple --- #!/bin/bash and the startup.d script doesn't do much either --- #pragma D option quiet BEGIN { start = timestamp; } /* stop tracing here */ mozilla$target:::main-entry { exit(0); } Just in case, here are the warm startup stats for Firefox --- fastest mac firefox! http://wagerlabs.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The statistics that you posted previously show that dyld is only taking about 50ms during Firefox' startup. Even if you could reduce that by 90%, it wouldn't really make a lot of difference given the total startup time. So I'm not sure where the low-hanging fruit is here... By low-hanging fruit I meant a measurement approach, not being able to improve results. I thought it would be rather quick to test the shared cache theory and prove it right or wrong. It appears, though, that 1) under no circumstances does update_dyld_shared_cache wants to put XUL in the cache, even if I try to force it, and 2) it doesn't matter. Proving this was quick, thus low-hanging fruit. There's a large difference between cold startup dyld statistics (25-50ms) and warm startup ones (700ms+). There's also a baffling lack of difference between Firefox and Safari cold stats, despite Safari pulling everything from the cache and Firefox using a large number of non-cached dylibs. If so then I cannot explain the difference between startup time as reported by my DTrace script and the dyld statistics. Note that I'm using the Apple 256Gb SSD in my 2.93Ghz unibody MacBook Pro. I thought initially that startup time reported by DTrace should equal the statistics reported by dyld but it's a difference of less than 1s for dyld and almost 10s for dtrace. --- sync && purge && DYLD_PRINT_STATISTICS=1 /Applications/Safari.app/ Contents/MacOS/Safari total time: 696.9 milliseconds (100.0%) total images loaded: 116 (114 from dyld shared cache, 114 needed no fixups) total segments mapped: 5, into 30 pages with 10 pages pre-fetched total images loading time: 204.9 milliseconds (29.4%) total rebase fixups: 1,298 total rebase fixups time: 0.1 milliseconds (0.0%) total binding fixups: 2,476 total binding symbol lookups: 234, average images searched per symbol: 1.6 total binding fixups time: 80.5 milliseconds (11.5%) total bindings lazily fixed up: 3 of 901 total init time time: 411.3 milliseconds (59.0%) total images with weak exports: 1 --- --- total time: 731.2 milliseconds (100.0%) total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups) total segments mapped: 49, into 5903 pages with 684 pages pre-fetched total images loading time: 235.3 milliseconds (32.1%) total rebase fixups: 149,011 total rebase fixups time: 3.7 milliseconds (0.5%) total binding fixups: 24,932 total binding symbol lookups: 797, average images searched per symbol: 2.3 total binding fixups time: 149.9 milliseconds (20.5%) total bindings lazily fixed up: 45 of 19,109 total init time time: 342.2 milliseconds (46.8%) total images with weak exports: 3 --- My DTrace script measures time to reach a static probe that fires as soon as main is entered. I thought that the only thing that happens before main is the fixup of shared libraries and the running of static initializers. cmd="./Minefield.app/Contents/MacOS/firefox-bin -no-remote -foreground -P 2" sync && purge && dtrace -x dynvarsize=64m -x evaltime=exec -c "$cmd" - wZs $1 --- END { this->total = timestamp - start; printf("Total: %u.%06ums\n", this->total / 1000000, this->total % 1000000); } --- total time: 25.9 milliseconds (100.0%) total images loaded: 106 (93 from dyld shared cache, 56 needed no fixups) total segments mapped: 49, into 5903 pages with 684 pages pre-fetched total images loading time: 5.6 milliseconds (21.7%) total rebase fixups: 149,011 total rebase fixups time: 3.0 milliseconds (11.6%) total binding fixups: 24,932 total binding symbol lookups: 797, average images searched per symbol: 2.3 total binding fixups time: 3.0 milliseconds (11.6%) total bindings lazily fixed up: 45 of 19,109 total init time time: 14.3 milliseconds (55.0%) total images with weak exports: 3 This email sent to site_archiver(a)lists.apple.com

1 0

Re: dyld cache: shared_region_roots/Applications.paths
by Steve Checkoway 28 Aug '09

28 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Aug 27, 2009, at 4:33 PM, Jonas Maebe wrote: Amdahl's law strikes again. -- Steve Checkoway "Anyone who says that the solution is to educate the users hasn't ever met an actual user." -- Bruce Schneier _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com The statistics that you posted previously show that dyld is only taking about 50ms during Firefox' startup. Even if you could reduce that by 90%, it wouldn't really make a lot of difference given the total startup time. So I'm not sure where the low-hanging fruit is here... smime.p7s

1 0

Re: dyld cache: shared_region_roots/Applications.paths
by Jonas Maebe 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 28 Aug 2009, at 01:03, Joel Reymont wrote: On Aug 27, 2009, at 11:47 PM, Jean-Daniel Dupas wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… May I ask you why do you think that dylib is the bottleneck ? It's the low-hanging fruit. The statistics that you posted previously show that dyld is only taking about 50ms during Firefox' startup. Even if you could reduce that by 90%, it wouldn't really make a lot of difference given the total startup time. So I'm not sure where the low-hanging fruit is here... This email sent to site_archiver(a)lists.apple.com

1 0

Re: dyld cache: shared_region_roots/Applications.paths
by Mo McRoberts 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Fri, Aug 28, 2009 at 00:03, Joel Reymont<joelr1(a)gmail.com> wrote: > Unfortunately, @executable_path in Firefox dylibs seems to prevent them from > getting into the cache, something I plan to fix using install_name_tool. Surely that’ll then prevent drag-and-drop installation of Firefox, moving it even further _away_ from your average Mac app? It strikes me that, from what you’ve posted here, the dyld issue as compared to Safari is that: - Firefox loads lots of dylibs - Those dylibs are specific to Firefox (and bundled with it) - Safari loads lots of dylibs - Those dylibs are used by lots of other things and part of shared frameworks It makes perfect sense that Safari’s frameworks would be cached while Firefox’s wouldn't, because nearly everything that Safari uses is used by other apps too; this doesn't apply to Firefox. Attempting to get Firefox’s (private) dylibs added to the shared dylib cache would surely be an abuse of mechanism. I dread to think what would happen if lots of app developers thought _their_ apps were important enough to warrant it. I can’t think of another way of putting this, but aren’t you trying to get all the advantages of apps built predominantly using hot, shared, dylibs in an app which eschews much of that to do its own thing? (I'm almost curious enough to go and find out what the playing field is for Camino in this respect, given it's more “native” than FF). By all means, make Firefox faster on the Mac—there are lots of people who prefer it over Safari, after all—but I can’t help but wonder if you might be fighting a losing battle in some avenues. M. -- http://nevali.net _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: dyld cache: shared_region_roots/Applications.paths
by Joel Reymont 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Aug 27, 2009, at 11:47 PM, Jean-Daniel Dupas wrote: It's the low-hanging fruit. --- fastest mac firefox! http://wagerlabs.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… May I ask you why do you think that dylib is the bottleneck ? My impression is that the main function is reached fast enough, but this is the UI initialization that is slow. I will get to that soon. For now, I just want to get Firefox into the dyld cache and see how that affects startup. Fortunately, shared_region_roots/Applications.paths sticks around after a reboot. Unfortunately, @executable_path in Firefox dylibs seems to prevent them from getting into the cache, something I plan to fix using install_name_tool. This email sent to site_archiver(a)lists.apple.com

1 0

Re: setuid root problem
by Terry Lambert 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -- Terry -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Thanks for the response Terry. As I noted, this tool is running from /Library/Frameworks. That's on the system disk, so I expect that its mounted with "suid". :-) Even after I manually set the suid bits (user and group), MoreSecurity still decided to put up the Authorization dialog. This means that the tool was not running as root. That's why I am so confused. To review the statements: - - Is this a nosuid volume? No. This is the system volume, assuredly mounted suid. - - Is this run from a script? No, the tool is run using fork/exec. Thank you for the ideas. ...Duane On Aug 27, 2009, at 3:35 PM, Terry Lambert wrote: We have a product that uses an the older root tool helper process from MoreSecurity. We have installations of the product where the setuid-root bit doesn't seem to work. The program always asks for authentication (outside of the 5 minute timeout period). The system is connected to Active Directory if that makes any difference. If you are using exactly the code in MoreSecurity, it specifically states that calling AuthorizationCopyRights() will result in a dialog because of the default rule in /etc/authorization. If you are running it from an FS with the "nosuid" mount option, then it won't launch SUID, even if those are the permissions (check the output of the "mount" command for the FS in question, but in general, any portable device like a thumb drive or any non-local device like an NFS FS will have "nosuid"). If you are failing to call setuid(geteuid()) in the helper to set your saved, real and effective uid, the suid-ness will not be inherited by the child of the helper. If you are running a script instead of an actual binary, then the suid will always be ignored, unless the interpreter you are running it under is itself suid and capable of enforcing for/against suid- ness of the script (and it takes responsibility for the security when it does this; an example is suidperl). ...Duane -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.10.0 (Build 500) Charset: US-ASCII wsBVAwUBSpcNwkrg9acQ4r2CAQgCfQf/bAFar5Z3C0OpYfTJ5JTwpoLVOQKrJFph 3nkt4GFmFgQM1cSB/93efutxpc1dpL8QilttUoRPerv1aDheCcHHNZx28PqnnpX8 uaTxtdMzmnGFH/k1cW5DUcimP/EfsOUU9parjbB7Wg2YSUBMRJigkm/k3svATfR3 R/BQmvJofJkhVKYcEq/LIweZWExnfGvTLfcLu0GqfMavkrvxTXLTirH6opAAyQXO jqPMWIyQK5J6jLRnGWdgRQj224SRMl1CYd/q7TTAk/nLm1CNUZmyQljwxje8f63+ 7D2r0zcGTR7GevjixolIZizMpGnwiK6NYkrdeJ6i4yPVigSYMboq/A== =pilT -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Can you file a bug report and provide a cut-down sample project that will reproduce the problem? It doesn't need to do anything useful, other than trigger the dialog. On Aug 27, 2009, at 3:50 PM, Duane Murphy wrote: On Aug 27, 2009, at 11:41 AM, Duane Murphy wrote: The tool lives in a framework inside of /Library/Frameworks. On other installations this works fine. On this installation, the setuid bit seems to have no affect. What could be happening here? What situations would cause setuid- root to fail to launch a tool as root? -- Terry This email sent to site_archiver(a)lists.apple.com

1 0

Re: setuid root problem
by Duane Murphy 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Thanks for the response Terry. As I noted, this tool is running from /Library/Frameworks. That's on the system disk, so I expect that its mounted with "suid". :-) Even after I manually set the suid bits (user and group), MoreSecurity still decided to put up the Authorization dialog. This means that the tool was not running as root. That's why I am so confused. To review the statements: - - Is this a nosuid volume? No. This is the system volume, assuredly mounted suid. - - Is this run from a script? No, the tool is run using fork/exec. Thank you for the ideas. ...Duane On Aug 27, 2009, at 3:35 PM, Terry Lambert wrote: > On Aug 27, 2009, at 11:41 AM, Duane Murphy wrote: >> We have a product that uses an the older root tool helper process >> from >> MoreSecurity. >> >> We have installations of the product where the setuid-root bit >> doesn't >> seem to work. The program always asks for authentication (outside of >> the 5 minute timeout period). >> >> The tool lives in a framework inside of /Library/Frameworks. On other >> installations this works fine. On this installation, the setuid bit >> seems to have no affect. >> >> The system is connected to Active Directory if that makes any >> difference. >> >> What could be happening here? What situations would cause setuid-root >> to fail to launch a tool as root? > > If you are using exactly the code in MoreSecurity, it specifically > states that calling AuthorizationCopyRights() will result in a > dialog because of the default rule in /etc/authorization. > > If you are running it from an FS with the "nosuid" mount option, > then it won't launch SUID, even if those are the permissions (check > the output of the "mount" command for the FS in question, but in > general, any portable device like a thumb drive or any non-local > device like an NFS FS will have "nosuid"). > > If you are failing to call setuid(geteuid()) in the helper to set > your saved, real and effective uid, the suid-ness will not be > inherited by the child of the helper. > > If you are running a script instead of an actual binary, then the > suid will always be ignored, unless the interpreter you are running > it under is itself suid and capable of enforcing for/against suid- > ness of the script (and it takes responsibility for the security > when it does this; an example is suidperl). > > -- Terry ...Duane -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.10.0 (Build 500) Charset: US-ASCII wsBVAwUBSpcNwkrg9acQ4r2CAQgCfQf/bAFar5Z3C0OpYfTJ5JTwpoLVOQKrJFph 3nkt4GFmFgQM1cSB/93efutxpc1dpL8QilttUoRPerv1aDheCcHHNZx28PqnnpX8 uaTxtdMzmnGFH/k1cW5DUcimP/EfsOUU9parjbB7Wg2YSUBMRJigkm/k3svATfR3 R/BQmvJofJkhVKYcEq/LIweZWExnfGvTLfcLu0GqfMavkrvxTXLTirH6opAAyQXO jqPMWIyQK5J6jLRnGWdgRQj224SRMl1CYd/q7TTAk/nLm1CNUZmyQljwxje8f63+ 7D2r0zcGTR7GevjixolIZizMpGnwiK6NYkrdeJ6i4yPVigSYMboq/A== =pilT -----END PGP SIGNATURE----- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: dyld cache: shared_region_roots/Applications.paths
by Jean-Daniel Dupas 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Le 27 août 2009 à 22:20, Joel Reymont a écrit : Applications.paths does stick around after a reboot. Is it safe for an application to add itself there? Thanks, Joel On Aug 27, 2009, at 12:06 AM, Joel Reymont wrote: How do entries get into the Applications.paths file below? Thanks, Joel --- --- fastest mac firefox! http://wagerlabs.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/devlists%40shadowlab.org This email sent to devlists(a)shadowlab.org _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… May I ask you why do you think that dylib is the bottleneck ? My impression is that the main function is reached fast enough, but this is the UI initialization that is slow. Even on a fast machine, it's possible to see a short period of time during startup where the Application menu is set to Firefox, but the remaining items are not visible. Can it be edited manually or is there a tool to add and remove entries? cat /var/db/dyld/shared_region_roots/Applications.paths /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder /System/Library/CoreServices/loginwindow.app/Contents/MacOS/ loginwindow /System/Library/Frameworks/ApplicationServices.framework/Frameworks/ CoreGraphics .framework/Resources/WindowServer /Applications/Safari.app/Contents/MacOS/Safari /Applications/Mail.app/Contents/MacOS/Mail /Applications/QuickTime Player.app/Contents/MacOS/QuickTime Player /Applications/iChat.app/Contents/MacOS/iChat /bin/sh /bin/bash This email sent to site_archiver(a)lists.apple.com

1 0

Re: setuid root problem
by Terry Lambert 27 Aug '09

27 Aug '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com We have a product that uses an the older root tool helper process from MoreSecurity. We have installations of the product where the setuid-root bit doesn't seem to work. The program always asks for authentication (outside of the 5 minute timeout period). The tool lives in a framework inside of /Library/Frameworks. On other installations this works fine. On this installation, the setuid bit seems to have no affect. The system is connected to Active Directory if that makes any difference. What could be happening here? What situations would cause setuid-root to fail to launch a tool as root? -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Aug 27, 2009, at 11:41 AM, Duane Murphy wrote: If you are using exactly the code in MoreSecurity, it specifically states that calling AuthorizationCopyRights() will result in a dialog because of the default rule in /etc/authorization. If you are running it from an FS with the "nosuid" mount option, then it won't launch SUID, even if those are the permissions (check the output of the "mount" command for the FS in question, but in general, any portable device like a thumb drive or any non-local device like an NFS FS will have "nosuid"). If you are failing to call setuid(geteuid()) in the helper to set your saved, real and effective uid, the suid-ness will not be inherited by the child of the helper. If you are running a script instead of an actual binary, then the suid will always be ignored, unless the interpreter you are running it under is itself suid and capable of enforcing for/against suid-ness of the script (and it takes responsibility for the security when it does this; an example is suidperl). This email sent to site_archiver(a)lists.apple.com

1 0