January 2009 - Darwin-dev - lists.apple.com

Re: cache clearing from keeping code and data in the same segment
by Joel Reymont 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=k/Rwwg0ES/OTL65Dm6N9UJ+Kmoe/z82zbfU7/5u4B1I=; b=Klkirc2zDQF569bkN3i3iq5OVho1QRIG6ndBJ9wIOX5tLRk7Q3sTJDsBpe+zNKZUD7 x1agg67Xd7AHM6pqJRpDkznrCUSzR7/XXtF1UMpU3Ig7kVf/jY/8fMNK9kZi/b/5hrY9 Riba2Ujw4x4dPwI+wr98PchOOT329+PXT2eao= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=H54qeXdczWjw3KGT7uh8ohuKiCr9vq/0lt8oOC6Dj8WpbyLmRm8KooVfzMX8svFMCG j71oduCDNkl2vIGJtoB90+uLFLWUnxhQFXGWBwfJURBEUrYfZGsowDitGh7pvZHX5lZo yOmW1Oqe0wTBHmbgG9DPQJU8EPONVLgebeg8s= On Jan 19, 2009, at 12:25 PM, Jonas Maebe wrote: --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Keeping code and read/write data close to each other (putting them in the same segment may qualify) may result in the icache getting invalidated on data writes (to deal with self-modifying code), [...] Such flushing, if any, would performed by the cpu though, the OS has no influence over that. So you are saying that this is a function of the CPU as opposed to the OSX itself? I am dealing with self-modifying code (Forth), although I'm willing to trade certain things for performance. This email sent to site_archiver(a)lists.apple.com

1 0

Re: cache clearing from keeping code and data in the same segment
by Jonas Maebe 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 19 Jan 2009, at 11:36, Joel Reymont wrote: I very much doubt that. Why do you think that is the case? Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Keeping code and data in the same segment apparently results in the clearing of the instruction cache on every data access. Keeping code and read/write data close to each other (putting them in the same segment may qualify) may result in the icache getting invalidated on data writes (to deal with self-modifying code), but I can't imagine how or why memory reads would result in icache invalidations (read-only data is common in code segments on many platforms). Such flushing, if any, would performed by the cpu though, the OS has no influence over that. Note that this would most likely be x86-specific. On e.g. the PPC you have to manually invalidate the icache when using self-modifying code. Also note that in general it's a bad idea to put code and data close to each other (even if no automatic flushing whatsoever happens), because it pollutes both the icache and dcache: the icache will partly contain data (which is never fetched from the icache) and the data cache will partly contain instructions (which are never fetched from the dcache), thereby reducing the effectiveness of both. This email sent to site_archiver(a)lists.apple.com

1 0

Re: cache clearing from keeping code and data in the same segment
by Joel Reymont 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=DY+r8O6DXbdvitdwzoV5xrvWGpjV82HJChpO77YBi0o=; b=pQ5wFXZO1CxBOKHkf4SZnybZPcjhwGBx+EtCuPt/k5fT2g4kKrleR8BvrNDpa7HD5E XfRWNffokJgB6nfYZSLXWyJaCivy7OE0J0rtbeoudktaJ4MQkIrcwhhmb6tjuP8xOGbS IGhhvxZUypELkjg3iEmUvQPBAW1Dd0ynXU4n8= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=Hi2bikn6cyidao8l9bc4rxHsy3NOu8mW6vggICxShCson3rVDuo7DbV4EYO4BO3s9I TYTVpdRV2Xwsx/7w7wp+9d1X6BO7qxMfPV/iRopHJqqUMF1gcdVIIQWUJKVXM8061DFY WfU+NrsGNjeUFS4wETGKm53utDeizAJ4yO2q4= Eli, On Jan 19, 2009, at 11:48 AM, Eli Bach wrote: I have my Forth running on the Mac, now it's time to improve it. Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I would guess that information like this isn't explicitly listed/ documented somewhere, other than perhaps by reading the code in Apple's developer tools (gcc/linker/etc), maybe in changelogs when Apple pushes their changes upstream. Wouldn't this be in the kernel somewhere? I'm trying to figure out where exactly. This is the kind of thing that would explain why Apple's engineers steered you towards at least using their linker. The linker is not an issue, I can split code and data into different segments just as well. This was something that someone else pointed out and I thought I'd get as much information on it as possible. This email sent to site_archiver(a)lists.apple.com

1 0

Re: cache clearing from keeping code and data in the same segment
by Eli Bach 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:subject :mime-version:date:references:x-mailer; bh=WR3RAF7KDhOQ9I9JVv4TW0B/fMC2513AXJlSk33eHzw=; b=he8upEfMGzMtIjhchGTVvZ/jT/2+OSIJSQdU8Ny9Lx9VOAx1chq01hDaFMLoDuuy4W 8gbqSkWM72E0JvRmQDxuSRmeZRy7UH2aTYZeUVipFdubca4RHS3/og5EegHCg0Ogy6uo EDRYebbX92kN+qxg5me0q4E1EmoWTevo2jknk= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:subject:mime-version:date:references :x-mailer; b=TrwuPio1zoi8h/zcPelYdToS7/bgsYIFdw5n+fesydZHGCVIisHaOTfLNN0n+d6kZf MftiUW9wvxrQvsV3ayBC6Dq0zNTpkaQ8Ruk3leRXQsvNEWYnHPYzui7FYuq1US5Pi+Vi QCcGASCzR9EyEh1x44tKJOM5oiJHlQeCi0HB0= On Jan 19, 2009, at 3:14 AM, Joel Reymont wrote: Eli, On Jan 19, 2009, at 11:11 AM, Eli Bach wrote: Where's this protection mechanism explained, though? Eli _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Maybe so your app doesn't get hacked by means of writing to the data segment, and then getting that data executed somehow. Thanks, Joel Are you looking for why your custom-written executable is running slow (just from looking at your previous posts to darwin-dev)? I would guess that information like this isn't explicitly listed/ documented somewhere, other than perhaps by reading the code in Apple's developer tools (gcc/linker/etc), maybe in changelogs when Apple pushes their changes upstream. This is the kind of thing that would explain why Apple's engineers steered you towards at least using their linker. Maybe it would be easier to port it to your build platform instead of trying to reverse- engineer it's functionality into your code. And it's not just what's in there today, it's what Apple updates tomorrow as well, as that will wind up biting you. But don't take my word for any of this, as I have no actual knowledge of the reasoning Apple used to decide to implement this 'feature'. This is just something I came up with that seems plausible. The only 'hacking' of applications that I have done involved patching traps on MacOS 9. This email sent to site_archiver(a)lists.apple.com

1 0

Re: cache clearing from keeping code and data in the same segment
by Joel Reymont 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=QoLyTPmZWgk1zo3A/64BEo5XXIBX//qEZ8teo+Arhjc=; b=lZXyl8WDQ9u+HUn2Kzn+H5knUoeR0gHN7pC7CqsT8n/Jsnyjf7qxuAl2RmEfAKU1S4 Ji6f5TciNTz4cO01hhxNKEGV0N3LjnnrwJFiJNaTHktBZ35zabfSIZxkKPNFUX94oYTZ LDI+Lm6QdAsukFZEUPn+rA8sVEyRyQOIQEpLE= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=p+NtvTKAIAueDxxxNppxUaOs9Yvt/xIlGK1Qt+A8oUSO2oSCWrw9I/qFyDALWr21Kh NfjLsABlWNGK2IaMbnbw4IpU/jIe2yEg14vC/6GmQVIvNY8Zhf+BVSiH9LOEIVcM7kpQ p2sX95ID2G3osBjxyHsObH0gQMqcp7kWs3BRc= Eli, On Jan 19, 2009, at 11:11 AM, Eli Bach wrote: Where's this protection mechanism explained, though? Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Maybe so your app doesn't get hacked by means of writing to the data segment, and then getting that data executed somehow. This email sent to site_archiver(a)lists.apple.com

1 0

Re: cache clearing from keeping code and data in the same segment
by Eli Bach 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:subject :mime-version:date:references:x-mailer; bh=vnO6u/RPpbg0fczgorJLlD8VpFwis9pQkXHPtZlvz+A=; b=KssZqjul2iVwTzaWxuvJpCIjDjQ798oEEaxGL35bk6wOux920hTqDvQJklsHGosMlP 0nHQ1pP470HVZgv80fB8i7ceZZM6//fdZbkpxiaed+nsVgLlfYiwHibBLTMNP0mmar2K c6vrEtFeqLWetVeX5vUDOiX4Dy3dMgknEW/oc= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:subject:mime-version:date:references :x-mailer; b=WAJSH4SUN6N4qVaO6ZsOxCgNU/qpOUrn795cVyiaFwnI9ZxTtK8Jb08xpm5ZgXkRe2 At3OgaW7CY7kK4uO12i04nuaqtyB7/aURmuFNwy5vtQpj4oULlvlAhrjeyZqPQtmpJvb 83c0tiq8AFzA5MFL3N6nkOX3Cj6ZtqkkhZks0= On Jan 19, 2009, at 2:36 AM, Joel Reymont wrote: Would someone kindly explain why or point me to an explanation? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Keeping code and data in the same segment apparently results in the clearing of the instruction cache on every data access. Thanks, Joel Maybe so your app doesn't get hacked by means of writing to the data segment, and then getting that data executed somehow. This email sent to site_archiver(a)lists.apple.com

1 0

cache clearing from keeping code and data in the same segment
by Joel Reymont 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :content-type:content-transfer-encoding:mime-version:subject:date :x-mailer; bh=J2wsgp0Wav0wS5+kGxfpbDPyyXxI0azAFpEi2BVKyiU=; b=FqZa7bCrS07y8+o7z30S7tpky31DqdIE3Q1fE++D8Qdji9FoIXlVQNDqjhHIBvVr2h Sa558jkoXyPn17AvLfj9lp9Lv5XcT9N5EVNAqkAFFAX1rgjpDQ4Uum7sD5Qxkw3xpbU7 njYiTR0S5jfoHPZBzua40K6DNFCmtYG5TUkL8= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:content-type:content-transfer-encoding :mime-version:subject:date:x-mailer; b=j4tIlUOcVOmmiCX6kGyfo4UNZVLk00tdIaao2+a4TT4OliOOlmbSo+c7j9+j4+wytI 2ULtcB9BaFF8mQbKc0j2Hv1juNlO3gf/cM2BbRY5+N7m+NjKDssJvU5UNQ3T8/xiSQzG vRcri1pL2ZKJHVHbsfX0iz59He66MvvYYdYIs= Would someone kindly explain why or point me to an explanation? Thanks, Joel --- http://twitter.com/wagerlabs _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Keeping code and data in the same segment apparently results in the clearing of the instruction cache on every data access. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Why is virtual memory so high? (crash)
by Mario Emmenlauer 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 2.0.0.18 (Macintosh/20081105) Me again, Its a 32bit application that should use all real memory possible (4GB), but it crashes after reserving only 2.5GB real mem when the virtual mem reaches 4GB. The crash is logical, but why does it start with 1.8GB vmem initially? On ppc it reserves only 350MB vmem. I used "heap" and "MallocDebug", both show only the ~40MB real memory that should be expected. Are there better/other tools I could try to inspect where the additional vmem is reserved? Best, Mario _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Mario Emmenlauer wrote: This email sent to site_archiver(a)lists.apple.com

1 0

Why is virtual memory so high? (crash)
by Mario Emmenlauer 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 2.0.0.18 (Macintosh/20081105) Hi, I have a large C/C++ project with thousands of lines of code and many thirdparty-libraries. When it startes up, it reserves around 1.8GB of virtual memory (40MB real memory), and I don't know why. Its a 32bit application that should use all real memory possible (4GB), but it crashes after reserving only 2.5GB real mem when the virtual mem reaches 4GB. The crash is logical, but why does it start with 1.8GB vmem initially? On ppc it reserves only 350MB vmem. In case thats relevant, the app opens some files and does some 3D graphics. It also uses QT amongst many other libs. Is that helpful? Best, Mario BTW: I don't completely understand who reserves the vmem. Is it my app, or is it the OS? Can I trace it? _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: bootstrap_register replacement
by Damien Sorresso 19 Jan '09

19 Jan '09

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Task special ports are inherited across fork(2). _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com On Jan 15, 2009, at 2:19 PM, Ed Wynne wrote: On Jan 15, 2009, at 2:40 PM, Damien Sorresso wrote: 0. In parent, create a new port (Port A) to which you hold a receive right. 1. Insert a send right into that port. 2. Cache your current task bootstrap port. 3. Set the task bootstrap port to Port A. 4. fork(2) / exec(2) your child. 5. In the parent, set your task bootstrap port back to the cached port. 6. Have the child send a message to the parent (via its bootstrap port) requesting all the information it needs, perhaps including the parent's bootstrap port. 7. Optionally, have the child reset its bootstrap port to the parent's bootstrap port. #7 should not be optional. Frameworks and other system services may depend on the bootstrap port being what it claims. It should mandatorily be set back, ASAP. Indeed. Though it should be noted that this isn't really that urgent if you create a bootstrap subset. Though I don't really recommend that. Mac OS X is trending toward flat per-user namespaces. -- Damien Sorresso BSD Engineering Apple Inc. smime.p7s

1 0