October 2006 - Darwin-dev - lists.apple.com

Re: Problem when compiling Xprt of X11R7.1
by Ben Byer 22 Oct '06

22 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 21, 2006, at 2:27 AM, Peter Dyballa wrote: libtool.m4 ltdl.m4 pkg.m4 xaw.m4 xorg-macros.m4 xorgversion.m4 xtrans.m4 I think I found the cause of your problems compiling Xprt: I have had luck with the following configure flags: Good luck! Ben _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I suggest you try upgrading to the latest libtool from It's being made ... It turns out that libtool was not the cause of the problem you are seeing; however, I suggest you upgrade anyway if you would like to build Universal binaries. (Otherwise, it may not matter.) If you install that libtool in /usr/local/bin, then run these commands in your xorg-xserver tree: mkdir -p /usr/local/share/aclocal What is this empty directory good for? If you make this directory and set ACLOCAL as below before you make all of the preqrequisites, some of them will place files there that you will need if you ever want to re-run autoconf on any of the packages. (Specifically, in mine I have these files: ... but again, this is not important if you do not re-run autoconf on any files.) export LIBTOOLIZE=/usr/local/bin/libtoolize Does this also make sure that the new /usr/local/bin/libtool will be found? My situation is a bit complicated having more up-to-date tools (than Apple delivers) in /sw/lib/odcctools/bin, /sw/bin, and finally /usr/local/bin too! Yes. This is why I finally gave in and made an /opt/xorg and placed new versions of everything there... otherwise, I had too much trouble reproducing build.sh is so nice and easy, compiles continuously for hours – which allows me also to create a custom script that only makes this or that and then some targets in each group are commented ... Yes, although that gets frustrating when you have to wait for everything to recompile just for one small change in one package. At least for the Xserver, I build it separately since there are a number of patches and special steps needed. When I use xorg-server-X11R7.1-1.1.0 without any patches, and try to compile Xprt, I get this: /bin/sh ../libtool --tag=CC --mode=link gcc -D_BSD_SOURCE - DHAS_FCHOWN -DHAS_STICKY_DIR_BIT -I/opt/xorg/include -I/opt/local/ include/freetype2 -I../include -I../include -I../Xext -I../ composite -I../damageext -I../xfixes -I../Xi -I../mi -I../miext/ shadow -I../miext/damage -I../render -I../randr -I../fb -g -O2 -o Xprt -L.. Xprt-attributes.o Xprt-AttrValid.o Xprt-ddxInit.o Xprt- Init.o Xprt-mediaSizes.o Xprt-Oid.o Xprt-spooler.o Xprt-Util.o Xprt- miinitext-wrapper.o Xprt-dpmsstubs-wrapper.o -L/opt/xorg/lib -L/opt/ local/lib -lX11 -lXfont -lm -lfontenc -lfreetype -lz -lXdmcp - lXau ../dix/libdix.la ../xkb/libxkb.la ../xkb/libxkbstubs.la ../ Xext/libXext.la ../dbe/libdbe.la ../XTrap/libxtrap.la ../record/ librecord.la ../render/librender.la ../composite/libcomposite.la ../ randr/librandr.la ../Xi/libXi.la ../xfixes/libxfixes.la ../damageext/ libdamageext.la ../Xi/libXi.la ../mi/libmi.la ../miext/damage/ libdamage.la ../miext/cw/libcw.la ../os/libcwrapper.la ../os/libos.la ps/libps.la raster/libraster.la pcl/libpcl.la pcl-mono/libpcl.la ../ mfb/libmfb.la ../mi/libmi.la ../cfb32/libcfb32.la ../cfb/libcfb.la ../ Xext/libXext.la -lm [...] usr/bin/ld: multiple definitions of symbol _serverGeneration /opt/xorg/lib/libXfont.dylib(miscutil.o) definition of _serverGeneration ../dix/.libs/libdix.a(globals.o) definition of _serverGeneration in section (__DATA,__data) /usr/bin/ld: multiple definitions of symbol _Xalloc /opt/xorg/lib/libXfont.dylib(miscutil.o) definition of _Xalloc ../os/.libs/libos.a(utils.o) definition of _Xalloc in section (__TEXT,__text) [...] The problem here is in the ordering of the dynamic libraries. If you move them ('-lX11 -lXfont -lm -lfontenc -lfreetype -lz -lXdmcp - lXau') to the end of that line, after libXext.la, it will compile correctly. Some of the other darwin-dev members (I will let the guilty parties step forward, if they so choose!) have come up with some patches that make the Xserver *almost* work -- it compiles, even runs!, but there is still a lot left to do. I don't know which patch fixes the above problem, but I put the patches up here: http://homepage.mac.com/ ben.byer/.Public/xorg-darwin-patches.tar.bz2 --with-mesa-source=/opt/xorg/build/extras/mesa/Mesa-6.5 --enable- debug --enable-xorg --enable-quartz This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem when compiling Xprt of X11R7.1
by Justin C. Walker 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 21, 2006, at 15:21 , Peter Dyballa wrote: Am 21.10.2006 um 21:02 schrieb Justin C. Walker <justin(a)mac.com>: If you mean this literally, I think it should read ACLOCAL="aclocal -I/sw/share/aclocal -I/usr/local/share/aclocal" (no space between -I and pathname). Thanks for the correction. Justin -- Justin C. Walker, Curmudgeon-At-Large, Director Institute for the Enhancement of the Director's Income -------- The path of least resistance: it's not just for electricity any more. -------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… No, Justin. The info file explains: Oops. I didn't look into that fully; I figured the flags would be passed as-is. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem when compiling Xprt of X11R7.1
by Peter Dyballa 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 21.10.2006 um 21:02 schrieb Justin C. Walker <justin(a)mac.com>: If you mean this literally, I think it should read ACLOCAL="aclocal -I/sw/share/aclocal -I/usr/local/share/aclocal" (no space between -I and pathname). No, Justin. The info file explains: `-I DIR' Add the directory DIR to the list of directories searched for `.m4' files. and when I set ACLOCAL to some -Ivalue I get: aclocal: unrecognized option -- `-I/usr/local/share/aclocal' -- Greetings Pete "A lot of people mistake a short memory for a clear conscience." Doug Larson _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: RAM Cache vs the Pageout Demon
by Michael Smith 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 21, 2006, at 12:02 PM, <matchmovie(a)yahoo.com> wrote: You are cannibalising your own pages, so this won't help you. In both cases, this can reduce the library's working set substantially. Don't make assumptions. Measure first. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… My commercial app has a ram cache system for image sequence playback; images can be tens of megabytes and there can be gigabytes of them. On OS X, when the RAM cache is large, it looks like the darwin pageout demon is activating prematurely, paging out part of the RAM cache preemptively to meet its free-space target, even though there is no competing demand. Consequently, the app thrashes catastrophically to its knees. The Darwin free page reserve target is actually very small; a few percent of physical RAM in a large configuration. The details of system activity seems to matter---I have two different possible display paths (CG and GL), one of which transiently allocates and frees an extra image buffer. The path that allocates/frees causes the machine to thrash at RAM cache sizes that the other path is unaffected by, even though the total consumption is about the same. I need to better understand/control this issue and develop countermeasures. The problem is simply that you are caching too much data; you need to understand the actual footprint of your working set so that you can cache less. I think OS X's preemptive pageout to maintain minimum free list (see http://developer.apple.com/documentation/Performance/Conceptual/ ManagingMemory/Articles/AboutMemory.html) is counterproductive, vs waiting for actual free-space demand. You do, eh? Well, let's assume just for the moment that the kernel needs those free pages in order to make forward progress when an application like yours thinks it needs every last page in the system. I'm investigating what the free-space target is (how it is set) and whether it might be changed. The free page reserve levels are not adjustable; they are already tuned to provide as many pages as possible to applications. This avenue will not yield you any useful results. The minimal fix is clearly to ratchet down the maximum permissible ram cache size well under (50%?) the machine's actual physical size. Though this is undesirable, it probably is better than letting the UI go unusable. There are several better techniques that you might consider applying. First and foremost though, you need to understand your actual working set across the two display paths you have. Once you have this in hand, you can more realistically assess the amount of free memory required to display an image before bringing it into your cache, and thus whether the sum of cached images poses a threat to your working set. Or, can the task priority be raised---will that give my app priority competing for free and inactive pages? Even if OS X wants to grab some pages to maintain its free list, I need it to grab them from inactive processes, not the ones it thinks are inactive in my app. If those tasks are truly more inactive than you, you will already have stolen all the pages from them that you can. I'm thinking that "wiring" 80% of the RAM into my app is unwise. Without understanding your application, it's difficult to see how you can justify holding that much unprocessed image data in memory. It sounds like you need to start by more carefully examining how you cache images, and what you think you're trying to achieve. Other possibilities include some sort of RAM-tickling demon. If you do that, be aware that laughing RAM is slower than RAM that is focussed on the job at hand. Suggestions welcome. In a followup message, you indicated that you are using third-party image loading libraries, and by this I infer that your interface to them is along the lines of "render this filename into a buffer". This robs you of a good deal of control over your working set size; you can pessimistically assume that the library will consume pages equal to 2x the size of the file at a minimum. 1x will go straight to the inactive queue, 1x will go to the active queue. In addition, it will require pages to back the buffer it renders into, plus any others required by intermediate datastructures. If you are willing to spend some time analysing these libraries (really mandatory, if you care about performance), you should examine how they read files. You may find a couple of easily-optimised cases: - The file is opened, read into a buffer, then closed. You can easily replace this with either a) mmap, or b) a read with F_NOCACHE set. - The file is processed as a stream. This is another candidate for F_NOCACHE, if the stream is not rewound. You also have poor realtime control over this sort of interface, which is a problem given the only reason I can think of you wanting to have many images buffered; I assume you want to be able to flip rapidly through a set. Given that you can't cache all of them, it strikes me as being more sensible to cache many fewer and begin your image flip speed reduction earlier, with a progressive slowdown while you are reading ahead/behind. Ultimately your long-sequence flip rate is bounded by how fast you can load images, not how many you can hold in memory, so wasting large amounts of memory with a partial subset of the image set is a false optimisation. Another point to consider; you say "images can be tens of megabytes". Do you display images at full resolution, or are you scaling them for display? If the latter, consider scaling at render time and just keep the display-size version in memory. If you support resizing, as long as you're not rescaling for live resize you can just catch the first window resize and re-render the full resolution version so that when the resize is over you're (nearly) ready to redraw. Likewise, if your usage model is such that zooms on images away from initial display size is rare, you can re-render on demand as and when zooming is required. This obviously trades a little lag on first resize/zoom vs. constant excess memory pressure while flipping, so you'll have to tune accordingly. The bottom line to all of this is that first you must understand what your application is doing. Once you understand the current state of things, you can consider what it actually needs to do, and how to get from where you are to where you need to be. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem when compiling Xprt of X11R7.1
by Justin C. Walker 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com A quick note: If you mean this literally, I think it should read ACLOCAL="aclocal -I/sw/share/aclocal -I/usr/local/share/aclocal" (no space between -I and pathname). Justin -- Justin C. Walker, Curmudgeon at Large Institute for the Absorption of Federal Funds ----------- My wife 'n kids 'n dogs are gone, I can't get Jesus on the phone, But Ol' Milwaukee's Best is my best friend. ----------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Oct 21, 2006, at 02:27 , Peter Dyballa wrote: Am 21.10.2006 um 09:09 schrieb Ben Byer: [snip] export ACLOCAL="aclocal -I /usr/local/share/aclocal" I used: ACLOCAL="aclocal -I /sw/share/aclocal -I /usr/local/share/ aclocal" This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem when compiling Xprt of X11R7.1
by Peter Dyballa 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 21.10.2006 um 09:09 schrieb Ben Byer: It's being made ... What is this empty directory good for? autoreconf -fvi ./configure (......) make (...) -- Greetings Pete A morning without coffee is like something without something else. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… What version of libtool are you using? Libtool 1.5.22 from Fink. I compile in GNU Emacs and set for compilation a particular environment with a PATH setting that makes sure that most tools (all I know of) are up-to-date and come from Fink. Libfreetype is a bit outdated, but XOrg Wiki states that an old libfreetype should be used. I suggest you try upgrading to the latest libtool from If you install that libtool in /usr/local/bin, then run these commands in your xorg-xserver tree: mkdir -p /usr/local/share/aclocal export ACLOCAL="aclocal -I /usr/local/share/aclocal" I used: ACLOCAL="aclocal -I /sw/share/aclocal -I /usr/local/share/ aclocal" export LIBTOOLIZE=/usr/local/bin/libtoolize Does this also make sure that the new /usr/local/bin/libtool will be found? My situation is a bit complicated having more up-to-date tools (than Apple delivers) in /sw/lib/odcctools/bin, /sw/bin, and finally / usr/local/bin too! Build.sh is so nice and easy, compiles continuously for hours – which allows me also to create a custom script that only makes this or that and then some targets in each group are commented ... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Problem when compiling Xprt of X11R7.1
by Ben Byer 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Oct 20, 2006, at 7:49 AM, Peter Dyballa wrote: Hello! http://finch.finkproject.org/~pogma/libtool/libtool-1.5.23a.tar.gz mkdir -p /usr/local/share/aclocal export ACLOCAL="aclocal -I /usr/local/share/aclocal" export LIBTOOLIZE=/usr/local/bin/libtoolize autoreconf -fvi ./configure (......) make (...) It will still not compile cleanly, but I think that will help. Ben _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… When on Mac OS X 10.4.8 Xprt is built by GCC 4.0.1 from dynamic and static libraries I get one message /sw/lib/odcctools/bin/ld: multiple definitions of symbol _serverGeneration /usr/X11R7/lib/libXfont.dylib(miscutil.o) definition of _serverGeneration ../../dix/.libs/libdix.a(globals.o) definition of _serverGeneration in section (__DATA,__data) What version of libtool are you using? I suggest you try upgrading to the latest libtool from If you install that libtool in /usr/local/bin, then run these commands in your xorg-xserver tree: This email sent to site_archiver(a)lists.apple.com

1 0

Re: RAM Cache vs the Pageout Demon
by Russ 21 Oct '06

21 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=IrdrdfA0RQme9hn0m3z1jWfKP+FU/09Yp8sSGzl6kpiI+hfrf4gfsLzOE0Ff+uzrB4KyfUWXHA2dI10h/CqDrMxU4hCTTf9pasR9qNXXxwXt0IbgDifNuvCVRYBoIL2wOK7dBoqhYpGGUgBeCBmNyBv5LDtOpsoh503u+BYwMXM= ; They can be based on disk data. The disk-based source data can come from any number of file-reading libraries, so turning off caching on a per-file basis isn't possible. It's certainly conceivable there could be a conflict between my RAM cache and the system file cache. In general, given a choice between paging out an apparently-inactive page from my process, or dropping a page from the file cache for a file requested by my app, it has to drop the (read-once!) file cache page 100% of the time. --- Paul Ripke <stix(a)stix.id.au> wrote: > On Fri, Oct 20, 2006 at 12:55:47PM -0700, Russ wrote: > > My commercial app has a ram cache system for image sequence playback; > images > > can be tens of megabytes and there can be gigabytes of them. On OS X, when > the > > RAM cache is large, it looks like the darwin pageout demon is activating > > prematurely, paging out part of the RAM cache preemptively to meet its > > free-space target, even though there is no competing demand. Consequently, > the > > app thrashes catastrophically to its knees. > > I assume all these images are read in from disk - since you're doing > your own caching, are you using F_NOCACHE with fcntl(2)? It may be > that the aggressive file caching done by the kernel is fighting > with your cache for RAM. > > Cheers, > -- > Paul Ripke > Unix/OpenVMS/TSM/DBA > I love deadlines. I like the whooshing sound they make as they fly by. > -- Douglas Adams > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: RAM Cache vs the Pageout Demon
by Paul Ripke 20 Oct '06

20 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Mail-followup-to: darwin-dev(a)lists.apple.com, Russ <matchmovie(a)yahoo.com> User-agent: Mutt/1.4.2.1i On Fri, Oct 20, 2006 at 12:55:47PM -0700, Russ wrote: > My commercial app has a ram cache system for image sequence playback; images > can be tens of megabytes and there can be gigabytes of them. On OS X, when the > RAM cache is large, it looks like the darwin pageout demon is activating > prematurely, paging out part of the RAM cache preemptively to meet its > free-space target, even though there is no competing demand. Consequently, the > app thrashes catastrophically to its knees. I assume all these images are read in from disk - since you're doing your own caching, are you using F_NOCACHE with fcntl(2)? It may be that the aggressive file caching done by the kernel is fighting with your cache for RAM. Cheers, -- Paul Ripke Unix/OpenVMS/TSM/DBA I love deadlines. I like the whooshing sound they make as they fly by. -- Douglas Adams _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

RAM Cache vs the Pageout Demon
by Russ 20 Oct '06

20 Oct '06

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=oIO4j1jFHEaQGEMA9DLWmCxV10a/9YUeVBQ9go3MUo5OgWpCObOFYH55kcniOe3Mj2o7wbFGov3EEOze/B8dmtTgR/aAGhSYWKttwo6IOiMU4fpvmst9BqZuf/P2FeH9G6cLvDUMk97SsGPIkesT3i3PHhTowwMepmnSUXMhi8w= ; My commercial app has a ram cache system for image sequence playback; images can be tens of megabytes and there can be gigabytes of them. On OS X, when the RAM cache is large, it looks like the darwin pageout demon is activating prematurely, paging out part of the RAM cache preemptively to meet its free-space target, even though there is no competing demand. Consequently, the app thrashes catastrophically to its knees. The details of system activity seems to matter---I have two different possible display paths (CG and GL), one of which transiently allocates and frees an extra image buffer. The path that allocates/frees causes the machine to thrash at RAM cache sizes that the other path is unaffected by, even though the total consumption is about the same. I need to better understand/control this issue and develop countermeasures. I think OS X's preemptive pageout to maintain minimum free list (see http://developer.apple.com/documentation/Performance/Conceptual/ManagingMem…) is counterproductive, vs waiting for actual free-space demand. I'm investigating what the free-space target is (how it is set) and whether it might be changed. The thing I've found so far is VM_V_FREE_TARGET in http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/sys… If there's a command-line version that might be used to examine/change these, that would be nice to know. The minimal fix is clearly to ratchet down the maximum permissible ram cache size well under (50%?) the machine's actual physical size. Though this is undesirable, it probably is better than letting the UI go unusable. Or, can the task priority be raised---will that give my app priority competing for free and inactive pages? Even if OS X wants to grab some pages to maintain its free list, I need it to grab them from inactive processes, not the ones it thinks are inactive in my app. I'm thinking that "wiring" 80% of the RAM into my app is unwise. Other possibilities include some sort of RAM-tickling demon. Suggestions welcome. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0