September 2007 - Darwin-dev

Source for dwarfdump/dsymutil?
by Timothy J.Wood 25 Sep '07

25 Sep '07

site_archiver(a)lists.apple.com Delivered-To: Darwin-dev(a)lists.apple.com Thanks! -tim _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… dwarfdump apparently comes from a project called 'dwarfutils-33', but this doesn't seem to be in the latest sources. Was this omitted intentionally? If there somewhere I can download a copy? google indicates that gcc patches weren't buildable for a while w/ o "a newer dwarfutils which don't exist outside of Apple", but it isn't clear if the dependency was removed or whether the source was released somewhere. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Logging physical/virtual memory usage of my app
by Wade Tregaskis 25 Sep '07

25 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com I have 24/7 that has a slow memory leak... I would like regularly log the real Total memory size(VSIZE ) that my app is using. What's the best way to retrieve this information from the System? You can also use top in logging mode, with a syntax something like: top -l <maxiterations> -S -s <interval> -U <username> -c e Wade _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Well, naturally there's tools at the end-user level like ObjectAlloc, MallocDebug and Shark, but if your app is frequently allocating & deallocating memory, they're probably not going to be practical to run over the time spans you'd like. So, you can roll your own recorder in several ways. It depends, naturally, on how you want to do so. "vmmap" will give you all sorts of numbers, and could be logged periodically using a script, and parsed and played with and all that. (there's also its cousin vm_stat which provides general system-wide stats, if that's of interest) There's no way I'm aware of to have it just look at one process, though you could pass the output through grep easily enough to filter it. If you want the information programmatically, you can include <mach/ mach.h> and then poke with such functions as task_info(), to get the task_basic_info for your chosen task (you can get the necessary task_t using task_for_pid(), defined in /usr/include/mach/ mach_traps.h). See http://web.mit.edu/darwin/src/modules/xnu/osfmk/ man/ for documentation. This email sent to site_archiver(a)lists.apple.com

1 0

Logging physical/virtual memory usage of my app
by Marc Van Olmen 25 Sep '07

25 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi, I have 24/7 that has a slow memory leak... I would like regularly log the real Total memory size(VSIZE ) that my app is using. What's the best way to retrieve this information from the System? thanks marc _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

P2P Streaming
by Thiago Silva 25 Sep '07

25 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=J41negrLB9Jdx7K+qnTsPWoTokLQCm4ISR5vdCBtEfo=; b=kI52CnULZcRiAZlW1sJXsaWVswKq6HeJ2D74cp1s2MMvtbVlJITV+nzPlgYyP7DqPJWBFlWXn2O3poS206Qrz4K+eibShNnckxXivpfYwxH5MUTd4QZ03e+6Wohid7FF82Dp12wn5gXUyDpvncK0lOEzDXyUTW+jgR5LyAfW5Ek= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=ZM1w7IY21Y9BoiNZBfk8T7E2j0aL5D19DS8OU/OQA9InP6+987OiceF+2dmLAaN5LL70KApRvPawWMHk9+3PVwH41i43XpydQI848r8LxUkjJUe/ZK1QPtVYJxJZx+J8asN3jM7wLjG6d9FDbZ9cc8FW3ByBoxRxsloBv6qZeIA= Hi, I would like to do some experiments in live streaming p2p. I would like to use DSS as my server and quicktime as my player. So what I need to do is, any peer must be able to connect in the server and play the streaming, as well serve another peers. In other words a peer need to be able to retransmit the streaming that it got from the server to another peer. Do you guys have any example of a program that connect in the DSS and play the content in the quicktime? Or any other reference based on that? Right now I am able to connect in the DSS server, store all data that it sends to me, and I am also able to accomplish with success the phases DESCRIBE, SETUP, PLAY of a quicktime player requisition. But I am not able to make the quicktime play the RTP packets that I am sending to it. Maybe with some examples I can fix it. Thanks. Pedpano _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Cleanup of pthread_key thread-specific data
by Sam Vaughan 25 Sep '07

25 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 25/09/2007, at 9:50 AM, Greg Parker wrote: Comforting words to hitch-hiking pthread coders everywhere. I opened Radar bug #5502493 for this. Thanks for your help! Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Sam Vaughan wrote: If my interpretation of the man page is correct, my code would call free() PTHREAD_DESTRUCTOR_ITERATIONS times in a row for each thread's buffer pointer when the threads are destroyed. This would obviously be bad. To fix it I'd need to have my own destructor function that knows the key and calls pthread_setspecific to clear the thread's key value after calling free() itself. I wrote a little program to see what behaviour really occurs on 10.4.10 and found that the key destructor is actually only called once, even if it doesn't clear the key value itself. Don't panic. Your code is fine. libc erases the thread-specific data before calling the destructor once. This is the correct behavior. You don't have to erase it yourself. The outer destructor loop is to handle the case where a thread- specific destructor itself sets some thread-specific value (perhaps for a new key, or a non-NULL value for an existing key that's already had its destructor called once). The next iteration of the loop will destroy that new value. If the thread-specific destructors are still inserting new thread-specific values after PTHREAD_DESTRUCTOR_ITERATIONS then libc just gives up and kills the thread, leaking whatever values are left. For example: your key's destructor is called once. Then some other destructor is called, and that destructor calls into your code. Your code sees that your thread-specific value is now NULL, and creates a new one. Next time through the destructor loop, your key's destructor will be called again to free this second value. Thanks for explaining this, Greg. After checking man pages and source on OpenBSD and Linux as well as the POSIX document at The Open Group I was happy to leave my code alone, but it did seem odd that so much man page space was devoted to the edge case you described. I think the man page should be explicit about the fact that the key's value is cleared just before the destructor is called to avoid the confusion. The Linux man page at <http://linux.die.net/man/3/ pthread_key_create> explicitly states that "the value of the key is set to NULL, and then the function pointed to is called with the previously associated value as its sole argument". The Open Group is explicit as well: <http://www.opengroup.org/onlinepubs/000095399/ functions/pthread_key_create.html>. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Cleanup of pthread_key thread-specific data
by Greg Parker 24 Sep '07

24 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Don't panic. Your code is fine. -- Greg Parker gparker(a)apple.com Runtime Wrangler _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Sam Vaughan wrote: If my interpretation of the man page is correct, my code would call free() PTHREAD_DESTRUCTOR_ITERATIONS times in a row for each thread's buffer pointer when the threads are destroyed. This would obviously be bad. To fix it I'd need to have my own destructor function that knows the key and calls pthread_setspecific to clear the thread's key value after calling free() itself. I wrote a little program to see what behaviour really occurs on 10.4.10 and found that the key destructor is actually only called once, even if it doesn't clear the key value itself. libc erases the thread-specific data before calling the destructor once. This is the correct behavior. You don't have to erase it yourself. The outer destructor loop is to handle the case where a thread- specific destructor itself sets some thread-specific value (perhaps for a new key, or a non-NULL value for an existing key that's already had its destructor called once). The next iteration of the loop will destroy that new value. If the thread-specific destructors are still inserting new thread-specific values after PTHREAD_DESTRUCTOR_ITERATIONS then libc just gives up and kills the thread, leaking whatever values are left. For example: your key's destructor is called once. Then some other destructor is called, and that destructor calls into your code. Your code sees that your thread-specific value is now NULL, and creates a new one. Next time through the destructor loop, your key's destructor will be called again to free this second value. This email sent to site_archiver(a)lists.apple.com

1 0

Cleanup of pthread_key thread-specific data
by Sam Vaughan 24 Sep '07

24 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thanks, Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I was concerned to read the following paragraph in the pthread_key_create man page this morning: If, after all the destructors have been called for all non-NULL values with associated destructors, there are still some non-NULL values with associated destructors, then the process is repeated. If, after at least [PTHREAD_DESTRUCTOR_ITERATIONS] iterations of destructor calls for out- standing non-NULL values, there are still some non-NULL values with asso- ciated destructors, the implementation stops calling destructors. The reason for my concern was that I have code that creates a key for per-thread buffers using pthread_key_create(&key, free). The idea is that the destructor for key values is the standard libc free() function. Each thread calls pthread_getspecific with the key to get their buffer. If it's NULL, as it will be the first time they call it, they allocate a new buffer and store its address with pthread_setspecific. They then rest assured that the buffer will be freed automatically when they terminate. If my interpretation of the man page is correct, my code would call free() PTHREAD_DESTRUCTOR_ITERATIONS times in a row for each thread's buffer pointer when the threads are destroyed. This would obviously be bad. To fix it I'd need to have my own destructor function that knows the key and calls pthread_setspecific to clear the thread's key value after calling free() itself. I wrote a little program to see what behaviour really occurs on 10.4.10 and found that the key destructor is actually only called once, even if it doesn't clear the key value itself. I then had a look in the libc source and saw that the thread-specific data cleanup function explicitly clears the key value just before calling the destructor. See _pthread_tsd_cleanup() in pthread_tsd.c, right down the bottom. The outer for() loop is actually completely redundant. So I'm now wondering if I should fix (and complicate) my code to match the man page in case libc gets fixed, or leave it alone and perhaps submit a bug report. Any suggestions? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Thread Priority?
by Terry Lambert 21 Sep '07

21 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi, - Some other unforseen latency = Your protocol needs more water in its bucket. = I highly recommend: TCP/IP Illustrated, Volume 2 Gary R. Wright and W. Richard Stevens Addison-Wesley Professional ISBN 978-0201633542 and: Elements of Network Protocol Design Mohamed G. Gouda Wiley-Interscience ISBN 978-0471197447 -- Terry _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… On Sep 21, 2007, at 10:26 AM, Basavaraj wrote: I also get this problem. Apple, please suggest some solution to this problem. Why reading data on network sometimes take more time than usually. In my case it is very critical, server drops video and audio packets if client don't receive in allowed time. Problem is more prominent in Linux and Mac OS X, but in Windows, it just works fine. If we launch or do some operation on some cpu intensive application or simple X11 apps, then problem is more frequent. Problem is not because of receiving data. I saw tcp dump on Ethereal, actual problem is with TCP acknowledgment. Acknowledgments are not going immediately after receiving packets. Disabling Naggle algorithm didn't helped. Please help any one or Apple. You are aware that TCP ACKs are not sent as a direct result of the receive interrupt, right? Data comes into the PHY, the ethernet chip DMAs the data into preallocated mbufs in physical memory for which there is a preestablished kernel virtual address space mapping, and then the mbuf data is put on a ring buffer, and a condition "data available" is signalled. Then the interrupt is ACKed, and processing at interrupt level is done. At that point, you wait for the scheduler to run the receive thread, which dequeues the data out of the ring buffer and feeds it to ether_input, which eventually calls ip_input, which in turn calls tcp_input, which does the TCP protocol processing. At some point in the protocol processing, TCP decides that the window is no longer full, and then decies to send an ACK via ip_output. ip_output queues the data onto the send ring buffer, and signals the ethernet driver that there is data to put in the DMA area with a descriptor so that the DMA engine on the chip will take it and send it. Assuming that the ACK is eventually sent without the data being retransmitted, it's possible that the receive window won't be advertised for some time (and the ACK sent) if: - You have a thread which is higher priority than the thread dequeuing the data from the receive ring to ether_input; for example, perhaps someone is running a thread at a higher real time priority than the thread servicing the queue. - The receive data buffer in the socket (so_rcv) is full because the user space program has not emptied it in a while, because the user space process has not been scheuled to run; for example, perhaps someone is running a thread at a higher real time priority than the thread in the process servicing the socket receive queue. In general, people who design protocols for transferring data over network links take latency into account in their designs, and buffer more data than they will need "if everything goes right", in case it doesn't. This means that there is data available, and there is no "skipping" or "freezing", because they have enough data available to continue processing data even if the network data is refilling those buffers at a variable rate. In technical terms, this is called "pool retention time", and it's part of standard queueing theory. It's the same reason thatt by the time McDonalds gets you your food on a tray, no matter how busy they are, there's a place for you to sit down and eat it. The classic example is a bucket that is being filled from the top at some average rate which in any given time slice is inconsistent, but which has a hole in the bottom whic causes it to drain at a constant rate. The question that is generally posed is "what is the minimum amount of water I have to put in the bucket before I unplug the hole in the bottom to ensure that there is a constant stream of water cominf from the hole?". It's possible to overcome all of these issues for a network protocol, of course, but then what you've built is a network protocol processor, rather than a general purpose computer, and everything that's not networking loses in any contention for resources between it and the networking. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Thread Priority?
by Shawn Erickson 21 Sep '07

21 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=FmF4eeM0VTWdQcHEXAfltgCTfNwtCba4Chs3xoMouRw=; b=rHI3jj2U1ZzKPEMaGOwWGXPPtjpxRYXD1VSrBcSSCXw7buto0DOisYAV/7pRw/P8cufhA0t+oPTqPnlWdGL2Lanja505OFcsn9OQARgcazpdo33lMJsfuaT8jtVCu44XX7fCbYX+xscYp7V4TUiBgB72dgSSA0EA1Yk0RLt6Sb0= Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=gHz2zNvM2sRinfh8hqhL4cE9wW1DOoLvR6new8A16zJHp5ERRhh2W9o0Jv0K9rCgO/SqN/VICK+IfqBEUnyCKzSnpJPF/xsoj41Zuk4UC95tv6uZQkjbPehs6XouCERz4eCD5GLFbm7MbNF9w5pm+xPKsy1pnHZ5uFeShLW4Xwc= On 9/21/07, Basavaraj <basavarajdevelop(a)gmail.com> wrote: > Apple, please suggest some solution to this problem. This is not an official channel to request help form Apple. If you need an answer / assistance use an official channel (or hope that someone responds on this list)... <http://developer.apple.com/technicalsupport/> -Shawn _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Thread Priority?
by Jonas Maebe 21 Sep '07

21 Sep '07

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 21 Sep 2007, at 19:26, Basavaraj wrote: Jonas _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Problem is not because of receiving data. I saw tcp dump on Ethereal, actual problem is with TCP acknowledgment. Acknowledgments are not going immediately after receiving packets. Disabling Naggle algorithm didn't helped. Have you also tried to disable delayed acks? (sysctl net.inet.tcp.delayed_ack, try setting it to 0) This email sent to site_archiver(a)lists.apple.com

1 0