Re: [OT] Mail hangs real-time processes
Re: [OT] Mail hangs real-time processes
- Subject: Re: [OT] Mail hangs real-time processes
- From: Jeff Moore <email@hidden>
- Date: Wed, 7 Jan 2004 10:57:22 -0800
On Jan 6, 2004, at 9:00 PM, Jens Bauer wrote:
Hi Jeff,
[Sorry, in my last reply, I forgot to choose the right reply-account,
so the message wasn't copied to the list]
On Wednesday, Jan 7, 2004, at 01:12 Europe/Copenhagen, Jeff Moore
wrote:
Before, I get into this, I remembered one other question that would
be helpful to know the answer to: When the hang happens, does the
mouse move normally? That is, does it stutter around or move smoothly
like normal?
At my place, I can still move the mouse.
Well, that makes it unlikely that a high priority thread is spinning
thus causing the problem.
Can you switch into other apps and interact with UI items? If you can,
it would be very interesting to know if you can play sound with another
app. The best way to find out would be to use the Sound Prefs Panel or
Volume Menu Extra to trigger an alert sound to play.
At any rate, 50000+ messages in one folder is probably on the large
side for mail folders. Opening one is going to cause a massive amount
of disk IO and possibly a fair amount of VM paging, especially given
the amount of RAM you have. Indeed, using fs_usage I see all sorts of
disk activity going on when I open one of my own small mail folders.
I don't see any VM activity, but my mailbox is small (and nobody in
my immediate vicinity had one any where near that big) so probably
won't have the same memory behavior.
Just in case anyone is wondering why I have that many msgs; it's the
CocoaDev list, and I like to use it for answering dev questions
quickly. ;)
Hee hee. I didn't mean that as a criticism just an observation =)
So, it could be that VM paging or Mail's disk IO is preventing iTunes
from getting at the audio data on the disk when it needs it. VM
paging would be the worst since it will additionally hold off the
HAL's IO thread which will cause glitches.
It's possible, that this is what happened. It's a long glitch. (not
"skips", nor "mutes", because it played from where it left; I
experienced "mutes" with MacAMP once a day, these mutes were for 3
seconds, but that's a completely different story).
So to see if the disk activity is causing troubles, I'd do the
following:
1) Shutdown the machine and cold boot it. Cold booting makes sure
that as little stuff is cached or otherwise already loaded as
possible which should exacerbate this problem if it is disk related.
2) Play back some audio using a tool that does all it's work directly
out of RAM, like HALLab. QTPlayer, iTunes, and most others buffer the
file data off the disk as needed so aren't suitable for this test.
You mean something like a tone-generator application ? I could write
one for that purpose, yes.
Exactly. I mentioned HALLab in particular because it has a file player
window that loads the entire file into memory at once to play rather
than buffering it off the disk as needed.
3) Launch Mail and open the big mail folder and see if it interrupts
the playback.
It might be helpful to run fs_usage as well in a shell somewhere so
you can see who's doing what sort of disk IO.
Would it be best to run this app via the network, or locally?
If possible, it's always best to run your performance monitoring tools
in a remote session, like an ssh session. This removes some of the
burden of running the tool from the machine you are trying to monitor.
For the record:
Sample(1) is a command line tool that can take a peak at the stacks
of all the threads in a process over time. It is a great tool for
diagnosing hangs, infinite loops, and other issues that can be seen
from a stack crawl.
Heh, OK, I was confused because of the name; speaking about audio, and
taking samples didn't make me think of the sampler. =)
The man page is probably the best documentation for it. While I find
the command line version the easiest to use, there is also a GUI
version called Sampler which is located in
/Developer/Applications/Performance Tools. I think the docs that
discuss the various performance tools also has some info on the GUI
version. Now that we have a bit more detail, a sample trace probably
won't be all that easy to get for the portion of time that it is
needed since the hang is so brief.
I can try and see if I can get it anyway. It's worth the try, if I
succeed. ;)
This is probably also a good time to highlight one of the cool features
we put into the HAL: IO cycle telemetry. The HAL has built into it the
capacity for logging the activity of the IO thread to see exactly
what's going on. It can dole this info out via Mach messages to other
processes on the same system. HALLab has a viewer for the HAL's
telemetry data. It would probably be useful to use HALLab to look at
the telemetry coming out of iTunes when the hang happens. Here's how to
do this:
1) Build HALLab. It can be found at
/Developer/Examples/CoreAudio/HAL/HALLab.
2) Launch iTunes.
3) Launch HALLab and make a new IO Cycle Telemetry window (in the File
menu).
4) Find the pid for iTunes. I usually do something like this on a
command line: % ps -aux | grep iTunes
5) Go to HALLab's telemetry window and enter iTunes' pid in the
Process ID field (the Get button doesn't work, yet).
6) Select the audio device you are using with the Device pop-up.
7) Enable the capturing of telemetry using the Capture Rate pop-up.
Any rate will work here, but I'd choose two seconds to minimize the
messaging that takes place.
8) Switch to iTunes and hit play. The telemetry window in HALLab will
start showing the data.
9) Launch Mail and make the hang happen.
10) After things come back to normal, hit stop in iTunes and set the
telemetry capture rate back to off.
11) Examine the telemetry and see what happened.
I'll say right off that the telemetry data is a bit hard to understand,
but it should make sense if you have an understanding of how the HAL
does it's job. If you want to try this out, what you should be looking
for is red lines in the telemetry. These will mark the cycles that most
likely correspond with the hang. All the intricacies of what the
telemetry means are too much to fit into this email, so feel free to
take screen shots and ask questions.
At any rate, you'll see the telemetry data presented with each IO cycle
on a single line with all time values displayed as milliseconds. The
fields correspond to the time stamps involved with the various phases,
including when the IO thread woke up to begin the cycle with the offset
from the previous cycle and how late the wake up was, the HAL's clock
rate scalar, the three time stamps passed to IOProcs (now, input,
output) and if the driver provided a new zero time stamp, it will get
printed out too. If there wasn't either non-zero input data or non-zero
output data for that cycle, then the line will be shown in gray. If
there was some kind of error or other exceptional situation during the
cycle, that line will be shown in red (you can use the Previous/Next
buttons to step from red line to red line). Each line in the table can
be opened using the disclosure triangle and all the raw data for the
cycle will be displayed.
--
Jeff Moore
Core Audio
Apple
_______________________________________________
coreaudio-api mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/coreaudio-api
Do not post admin requests to the list. They will be ignored.