Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?)

Very nice work.

There's still more improvement to be done...But Tiger now puts us in a better position to change some implementation details to make things faster.

FWIW, when I was implementing my filter interrupt, I looked
at this code, and it left me dazed and confused ;). Can you
explain something?

In IOWorkLoop::threadMain(), it blocks with thread_block_parameter((thread_continue_t)threadMainContinuation, this);

Does that mean that when it wakes up, it start from
IOWorkLoop::threadMainContinuation, which re-enters
threadMain()?

Yes.

What's the advantage to doing it this way?

Well, if you look at thread_invoke() in sched_prim.c, it appears that it allows the scheduler to do a stack handoff between the newly selected runnable thread and the previous thread (so it doesn't have to do a stack allocation when it does the context switch). In essence, I believe it frees the system up to quasi-reclaim the wired memory used for that kernel stack (if that makes sense to you).

Yes, the DART setup was another major improvement that I noticed.
It decreases our large message latency dramatically between Panther
and Tiger. Eg, 2801us -> 2311us for a 1MB transfer.

Yes. Populating the DART table twice was not good! Using kernel tracing is really helpful for finding these bubbles in the pipeline. We spent a lot of time analyzing kernel traces and instrumenting the entire storage stack to reduce this latency and others. The net results are quite noticeable (in Tiger, we do about twice as many IOPS as Panther using 4K I/Os). Our streaming performance increased slightly as well (preparation time for first I/O was reduced). Streaming performance isn't as highly dependent on the preparation time since UBC issues multiple async I/Os and overlaps the preparation time of the I/Os with I/Os already in flight to the disk (and since disks are slow, we always keep the pipeline flowing).

Any chance of improving your IP stack latency? I see roughly 45us
1/2 RTT latency using the netperf UDP_RR pingpong tests between 2 dual
2.0GHz G5s running ethernet emulation on top of the Myrinet device I
keep talking about. This compares to 10.9us on a pair of 1.8GHz
dual opeteron 244s running linux with the same NIC. This is just
a one-byte UDP packet ping-ponging, so there should be no chance
of lock contention, etc.

I would suggest filing a Radar bug about this performance. Any sort of benchmarks which indicate a real-world performance loss (and not just a micro-benchmark) would be good to use as a test bed for any possible fix.

In some cases, the Opteron will have higher bandwidth simply due to its on-chip memory controller architecture and DDR2 SDRAM (guessing you have this on your system - maybe not).

Then again, don't improve it. Bad IP latency is one of the things
paying my check ;)

Ha!

-- Chris

------------------

6 Infinite Loop

M/S 306-2MS

Cupertino CA 95014

phone: (408) 974-4033

fax: (408) 862-7577

email: email@hidden

References:
	>why no IOSimpleLockSleep()? (From: Andrew Gallatin <email@hidden>)
	>IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Godfrey van der Linden <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Andrew Gallatin <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Godfrey van der Linden <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Andrew Gallatin <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Andrew Gallatin <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Chris Sarcone <email@hidden>)
	>Re: IOFIES Filter wakeup (was: why no IOSimpleLockSleep()?) (From: Andrew Gallatin <email@hidden>)