Re: NFS limited to 4K reads/writes
Re: NFS limited to 4K reads/writes
- Subject: Re: NFS limited to 4K reads/writes
- From: Andrew Gallatin <email@hidden>
- Date: Fri, 1 Feb 2002 09:53:58 -0500 (EST)
John Clark writes:
>
>
Am Donnerstag den, 31. Januar 2002, um 08:52, schrieb Andrew Gallatin:
>
>
>
>
> By poor performnace, I mean that Darwin reads at 12MB/sec over a
>
> 2Gb/sec link. Tru64 and FreeBSD can both read at 50MB/sec. (this is
>
> from a file warm in the server's cache; no disk I/O). Darwin performs
>
> just as well as the competition when its the server...
>
>
Since network performance triggers my carnivore monitoring software...
I understand.. I should have posted more details.
>
I'll assume that by '2 Gb/sec' link, one is talking about gigabit
>
ethernet
>
in full duplex mode.
No, actually. The link is Myrinet-2000 (Full-duplex 2+2 Gb/sec).
>
How are you measuring your performance. Also what are the hardware
>
details, for example Darwin on a G4 with the internal gigabit ethernet
>
interface; Tru65 or FreeBSD on a ???.
The mac is a dual g4 800MHz, running 10.1. The PCI bus is capable of
roughly 240MB/sec for reads and writes. The FreeBSD host is running
FreeBSD 4.5-RELEASE on a dual 1Ghz PIII with a Serverworks LE chipset
whose PCI performance is roughly 450MB/sec for reads and 500MB/sec for
writes. The alpha is a 500MHz 21264 (a Monet) with a tsunami chipset
whose PCI performance is rougly 190MB/sec reads and 240MB/sec writes.
The Myrinet device is a a 64-bit/66MHz device (running at
64-bits/33MHz in the Mac and the Alpha) and these are actual, observed
performance numbers with the hosts in question.
In OS-Bypass mode (no copies DMA to/from wired user buffers, no
sockets, no interrupts) the device can sustain 200MB/sec between the
FreeBSD host and the mac (unidirectional, symetrical).
>
Also have you tried ttcp and verified the 50 MB/s is symetrical.
More or less. In ethernet emulation mode, I use netperf, and its
actually quite a bit better than 50MB/sec.. The following is from
using netperf between the mac and the FreeBSD box. The ethernet
emulation MTU is 9000 bytes.
The mac can send TCP at 670Mb/sec and receive at 750Mb/sec (256K
socket buffers). With 8K UDP datagrams, it can send at a hair over
1Gb/sec and receive at 500Mb/sec. Actually ~900Mb/sec makes it into
the host and a little less than half are dropped due to full socket
buffers. With 4K datagrams, it can send at a hair under 700Mb/sec and
receive at a little over 400Mb/sec. These numbers don't change much
when I boot the PC into linux 2.4.16 (the tcp send numbers decrease a
little, and the receive numbers increase a little, udp stays the
same).
Trust me -- its the NFS read/write size that's killing perf, or at
least making a major contribution to it. If I cut the read/write size
down to 4k on the FreeBSD host, the read bandwidth drops in half from
50MB/sec to 25MB/sec:
# mount_nfs -U -a 4 -w 16384 -r 16384 g4-my:/private/var/tmp /mnt
# dd if=/mnt/zot of=/dev/null bs=64k
2400+0 records in
2400+0 records out
157286400 bytes transferred in 3.145618 secs (50001749 bytes/sec)
# umount /mnt
# mount_nfs -U -a 4 -w 4096 -r 4096 g4-my:/private/var/tmp /mnt
# dd if=/mnt/zot of=/dev/null bs=64k
2400+0 records in
2400+0 records out
157286400 bytes transferred in 6.204624 secs (25349868 bytes/sec)
FWIW, the mac has always felt rather slow for NFS even on a 100Mb/s
network. (reads at 5MB/sec rather than 10 like FreeBSD & newer
versions of linux), so this is something of a problem even on
commodity networks. I just never bothered looking into it until I was
stress testing my Myrinet driver and noticed how low the bonnie
numbers were..
Cheers,
Drew