On 09.04.2008, at 21:09, email@hidden wrote:
Greetings,
Here's my situation: I have a custom piece of data logging hardware
that
generates about 50MB/sec of data and sends it out over a gigabit
ethernet
link as a steady stream of UDP packets (1k packet every ~20
microseconds,
very periodic, not bursty at all) broadcasted to the local subnet.
I want
to log all this data to disk for offline analysis. As a first
step I
wrote a program to just listen for the UDP packets and verify that
they
were all being received. This is easy to do because each packet has
a 16
bit sequence ID in it. Much to my dismay, I was unable to reliably
achieve this. I am running 10.5.2 on a 2.33 gHz MacBook Pro with
2G of
memory. After making SO_RCVBUF big enough, I was still
occasionally
missing a block of 100+ packets at a time. Looking in the
system.log file
I realized that each dropout was correlated to a message like:
Apr 9 13:38:51 Macintosh-6 kernel[0]: AppleYukon2:
00000000,00000000
skgehw - cppSkDrvEvent - SK_DRV_RX_OVERFLOW: rcv fifo overflow
Apr 9 13:38:51 Macintosh-6 kernel[0]: AppleYukon2:
00000000,00000000 sky2
- RX ring overflow -- dropped a packet
I think your problem is related to the fact that your userspace
application can't keep up with the sent data from the kernel and thus
the buffer in the kernel overruns. 50MBytes/sec of data is quite a
bit
and because your application writes it to disk, you need disks which
can write that fast (my MacBook Pro disk can only handle something
like 35MB/sec) If they are not as fast, you will naturally run out
of
buffer space at some time. I would suggest you redo this experiment
with a striped array of firewire 800 disks.
If you of course only write it to memory, its a different story.
You can test this, by writing a sample application which only reads
the id and drops the packet without doing anything else and see if
that still doesn't keep up. You might have to think about how you
read
the UDP packets and if there's anything you can do there to improve
that speed.
I believe your problem however is simply disk speed.
Andreas Fink
Fink Consulting GmbH
Global Networks Schweiz AG
BebbiCell AG
---------------------------------------------------------------
Tel: +41-61-6666330 Fax: +41-61-6666331 Mobile: +41-79-2457333
Address: Clarastrasse 3, 4058 Basel, Switzerland
E-Mail: email@hidden
www.finkconsulting.com www.global-networks.ch www.bebbicell.ch
---------------------------------------------------------------
ICQ: 8239353 MSN: email@hidden AIM: smsrelay Skype: andreasfink
Yahoo: finkconsulting SMS: +41792457333
http://a-fink.blogspot.com/ A developers view about iPhone SDK
