> One obvious suggestion would be to turn off delayed ACK on the Mac OS X box.
>
> sudo sysctl -w net.inet.tcp.delayed_ack=0
> You can do this using UNIX sysctl or my own IPNetTunerX
> parameters available and the effect of tuning.
>
> I noticed just now on 10.4.8 that the value for delayed Ack is "3" which
> surprises me. Previously I only noticed it could be "on" (1) or
> "off" (0). This may have some bearing on why Mac OS X is ACKing so lazily.
I think I tried that, but I'll doublecheck. (Yes, on Panther it's "1" - I never noticed that!)
We tried several settings:
net.inet.tcp.delayed_ack=0 ("off"):
Speed: fast
Packet ratio (Server/Client): 0.81
net.inet.tcp.delayed_ack=1 ("on"):
Speed: slow
Packet ratio (Server/Client): 5.0
net.inet.tcp.delayed_ack=2 ("more compatible"):
Speed: fast
Packet ratio (Server/Client): 1.50
net.inet.tcp.delayed_ack=3 ("?"):
Speed: slow
Packet ratio (Server/Client): 4.43
net.inet.tcp.delayed_ack=4 ("?"):
Speed: fast
Packet ratio (Server/Client): 0.76
net.inet.tcp.delayed_ack=5 ("?"):
Speed: fast
Packet ratio (Server/Client): 0.81
So, we could use the "more compatible setting". The downside is that much more packets (3 times more) are sent. This uses more bandwidth, and on the server side a considerable amount of cpu-time (which can't be speed up by adding CPUs or RAM). Short, it's a good workaround, but I'd prefer to get setting "1" working. Any ideas?
> This certainly looks like a delayed_ack issue. I've just never seen the
> delay become quite so large. Are your two machines directly attached to
> the same LAN, or is there a router between them? I ask because TCPs
> slow start behavior is different for a host on a non-local network.
They are attached to different subnets, and as you suspect, the problem only occurs if they are. If I attach the Mac to the same subnet as the Sun, it just works.
We tried to set the TCP-LAN-values for the TCP-WAN-variables on the Sun, so that TCP should react the same way in LAN and WAN. Strangely, it didn't change anything. :-/
> Notice you can also adjust the maximum ACK delay time (net.inet.tcp.delacktime). The default is 50ms.
I will tinker with this setting. Question: For which case is this delay used? I mean, the delay seems to change as needed...so shouldn't there be a range defined, or settings for each "state" of connection?
This setting does not seem to be honoured. :-(
(Hint from Justin)
> This is the comment from tcp_input.c (ppc darwin 8.0.7 source), where DELAY_ACK is defined:
>
> * Indicate whether this ack should be delayed.
> * We can delay the ack if:
> * - delayed acks are enabled (set to 1) and
> * - our last ack wasn't a 0-sized window. We never want to delay
> * the ack that opens up a 0-sized window.
> * - delayed acks are enabled (set to 2, "more compatible") and
> * - our last ack wasn't a 0-sized window.
> * - if the peer hasn't sent us a TH_PUSH data packet (this
> * solves 3649245)
> * - the peer hasn't sent us a TH_PUSH data packet, if he did,
> * take this as a clue that we
> * need to ACK with no delay. This helps higher level protocols
> * who won't send
> * us more data even if the window is open because their last
> * "segment" hasn't been ACKed
> * - delayed acks are enabled (set to 3, "streaming detection") and
> * - if we receive more than 4 full packets per second on this
> * socket, we're streaming acts as "1".
> * - if we don't meet that criteria, acts like "2". Allowing faster
> * acking while browsing for example.
I'll tinker with this. I wonder if behaviour will change.
See above.
(Hint from Josh)
> Actually, with 10.4.8, SACK is there and enabled by default.
>
> sysctl net.inet.tcp
>
> You should see:
>
> net.inet.tcp.sack: 1
> net.inet.tcp.sack_maxholes: 128
> net.inet.tcp.sack_globalmaxholes: 65536
> net.inet.tcp.sack_globalholes: 0
Uh. I will try disabling SACK.
Disabling SACK did not help. We also noticed that issue occurs with 10.4.7 as well, where SACK is not yet implemented.