I have recently had installed a second ADSL line to give a bit of
security for if we have trouble with our existing line/provider.
...
I was hoping to list the 2nd line as an MX backup so if line 1 is
down there would still be a route through to our mail server.
Unfortunately, I don't think that it's going to work that way. In
this configuration, the box can receive traffic on either of its two
interfaces, no problem. The trouble is that the mac will always use
its "default gateway" for all outbound traffic that doesn't have an
explicit route (regardless of the interface the traffic came in on).
The default route on the mac is the gateway on the card with the
highest priority. As you observed, whichever of your two cards is
the primary is the one that will "work". In reality, both cards are
receiving traffic just fine, it's just that the return traffic is
getting routed out the "wrong" interface and probably being dumped by
the host on the other end (because it appears to come from the
"wrong" host). You can verify this by running tcpdump or another
sniffer; you'll see the packets coming in, and the packets going out,
but the outbound traffic will have the wrong source IP for the
network it's on.
What you WANT is for the machine to "remember" which interface
traffic came in on, and use that interface to route the responses
back. I'm not sure if this is possible under Mac OS X; you can do it
with some trickery under OpenBSD (using PF's "route-to" directive),
and probably under other systems as well (haven't tried). It's not
ideal, but short of getting your own netblock and getting your ISPs
to participate in a legitimate routing protocol (BGP) with you, it's
your cheapest way to get link redundancy.
If this is possible under OS X, someone please correct me. I haven't
done enough firewall hacking on OS X to know for sure, but a quick
read of the ipfw man page didn't yield anything obvious.
