SUMMARY: Network degradation with a dual-homed E4500

Problem summary:
----------------
When I brought up hme1 on my E4500, network performance degraded. I was 
doing:

sudo ifconfig hme1 plumb
sudo ifconfig hme1 192.168.1.2 broadcast 192.168.1.255 netmask \ 
255.255.255.0 up
sudo ndd -set /dev/ip ip_forwarding 0
----------------


Thanks to:
----------
Darren Dunham
Peter Laws
Wade Stewart
----------


Solution summary:
-----------------
The two networks to which my E4500 is connected were on the same switch, 
but different VLANs. Tragically, both interfaces were using the same MAC 
(ethernet hardware) address, which was confusing the poor switch and 
causing the bad performance that I was seeing.

So, I used the quick fix suggested by Darren Dunham:

DD> You'll need to use 'ifconfig <int> ether <mac>' to force a differing
DD> address on one of the interfaces.

Now I am bringing up the interface like this:

sudo ifconfig hme1 plumb
sudo ifconfig hme1 ether 08:00:20:e6:3a:2d
sudo ifconfig hme1 192.168.1.2 broadcast 192.168.1.255 netmask \ 
255.255.255.0 up

And it works quite nicely.
-----------------


Troubleshooting details:
------------------------
It was Darren Dunham that put me on the path to finding what was causing 
the problem:

DD> Does 'masta' report any errors on the 192.168 interface?  Is that
DD> network a switch or a hub?  Is the duplex matched on that interface?
DD> Does 'masta' have an arp entry for the other machine?

As I stated above, the two networks were on the same switch, but one was 
a VLAN. Poking at the ARP cache on 'masta' (another box on the VLAN--the 
192.168.1.0 network), I noticed something that I had not the first time:

:jmglov@masta; arp
Address             HWtype  HWaddress           Flags Mask  Iface
192.168.1.2         ether   08:00:20:E6:3A:2C   C           eth1
foo.incogen.com     ether   08:00:20:E6:3A:2C   C           eth0
                             ^^^^^^^^^^^^^^^^^

See the problem? Both interfaces are using the same hardware address. 
This is problematic when both networks are on the same switch.

So, two solutions became available. The one I went with was manually 
setting the hardware address of the second interface with the 'ifconfig 
hme1 ether 08:00:20:e6:3a:2d' command, suggested by Darren Dunham (and 
choosing a hardware address by simply incrementing the system's hardware 
address by one). Of course, this would cause problems if we get another 
interface / system with that hardware address and put it on the same 
switch, but that is extremely unlikely.

The second solution, suggested by all three of my expert sysadmins, but 
voiced here by Wade Stuart:

WS> My guess is you have the e4500 still using the system MAC address
WS> rather then a unique per card MAC.
WS> At the eeprom "ok" prompt type:
WS> setenv local-mac-address? true
WS>
WS> This forces each interface to give a unique MAC not the system MAC.
WS> There are other settings you need for QFE cards but your example
WS> shows happymeal (hme) cards. =)

I shall follow his advice when I can afford to reboot the box.
------------------------

Thanks to all who responded for getting this fixed within half an hour 
of my post. *This* is how sunmanagers is supposed to work! :)


-- 
Josh Glover <jmglov@incogen.com>

Associate Systems Administrator
INCOGEN, Inc.
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers