SUMMARY: Solaris 10 Sparc bge issues - Repeated link up messages, no link down

Hello!


Ended up with installing a Sun QuadGB card X4445A and disabling the onboard
Broadcom NICs. Machine has run without problems for three weeks now!

My guess is that there is some kind of incompability between the QuadGB 
Broadcom card and our Cisco switch/Cisco IOS. 

For those of you with access to Experts Exchange:

http://www.experts-exchange.com/OS/Unix/Solaris/Q_24403820.html

BR,

Nils



-----Original Message-----
From: NKS [mailto:sunadmin@norsar.no] 
Sent: Wednesday, June 10, 2009 9:32 AM
To: sunmanagers@sunmanagers.org
Subject: Solaris 10 Sparc bge issues - Repeated link up messages, no link down

Dear all,

I have several SunFire V240 with identical hardware and identical operatimg 
systems. 
Solaris 10 10/08 release. Latest recommended & security patches. Quad onboard 
ethernet
(Broadcom) with two ports used (0 & 1) on different subnets. iSCSI-connected 
storage. 
Host is connected to a Cisco switch. Auto-sense speed/duplex.

var/adm/messages:

/var/adm/messages.2:May 19 08:33:58 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.2:May 19 21:30:18 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge0: link up 1000Mbps Full-Duplex
/var/adm/messages.2:May 20 15:27:50 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.2:May 21 05:16:11 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May  9 09:02:34 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 13 14:14:48 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 09:47:07 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge0: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 15:13:23 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge0: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 15:17:35 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge0: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 15:55:20 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge0: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 15:55:41 iscsi-freya genunix: [ID 408114 kern.info] 
/pci@1f,700000/network@2 (bge0) online
/var/adm/messages.3:May 14 15:57:44 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
/var/adm/messages.3:May 14 16:46:47 iscsi-freya bge: [ID 801593 kern.notice] 
NOTICE: bge1: link up 1000Mbps Full-Duplex
........

Network interface does not go down; only interface up messages.

# uname -a
SunOS iscsi-freya 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V240


# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 
1
        inet 127.0.0.1 netmask ff000000
bge0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
        inet 10.10.11.10 netmask ffffff00 broadcast 10.10.11.255
        ether 0:3:ba:9f:a2:21
bge1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
        inet 10.10.10.3 netmask ffffff00 broadcast 10.10.10.255
        ether 0:3:ba:9f:a2:22

# kstat -m bge | grep chip_type
        chip_type                       5794
        chip_type                       5794
        chip_type                       5794
        chip_type                       5794
      
# modinfo | grep bge
141 7b6e0000  148f8 202   1  bge (BCM579x driver v0.56)

# dladm show-dev
bge0            link: up        speed: 1000  Mbps       duplex: full
bge1            link: up        speed: 1000  Mbps       duplex: full
bge2            link: unknown   speed: 0     Mbps       duplex: unknown
bge3            link: unknown   speed: 0     Mbps       duplex: unknown


Tried to force fdx/1000 on the bge-interfaces. On my Cisco 4006 switch:

   conf t
   int g4/7
   speed 1000
   int g4/15
   speed
   ^Z
   wr mem

Then edited /platform/sun4u/kernel/drv/bge.conf. First I changed

   adv_autoneg_cap       = 0;
   adv_1000fdx_cap       = 1;

and rebooted. The network did not come up at all - no link on the interfaces 
on the switch.

Then I reset the bge.conf to default and changed

   speed                 = 1000;
   full-duplex           = 1;

and rebooted. No network this time either...Everything on one line in bge.conf
gave the same result...

# dladm show-dev
bge0            link: down      speed: 0     Mbps       duplex: unknown
bge1            link: down      speed: 0     Mbps       duplex: unknown
bge2            link: unknown   speed: 0     Mbps       duplex: unknown
bge3            link: unknown   speed: 0     Mbps       duplex: unknown

# ndd -get /dev/bge0 adv_autoneg_cap
0
# ndd -get /dev/bge0 adv_1000fdx_cap
1

Changed back to default settings and rebooted. 

Then I tried the following:

/usr/sbin/ndd -set /dev/ip ip_path_mtu_discovery 0
/usr/sbin/ndd -set /dev/tcp tcp_mss_max_ipv4 1460

http://www.cisco.com/en/US/tech/tk870/tk877/tk880/technologies_tech_note09186a00
8011a218.shtml

Didn't help either...

Any ideas on how to fix this?

BR,

Nils
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers