Sorry, this summary dit not got back here.
It has been 5 days. I saw it on info.sun-managers,
people got back to me but I suspect that
not everybody got it, so here it is again:
Yes it is possible, but it is not that simple.
The Sun is a SparcStation10 with a FSBE board.
This give me 2 ethernet ports.
The idea of hooking 2 ethernet ports on the same
network is not a gain because of the limitation
of 10Mb/s of that network.
The Kalpana Ethernet switch is a device that
switch ethernet packets from port to port almost
like a bridge depending on the MAC address.
Now, using two ethernet ports with a Kalpana will
give us about 2 x 10Mb/s network as the
switch take care of the traffic.
The important aspect of this configuration is
to have the server thinks it has 2 subnets
while the rest of the sun believes there is
no subnet.
To get it to work I needed to do the following:
1) Turn off IP_FORWARDING in the config of the kernel,
this was causing a lot a bad routing information
on the networks
options IPFORWARDING="-1" # see /sys/netinet/in_proto.c
2) Assign a different Ethernet address to le1
This is required by the Kalpana. With the
same MAC address on 2 ports, the switch keep
updating it's routing table.
I added the following to rc.boot
if [ "$1" != "xx0" ]; then
+ ifconfig $1 ether "`shcat /etc/ether\.$1`"
ifconfig $1 "`shcat /etc/hostname\.$1`" netmask + -trailers up
fi
Where I put the Sun assigned ether in /etc/ether.le0 and
a different one for /etc/ether.le1
3) Create 'virtual' subnet on the SS10 by using
bogus 138.11.48.214 for le0 on 'pseudo' network 138.11.0.0
bogus1 138.11.129.214 for le1 on 'pseudo' network 138.11.128.00
Note: You need to be able to broadcast 138.11.255.255 so
the higher subnet is the most important. Also, because
the rest of the network is on net 138.11, we might
need to cover the full address range with all the
interfaces.
4) Add the following to /etc/networks
138.11 255.255.128.0
5) Modify /etc/networks to create subnet for the SS10
loopback 127
matrox-ether 138.11.0
matrox-ether1 138.11.128
6) Move the following ifconfig before NIS is started
in /etc/rc.local in order to have the Sun believe
it's /etc/networks & /etc/netmasks.
ifconfig -a netmask + broadcast + > /dev/null
7) Prevented in.routed from starting in /etc/rc.local
You don't want the Sun routing packets, you want
the Kalpana doing it.
8) Add necessary route with 'route add ...'
9) Change half to the clients address to 132.11.129.???
10) Place those clients on subnet 138.11.0 on a different
port of the Kalpana than the clients on net 138.11.128.
11) Modify automount maps so that client on pseudo network
138.11.128.0 mount from 138.11.129.214
Without this modification, outbound NFS traffic for
subnet 138.11.128 was going to le1 but the inbound
traffic was going to le0.
The setup is running without noticable problem for about
15 days now. (There is some strange things like two
ntp connection, but does not cause any problem)
I have no tools to mesure if we effectively get
2 x 10Mb/s but netstat show that the traffic is
split in half between each interface and that the collision
rate is back to normal level (< 1.25%).
uptime
7:32pm up 6 days, 6:44, 2 users, load average: 1.28, 0.77, 0.72
ifconfig -a
le0: flags=63<UP,BROADCAST,NOTRAILERS,RUNNING>
inet 138.11.48.214 netmask ffff8000 broadcast 138.11.127.255
le1: flags=63<UP,BROADCAST,NOTRAILERS,RUNNING>
inet 138.11.129.214 netmask ffff8000 broadcast 138.11.255.255
netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
le0 1500 138.11.0.0 bogus 21516762 109 20531289 1 74122 0
le1 1500 138.11.128.0 bogus1 18692667 8 23570747 0 290294 0
lo0 1536 loopback localhost 2700868 0 2700868 0 0 0
netstat -r
Routing tables
Destination Gateway Flags Refcnt Use Interface
localhost localhost UH 6 2651461 lo0
138.11.128.0 bogus1 U 4 15335298 le1
matrox-ether bogus U 60 16521136 le0
matrox-token alf-ether UG 0 0 le0
Thanks to the following peoples and in particular to
Mike Raffety to pointing out the concept of letting
the server and only the server believe that there is subnets.
From: Ian Cassidy <pru!cassidyi>
From: Michael Assels <mjassels@cs.concordia.ca>
From: pbg@cs.brown.edu (Peter Galvin)
From: hyder@niwot.scd.ucar.EDU (Paul Hyder)
From: Jeff Bacon <bacon@mtu.edu>
From: Mike Raffety <miker@il.us.swissbank.com>
From: stern@sunne.east.sun.com (Hal Stern - NE Area Systems Engineer)
Original question:
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Message-Id: <9306220555.AA15108@bigbrother.matrox.com>
Subject: 2 Ethernet from a SS10 on Kalpana possible?
To: sun-managers@eecs.nwu.edu (Sun Managers)
Date: Tue, 22 Jun 1993 01:55:45 -0400
X-Mailer: ELM [version 2.3 PL11]
Status: OR
Hello,
We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
still use the same network (that is no subnet).
>From what I read on the archive, it does not seem to be possible because
SunOS will only use le0 in output.
Kalpana send me a configuration sheet regarding NC400 coprocessors, but
this is not our setup, we have only le0 and le1 interface.
They specify that each ethernet adapter should use a different MAC address.
This is configurable with ifconfig by using
8:0:20:1a:f5:a3 for le0 and
48:0:20:1a:f5:a3 for le1
I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
this issue.
Kalpana also say that internal routing should be disable on the SS10,
so I have to turn off IP_FORWARDING.
What about the IP address, can I use the same 138.11.48.214 on both
ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
and 138.11.48.215 for le1 instead?
Is using 2 NC400 different than using 2 lex interface in regards to routing?
Does somebody already try this?
Replies:
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: Ian Cassidy <pru!cassidyi>
> Real Page ,
Definently don't use the same IP address on both ehternet adaptors. This will cause duplicate IP address error message.
Is the bottleneck your encountering on your network due to traffic or the ethernet interface of the SS10 is unable to handle the amount of information being directed towards ?
There are 3 solutions.
1. maybe think about a subnet using the ss10 as the router (unless you already have a router)
2. look at using ATM to give you 50 MB/s to 10K Mb/s or FDDI, 100 Mb/s ethernet.
Ian Cassidy
Prudential Life and Pensions
Reading
U.K.
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: Michael Assels <mjassels@cs.concordia.ca>
You wrote:
=> We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
=> adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
=> the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
=> still use the same network (that is no subnet).
=>
=> >From what I read on the archive, it does not seem to be possible because
=> SunOS will only use le0 in output.
If you specify routes through le1, then that's where the packets will go.
Note that you'll need to specify routes to your local subnet as static
host routes, which may be a bit of a nuisance if there are many hosts on
the subnet.
=> Kalpana send me a configuration sheet regarding NC400 coprocessors, but
=> this is not our setup, we have only le0 and le1 interface.
Sorry. Can't help here.
=> They specify that each ethernet adapter should use a different MAC address.
=> This is configurable with ifconfig by using
=> 8:0:20:1a:f5:a3 for le0 and
=> 48:0:20:1a:f5:a3 for le1
=> I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
=> this issue.
That's what we do, and there are no problems. A warning: Make sure that
there is no time when the two interfaces are both up with the same ethernet
address, or you'll get a very confused switch when it sees the same address
on two ports. As far as I can tell, this means doing
ifconfig le0 138.11.48.214 netmask 255.255.255.0 broadcast ...
...
ifconfig le1 ether 48:0:20:1a:f5:a3
ifconfig le1 138.11.48.215 netmask 255.255.255.0 broadcast ...
in that order (i.e., the first thing you do with le1 is change its ethernet
address). Watch out for
vmunix: duplicate IP address!! sent from ethernet address: 0:0:f4:80:7:83
messages in /var/adm/messages. I believe these come during the very brief
time when le1 has no specified IP address. In any event they are only a
nuisance, not a real problem.
=> Kalpana also say that internal routing should be disable on the SS10,
=> so I have to turn off IP_FORWARDING.
This won't hurt, but it shouldn't even be necessary if only one subnet is
connected to the switch. (We have 4 subnets on the switch, so we do turn it
off.)
=> What about the IP address, can I use the same 138.11.48.214 on both
=> ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
=> and 138.11.48.215 for le1 instead?
You'll definitely want two IP addresses. I'm not sure exactly what would
happen with only one, but at very least you would would get multiple
responses to arp, which cause some confusion.
Good luck.
Michael
--
+-------------------------------+----------------------------------+
| Michael Assels, Analyst | Everything is what it is and not |
| Comp. Sci., Concordia Univ. | another thing. |
| 1455 de Maisonneuve O | - Aristotle (or someone else) |
| Montreal, Quebec, H3G 1M8 +----------------------------------+
| Voice: (514) 848-3035 | mjassels@cs.concordia.ca |
+-------------------------------+----------------------------------+
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: pbg@cs.brown.edu (Peter Galvin)
I have heard from Sun that indeed, you cannot do this because the Sun
will only use ethernet, even if two are connected. We solved this
problem by having multiple servers talking to multiple kalpanas, but
only one connection between any two boxes.
--Peter
------------------------------------------ --------------------------------
Peter Baer Galvin (401) 863-7623
Systems Manager, Brown U. CS Department pbg@cs.brown.edu
Box 1910 (115 Waterman Street) uunet!brunix!pbg
Providence, RI 02912 (02906) pbg@browncs.bitnet
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: hyder@niwot.scd.ucar.EDU (Paul Hyder)
The only thing that has been left out in your description is the
problem relating to the IP addresses on the two interfaces. The two
interfaces on the Sun MUST think they are on different >networks<. You
will need to set special subnet masks on this host or contrive a setup
with a different IP net or subnet address.
This need for different networks on interfaces produces a secondary
problem: routing. You will need to tell the Sun which route to use for
specific traffic. Adding static host routes is probably the best path.
In general remember that traffic for a specific destination will need
to always use a specific interface. Dynamic routing is likely to be
impossible.
If what you were expecting is the logical equivalent of 20Mbit Enet
then you are likely to be less than satisfied. If what you wanted is
two logically separate 10Mbit Enets then it works just fine.
Paul Hyder
Network Engineer
NCAR
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: Jeff Bacon <bacon@mtu.edu>
> We just received a Kalpana 1500C Etherswitch and I want to use 2 ethernet
> adapters (le0, le1) from the same SS10 (SunOS 4.1.3) on two ports of
> the Kalpana. This would give me 2x10Mb/sec throughput on the SS10 and
> still use the same network (that is no subnet).
>
> >From what I read on the archive, it does not seem to be possible because
> SunOS will only use le0 in output.
well, SunOS will only use one interface per subnet in output. it doesn't
necessarily have to be le0.
but you are basically correct, and hence the problem.
> They specify that each ethernet adapter should use a different MAC address.
> This is configurable with ifconfig by using
> 8:0:20:1a:f5:a3 for le0 and
> 48:0:20:1a:f5:a3 for le1
> I read that 802.3 permit to "or" 0x40 with the higher address bit to resolve
> this issue.
yup
doesn't really matter one way or the other so long as you grab some
ether addr that isn't in elsewhere on the subnet.
> Kalpana also say that internal routing should be disable on the SS10,
> so I have to turn off IP_FORWARDING.
shouldn't make any difference, cause the two ports are on the same subent
so it won't forward anyway.
> What about the IP address, can I use the same 138.11.48.214 on both
> ethernet adapter? (Probably not) Should I use 138.11.48.214 for le0
nope.
> and 138.11.48.215 for le1 instead?
yup.
this whole area - the idea of having two interfaces on the same subnet -
is, shall we say, an unknown. you're not supposed to do it. you're
not supposed to be able to do it. Unix doesn't know how to cope with it.
I have some other thoughts on the matter though. Currently working on
a paper that I'd like to present at this year's Usenix LISA conference
on a way around the problem.
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: Mike Raffety <miker@il.us.swissbank.com>
You have to tell the Sun (but NOTHING else) that the network is subnetted,
and have half of your clients on one subnet, and half the other (but DON'T
tell the clients about the subnetting, so they'll still talk directly to
each other, without a router).
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
From: stern@sunne.east.sun.com (Hal Stern - NE Area Systems Engineer)
while you can do this, you won't get the full benefit
of two ethernets for *output*. on input, you will.
you need to assign two IP addresses and two names
to the interfaces, just like using two nc400s. there
is no difference in the config (le, nc, doesn't matter;
they've just device names).
make sure that half of your clients use the name on le0
and half use the name on le1, otherwise all traffic will
come in on the le0 interface anyway.
--hal
Since then I received the following remark from Mike
Return-Path: <miker@il.us.swissbank.com>
Date: Mon, 19 Jul 93 11:16:46 CDT
X-Organization: Swiss Bank Corporation
To: Real.Page@matrox.com
Subject: Re: SUMMARY: 2 Ethernet from a SS10 on Kalpana possible?
BTW, subnet addresses of all-zeroes and all-ones are supposed to be
illegal, according to the RFC. I.e., if you subnet 138.11 with a
netmask of 255.255.128.0, you get NO usable subnets, because 138.11.0
has an all-zeroes (a single bit equal to zero) subnet, and 138.11.128
has an all-ones (a single bit equal to one) subnet. So you choose a
netmask of 255.255.192.0, which gives you four, minus two for the
all-zeroes and all-ones, a net of two subnets, with hosts numbered from
138.11.65-126, and 138.11.129-190.
It so happens Sun doesn't enforce this limit (yet!), but most router
manufacturers do, so you might want to keep it in mind for the future.
-- Real Page ! Real.Page@Matrox.COM Gerant des Systemes Informatiques ! (514) 685-7230 #2359 Systemes Electroniques Matrox ! (514) 685-7030 Fax Dorval, Quebec !
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:08:02 CDT