I haven't had a chance to fully check out all of the following info but
as promised here the summary! I haven't had much of a chance to look
all of this over yet.
My original question:
I think this is a 2 part question regarding benchmarking new and current
systems. I am looking for info or pointers to info on 1-understanding
the different types of benchmarking and what all of the numbers mean and
2-testing and benchmarking my current systems/network to see where I'm
at.
I have 2 seperate areas I am interested in. First is networking/IO
speed. I know the actual network topology is something seperate so I
guess I'l be focusing on internal buss/IO speed for things like NFS and
web activity. Second is processing power/buck. I'm not trying to
compete with super computers but I want to know what makes a Sparc or
alpha better than a PC (or is it?).
I'm looking for info on what are the benchmarks that manufacturers use;
what do they mean; how does it all fit together...ie. one machine has a
higher megaflop rating than another but is still wholly slower.
I realize most of this means understanding the architecture of the
machine. I'm looking for some way to understand the differences in
speed performance and price.
Some of my biggest questions are: when is it better to get a
supercharged PC and run solaris x.86 or when to buy an ultra; should I
put more memory in or get a faster/better cpu; how do I compair one
machine to another overall?
------------------------------------------------------------------------
Thanks to:
-----------------------
Rich Kulawiec <rsk@itw.com>
First off, here are two good places to start:
http://hpwww.epfl.ch/bench/bench.FAQ.html Benchmarks FAQ
http://hpwww.epfl.ch/bench/SPEC.FAQ.html SPEC FAQ
Secondly, here's my take on benchmarks, based on 18 years of experience,
16 of them on the Arpanet/Internet in a variety of capacities.
They're almost meaningless.
Let me explain that. First of all, Moore's Law says that computing bang
for the buck doubles every 18 months. That's probably an underestimate;
computing horsepower is now amazingly cheap compared to other costs:
administration, programming, maintenance, software, etc. But it's
sometimes more visible than those costs, and so folks spend lots of
time trying to minimize it while running up their costs in the other,
less-visible areas.
Second, benchmarks are (as you probably know) highly subjective measures
of system performance. Database vendors are notorious for tuning
systems for their product, then running benchmarks against their
competitors.
Even third-party benchmarks, such as those done by magazines who are
assumedly trying to be objective, frequently suffer from unstated bias.
The problem with all these measurements is that they attempt to
characterize
the performance of complex systems (CPU, memory, disk, bus, network,
etc.)
as a single number -- and that oversimplication loses too much
information
to be useful. Even benchmarks targeted at a specific application area
often wind up comparing apples and oranges, e.g. web servers on
different
platforms. This is darned unfortunate, because what it means is that
the only real way to understand aggregate system performance is to
gain experience with it --- it'd be a lot easier if we could just
consult
a table of numbers and be done with it. ;-)
Third, benchmarks are frequently run on misconfigured or untuned
systems.
I'm not talking optimized-within-an-inch-of-its-life; I'm talking basic,
first-order attempts to make machines run reasonably well. I've seen
benchmarks run on Sun Sparc-10's with 16M of memory, which is a
pathetically
tiny amount; I've seen SunOS tests run using the generic kernel, which
includes support for everything under the sun (no pun intended) and
which
can be stripped to something much more reasonable with a few minutes
work.
And so on -- and of course, vendors echo this gripe, although *their*
idea of useful tuning goes quite a bit further than mine.
Fourth, software is rapidly outpacing hardware. For example, some of
the early versions of Java were clunky -- leading some of the industry
pundits to scoff at it. But it's getting faster awfully quickly; it'll
downright speedy before the winter snow melts. A Java benchmark run
today
under Solaris 2.5 on a Sparc 5 will be essentially meaningless by March.
Not only that, the choice of software has a lot to do with performance:
a Pentium box with reasonably fast memory and disk, running BSDI's
web server setup (properly configured), can absorb a million hits a day.
The same box running NT will melt if you try this stunt.
Bottom line? Assume that hardware fast enough for what you need will
be there, because unless you are doing some *awfully* specialized work,
it will be. (I have done some awfully specialized work, i.e. medical
image processing with *large* datasets; even then 90% of the problems
could be solved with off-the-shelf hardware and software.) Focus
instead
on issues like administration, scalability, software cost,
configurability,
and so on, because you can easily spend a lot more money there --
potentially
on the wrong things. Make sure your solution is based on open systems:
that means Java, not ActiveX; TCP/IP, not IPX; SMTP, not Microsoft Mail.
(Part of the reason for this last item is that it will let you upgrade
your hardware when you feel like it, not when a vendor who sold you
a proprietary solution gets around to supporting a new platform.)
I now relinquish the soapbox. :-)
-------------------------------------------
em@icess.ucsb.edu (Ed Mehlschau)
As a partial answer to your questions, check out the "Performance
Database
Server" on the web. Sorry, the URL is not handy.
-------------------------------------------------------
robin.landis@imail.exim.gov
You might find these references interesting.
http://www.sun.com:80/sunworldonline/swol-08-1996/swol-08-perf.html
http://www.sun.com:80/sunworldonline/swol-09-1996/swol-09-webbench.html
--
***************************************************************
Systems Administrator
-----------------------
Space Sciences Building
Vic Germani Cornell University
(607)-255-3434 402 Space Sciences
sysadmin@astrosun.tn.cornell.edu
***************************************************************
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:11:18 CDT