Hello everybody,
I owe a summary to both "sun-managers" and "veritas-vx" list members since I
posted to question (well, with some differences) to both lists, which are
excellent sources of information and help. I'd like to thank all those who
helped on both lists, specially:
Martin Oksnevad
Doug Hughes
Lupe Christoph
Buddy Lumpkin
Brett Lymn
Michael Stapleton
Mike Salehi
Michael Key
Who provided me with detailed insight and pointers to tools/docs. THANKS
GUYS!
QUICK ANSWER:
The page in/out shown in vmstat is not necessarily a bad thing. I should be
worried about the "scan rate" (sr) field, which if goes hi, is an indication
of memory starvation (in my case it is almost always zero). So right now
I'm not worried about performance neither with applications nor with RAID
configuration (well, of course disk access performance is something that I'm
working on separately). * detailed discussions + links follows. Specially
check out the link at the bottom! *
ORIGINAL POST (Sent to sun-managers), a similar post with veritas concerns
sent to veritas-vx:
>
> Greetings Sun Managers,
>
> On an E450/1GB RAM/Solaris 2.6/recommended patch/Veritas 3.0.1 I see this
> strange paging activity:
>
> Vmstat 5 shows almost 100+ page outs and 30+ page ins on every row. Sum
of
> RSS of all the processes is about 700MB,
> so I assume I have about 300MB of free RAM and no paging should occur.
Sum
> of SIZE is 1.5GB if that matters.
>
> I have not included the details of the products it runs, since in the
first
> place, regardless of the products, how can this happen, and where should I
> look into?
DETAILED DISCUSSIONS (I have edited and ommited redundant parts of answers):
1GB memory doesn't give you 1GB available memory when running the system.
If you boot your system in single user mode without swap partition(s)
you will discover that you have only 800-850mb memory available
(use swap -s), the rest (150-200mb) is reserved by the system.
So with 700mb of processes and only 800mb available memory you can expect
paging.
We have a 4-CPU 4GB E450 and we only have 3.5GB memory available,
ie. more than 600mb is reserved by the system. Less than 90% of
memory you add to a Sun system becomes available to user processes.
My suggestion is to add minimum 512mb memory to your system.
If you have only one 1GB memory kit (4*256mb SIMMs) you should consider
a second one 1GB memory kit if you want memory interleaving and can
afford 1GB memory. If you have two 512mb memory kits (4*128MB SIMMs)
you may want to add two 512mb memory kits to get memory interleaving.
Martin
--------------------------------------------------------------------
page in's and outs are not an indicator of a memory starved system. In fact,
any time you use mmap() it uses the pager. This means that when you simply
copy
a file page in's and outs will go crazy (truss cp if you want to see cp's
use
of mmap()).
The only real indicator of a memory starved system is by paying attention to
the sr field in vmstat. If the sr field is consistently non-zero, then you
need
more memory ... period.
The sr field refers to a clock algorithm where the first hand sets a
reference
bit and the second hand clears it if it has been used before the second hand
gets there, if not, the page moves out to swap.
~Buddy
----------------------------------------------------------------------
> Sum of RSS of all the processes is about 700MB,
This is not the correct thing to to - the number you get here will
just about always bee too high. The RSS number includes shared text
pages from shared libraries - each process will have these counted in
their RSS but there really is only one copy of the pages in memory.
Brett
----------------------------------
With vmstat look at the sr "scan rate". over 200 for extended period of
time indicates a need for more ram. Minutes to houres. Although if you
have ALOT of file i/o the scan rate can be high. You can use iostat -xpnc
to look at disk activity. The paging activity to the swap partition gives
you and indication of application paging. You should look into the
memtool, you can get it at http://playgroung.sun.com/pub/memtool. It give
you a true idea of your memory usage. Also read the vmsizing.ps.
Michael
---------------------------------------------------------------------
Manesh, get the SE programs and see if there is an issue, there is a
pointer to it on sunworld.com.
Mike
---------------------------------------------------------------------
The major issue with Raid 5 in general is really with write operations. If
your application does heavy writes Raid 5 is not a good solution, a better
solution would be Raid 0+1 or 1+0, if you do mostly reads from your Raid 5
volumes you should actually get pretty good performance.
Michael Key
----------------------------------------------------------------------
I identify 3 separate issues.
1) if you've got VxVM > 3.0 it's perfectly safe and reasonably to
convert your raid5 to raid-1+0 using vxassist with relayout
2) If you've got an IO issue, it should show up as I/O or wait time
and not excessive paging. You should also see long response times when
using iostat (responses for queries)
3) If you've got a paging issue, you should be able to see some wait
and system time (when looking at CPU utilization.) What do you count
as excessive paging? Do you have numbers? Have you checked the percentage
of memory used by your apps? RSS vs SIZE?
paging and I/O are probably not related though unless you are paging
TO your RAID-5 (which would certainly be a bad idea)
Doug
---------------------------------------------------------------------------
> > 2) If you've got an IO issue, it should show up as I/O or wait time
> > and not excessive paging. You should also see long response times when
> > using iostat (responses for queries)
I/O wait just means that a processor is idle while I/O is going on.
This is not bad. It just means a lightly loaded machine. If iostat -x
shows %w (*disk* wait), this means overloaded disks.
> Well, I see an average of 25-30% busy time on one of my hard drives, so I
> presume it is quite busy. Service time is also high (around 15-20ms) for
> that specific disk. But if it is caused by the RAID 5 configuration,
> shouldn't I see more than one busy disk?
What are you using to look at the disks? iostat -x?
Is there no load on the machine, so you do not expect to see *any*
activity? Or is there load, and you see only one disk in the RAID 5
as active, as opposed to seeing all of them busy?
If you have load, 25~30% is not "high", nor is a service time
of 15~20 ms.
Perhaps you can explain the layout in a little more detail (vxprint...)
and include iostat -xn output.
> > 3) If you've got a paging issue, you should be able to see some wait
> > and system time (when looking at CPU utilization.) What do you count
> > as excessive paging? Do you have numbers? Have you checked
> > the percentage
> > of memory used by your apps? RSS vs SIZE?
> I have numbers that are not that high per se, but I see them consistently.
> Sum of RSS on all of my processes stays around 700MB. This is while I
have
> 1GB of RAM. Sume of SIZE is around 1.5GB, but I'm not worried about that.
> With a total RSS of 700MB and 1GB of physical RAM, do you expect to see
even
> one page out? vmstat 5 shows me an average of 100+ page outs and 30+ page
> ins. I do not think this is healthy.
Again, if you do not have any activity, this is suprising. If you do,
page in/out are OK. Most I/O goes through the paging mechanism,
so you see it as page in/out.
If your system is memory starved, you see the scan rate (vmstat 'sr'
column) go up. Triple digit scan rates are really bad. Short bursts
are still ok, especially when programs start.
Unless you see strange load characteristics on your disks, I presume
you get better support from the sun-managers mailing list or the
solaris newsgroup. You may also want to check out Adrian Cockcrofts
book or FAQ:
http://www.sunworld.com/common/cockcroft.letters.html
Check the links at the bottom of this page.
Lupe
(And here's Doug's reply to part of comments from Lupe:)
>I/O wait just means that a processor is idle while I/O is going on.
>This is not bad. It just means a lightly loaded machine. If iostat -x
>shows %w (*disk* wait), this means overloaded disks.
>
I/O wait indicates many different things. It can mean you're doing a backup
(tape drives yield tremendous I/O wait CPU indication), disk is busy doing
I/O, paging, etc. It can mean anything from a lightly loaded machine to
a heavily loaded machine.
Note, I'm not saying that I/O wait means that disks 'are' overloaded. I am
saying that overloaded disks will often show up in I/O wait. There is
a difference. It's not a symmetrical relationship. More than 25%
(consistent)
I/O wait deserves at least a little bit followup to see what's going on,
in my experience. It could be just a backup, or tape rewinding/fsf, or just
normal heavy/light load from lots of disk activity.
Doug
===============================================
And of course, almost everybody mentioned Adrian Cockroft's famous book (I
enjoyed the first edition very much, but I have not had a chance to go over
the second edition), and also Douge mentioned Brian Wong's book. I wanted
to put pointers to these books here, but I found a better link that you'll
enjoy:
http://www.sun.com/sun-on-net/performance.html
> Nasser K. Manesh
> UNIX System Administrator/Webmaster
> Penske e-Business Group
> Email: nasser.manesh@penske.com
> Voice: (610) 796-6527
> Fax: (610) 796-4387
>
>
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:14:09 CDT