SUMMARY: heavy paging

Hi All,

I would like to thank all who answered and explain that sched is just
waiting for the disk and should not be blame on. Especial thanks to Gary
Paveza who payed my attention on high fpi which means an active reading from
the file system.

iosnoop from DtraceToolkit showed me a process which was reading a lot small
files. Files were moved to CFS and the problem has been solved.

Thanks again and best regards,
es

>
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 16 Mar 2010 18:08:11 +0100
> From: Eugenia Shuiskaya <redtigra@gmail.com>
> Subject: heavy paging
> To: sunmanagers@sunmanagers.org
> Message-ID:
>        <5c0b76411003161008w7637b6f8r3dbbcb681d26e5f2@mail.gmail.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi All,
>
> We run into heavy I/O on our zfs disk and could not find the reason.
> Solaris
> 10 10/09.
>
> Here is zfs info:
>
>  pool: rpool
>  state: ONLINE
> status: The pool is formatted using an older on-disk format.  The pool can
>        still be used, but some features are unavailable.
> action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
>        pool will no longer be accessible on older software versions.
>  scrub: none requested
> config:
>
>        NAME          STATE     READ WRITE CKSUM
>        rpool         ONLINE       0     0     0
>          mirror      ONLINE       0     0     0
>            c1t0d0s0  ONLINE       0     0     0
>            c1t1d0s0  ONLINE       0     0     0
>
>
> dtrace script iotop shows:
>
> # iotop  d sd0  Co **
>
>  UID    PID   PPID CMD              DEVICE  MAJ MIN D         DISKTIME
>
>    0      0      0                  sd0      32   0              60104
>
>    0      0      0 sched            sd0      32   0 W          4554988
>
>
> 2010 Mar 16 17:51:16,  load: 3.25,  disk_r:      0 KB,  disk_w: 233648 KB
>  UID    PID   PPID CMD              DEVICE  MAJ MIN D         DISKTIME
>    0      0      0 sched            sd0      32   0 W          4864154
>
>
> I run whoispaging.d ang got the result:
>
>
> # ./whoispaging.d
> Who's waiting for pagein (milliseconds):
> Who's on cpu (milliseconds):
>  ScriptAgent                                                     20
>  Supervisor                                                      24
>  dtrace                                                          28
>  psSTAT64_10                                                     33
>  data                                                            51
>  dmh                                                             65
>  fsflush                                                        178
>  rdh                                                           1999
>  rlh                                                           6515
>  sched                                                       217344
>
>
> I assumed that the virtual memory is low, but prstat shows I'm wrong:
> #prstat -Z...
> ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
>     0      188   40G   38G    59%   4:11:50 3.4% global
>
> # vmstat -p 3
>     memory           page          executable      anonymous
> filesystem
>   swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo
> fpf
>  68783448 43892160 235 226 22 0 3    0    0    0    0    0    0 1531   23
> 22
>  41939512 15828032 298 871 0 0  0    0    0    0    0    0    0  868    0
> 0
>  41902168 15797808 197 123 0 0  0    0    0    0    0    0    0 1312    0
> 0
>  41911104 15782904 320 3 0  0   0    0    0    0    0    0    0 2859    0
> 0
>  41910920 15759696 232 177 0 0  0    0    0    0    0    0    0 1007    0
> 0
>  41823336 15734752 212 6 0  0   0    0    0    0    0    0    0 2132    0
> 0
>  41779384 15712160 177 4 0  0   0    0    0    0    0    0    0 1697    0
> 0
>
> A lot of free memory - and high fpi. It's paging a lot.
>
> And I cannot understand, how I could find the reason of such heavy paging.
> Any advise, thought, opinion would be very appreciated.
>
> Thanks a lot and best regards,
> Evgenia
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers