SUMMARY: ZFStorage 7120: too slow NFS performance

gottit...

Some late-2010 answers, and (thanks to several good intentions that
started Jan-1 00:00:01) more early-2011 answers, but nearly concentrated on
network latencies such as on a busy shared LAN.

However, our tests were done over a dedicated point-to-point Ethernet
cable between ZFStorage server and NFS client.

Others pointed me to the "Analytics" functionality in the ZFStorage web
gui, which clearly uses DTrace to capture counters and so. 

However the "Analytics" didn't provide me a means to view "F_SYNC" calls
that might be requested by the NFS clients.


I started to wonder if the "chown -R" duration (to change 28000) files
would maybe still be ok, taking approx 9-12 msecs per NFS_SETATTR call.

Focusing again on the slow login (or Firefox startup) from a Ubuntu
NFS-client into a GNOME desktop, I noticed that crucial processes where
waiting *long* time for locks...

Searching further for NFS and locks relations, provided me the solution :

On the NFS-client, mount using the "nolock" option ; according to the man
page:

"lock/nolock : Selects  whether  to use the NLM sideband protocol to lock
files on the server. If neither option is specified (or if lock is
specified), NLM locking is used for this mount  point. When using the
nolock option, applications can lock files, but such  locks provide
exclusion only against other applications running on the same
client...Using  the nolock  option  is  also  required when mounting
exports on NFS servers that do not support the NLM protocol."

Did so, login and launch of GNOME apps and Firefox happens light-speed
fast.


thx all for participating in this quest!

Rob


On Fri, 31 Dec 2010 13:07:16 +0100, Rob De Langhe
<rob.de.langhe@twistfare.be> wrote:
> all,
> 
> continuing our quest for better NFS performance out of a ZFStorage 7120
> array:
> 
> Testing from a Linux client (CentOS 4.5) with a dedicated 1Gbit link to
> the ZFStorage array;
> the ZFSstorage provides an NFS-share of a (duh) ZFS filesystem which is
> configured on a pool of double parity RAID and *without* (ZIL) logging.
> 
> - the performance is near network-speed when accessing a huge NFS share
> with reads only (10GB file in 115secs, or 91 MBytes/sec = 728 Mbit/sec),
or
> when writing a 10 GB chunk (180 secs, or 57 MBytes/sec = 456 Mbit/sec)
to
> it
> 
> - when I run "du -sk" from the client of 24GB NFS-mounted directory
> containing 28000 files, it takes only 5 secs, so very good, but that's
of
> course not doing any real data transfer (just reading directories)
> 
> - when I really read data with "tar cf - $nfsdir | wc" from the client
of
> these 24GB NFS-mounted data, it takes about 391 secs or 61 MBytes/sec (=
> 488 Mbit/sec, not too bad)
> - doing exactly the same operation on the ZFStorage itself, it takes
only
> 112 secs or 214 MBytes/sec (with single-parity, that's reduced to 25
secs)
> 
> - the performance is simply bad when doing many updates like a simple
> "chown -R NNN" in the NFS mount that contains approx 28000 files : 5-6
> minutes to modify all these 28000 files
> - when I do exactly that same "chown -R NNN" command on the ZFStorage
> itself, the command is completed within half a second (ok, sounds like a
> pure-cache operation)
> 
> - also, when mounting a directory from the ZFStorage as home-dir for one
> of the users on the Linux client, the login takes 1-2 minutes before
done,
> and launching Firefox on the client takes 4-5 minutes before done !
> Needless to say that doing the same with a local home-dir on the Linux
> client, everything gets done in a few seconds
> 
> I tried all the same with Solaris(-8) clients, or CentOS 5.5 clients,
with
> all relevant alternative NFS-mount options on the clients (such as
> rsize=32768, wsize=32767, async, noatime, proto=udp) but no
accelleration
> possible.
> 
> Reconfiguring the pool on the ZFStorage towards a single-parity "small
> stripe" gave little or no improvement of the performance of the "chown
-R"
> test.
> 
> I read some pages on the net discussing a potential FSYNC that would be
> requested by the NFS client after each attribute change (like the
chown),
> but can't find any indication of FSYNC in "snoop" on the ZFStorage or in
a
> "truss" on Solaris-clients or "strace" on Linux-clients that proofs
this.
> 
> Also, that would not explain why logging in and reading plus updating
> several small files takes such a long time.
> 
> When we did the same in another environment with a Solaris NFS-server
> providing home dirs, no such delays with logins or Firefox startup
existed,
> so another configuration must be possible on this ZFStorage to achieve
> similar performance.
> 
> 
> thx in advance for any 2010 feedbacks !
> Rob
> _______________________________________________
> sunmanagers mailing list
> sunmanagers@sunmanagers.org
> http://www.sunmanagers.org/mailman/listinfo/sunmanagers
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers