Re: FreeBSD 9.2 NFS server and Mac 10.8/7/6 clients s-l-o-w finder

Subject: Re: FreeBSD 9.2 NFS server and Mac 10.8/7/6 clients s-l-o-w finder
From: Eric Browning <email@hidden>
Date: Wed, 16 Oct 2013 09:53:57 -0600

Thanks for the reply Guy,

I read through the FreeBSD list link you sent and experimented with the settings and it unfortunately produced no difference in performance.

Below are my settings for /etc/nfs.conf on my 10.8 client. I know I'm setting some default values like tcp and vers=3, just wanted to rule them out.

# Generated automatically by NFS Manager 2013-10-16 14:39:45 +0000

nfs.client.allow_async = 1

nfs.client.mount.options = noatime,tcp,vers=3,rsize=32768,wsize=32768,rdirplus,locallocks,inet,nfc

nfs.client.statfs_rate_limit = 1

nfs.client.is_mobile = 0

I have attempted to set nfs.client.nfsiod_thread_max = 1 and the default of 16, neither has made an impact on performance.

I have been logged on for only 46 minutes and this is what my nfsstat -c output looks like. The FreeBSD-fs list is correct that the mac clients produce a heck of a lot of getattr and access requests.

Client Info:

RPC Counts:

Getattr Setattr Lookup Readlink Read Write

815647 1628 7750 10 8327 9488

Create Remove Rename Link Symlink Mkdir

884 562 468 44 3 2

Rmdir Readdir RdirPlus Access Mknod Fsstat

4 0 966 477134 0 2329

Fsinfo PathConf Commit

1 1 0

RPC Info:

TimedOut Invalid X Replies Retries Requests

0 0 0 0 1325248

Cache Info:

Attr Hits Misses Lkup Hits Misses BioR Hits Misses

6419016 28287 242318 7156 2181527 8130

BioW Hits Misses BioRLHits Misses BioD Hits Misses

39559 9488 1572 10 114987 2290

DirE Hits Misses

35356 14

On Wed, Oct 16, 2013 at 9:09 AM, Guy Helmer <email@hidden> wrote:

On Oct 15, 2013, at 7:30 PM, Eric Browning <email@hidden> wrote:

I'm currently debugging slow NFS performance on mac clients. Running FreeBSD 9.2 stable with a mix of 200 Mac OS X 10.8, 10.7 and 10.6 clients and NFS is configured with 256 threads, nfs3, tcp. The server is pretty beefy, 16 cores, 64GB RAM, pure SSD storage pool. Usage of the SSDs is only 7% busy at it's busiest.

The biggest issue is when the finder attempts to show the contents of a folder it takes 25 seconds or longer for anything to appear. With Terminal however it appears instantly under any load condition when using ls -l. Home directories are in /Network/Servers and listed as such in open directory Mounts.

Periodically I get this error on the mac side when listing directories in finder:

automountd[1766]: set_and_fake_mapent_mntlevel: subdir=/Contents error: Contents not found in map=-fstab

Not sure what to do about the above error. Apple says fstab is deprecated but it can be edited with vifs. FreeBSD doesn't user UUID's but fstab expects a UUID map... I am at my wits end with this problem. do I need to change to a different mount point like /net instead?

I'm also wondering with 200 clients with five nfsiod threads (~1000 nfsiod threads total) am I oversubscribing the nfs server? I have changed the client macs nfsiod threads down to 1 each to see if this is indeed the issue tomorrow but have this nagging feeling it won't change. The freebsd list says most of my RPC is Access calls (getattr, etc) and will only use the threads when writing and for readahead. I have the mac nfs.conf, and auto_master listed below. I'm also attempting async with this nfs conf tomorrow as well. I definitely think it's load based because it gets slower and slower as people logon but the server isn't being overtaxed that I can tell.

The reason for the client mount option of "nfc" option comes from this archlinux article about macs and NFS: https://wiki.archlinux.org/index.php/NFS_Troubleshooting

---nfs.conf---
# Generated automatically by NFS Manager 2013-10-14 16:50:55 +0000

nfs.client.allow_async = 1
nfs.client.nfsiod_thread_max = 1
nfs.client.mount.options = nfc,noatime,vers=3,rsize=32768,wsize=32768,rdirplus,locallocks,inet
nfs.client.statfs_rate_limit = 1

nfs.client.is_mobile = 0

---auto_master---

#
# Automounter master map
#
+auto_master # Use directory service

/net -hosts -nobrowse,hidefromfinder,nosuid

/home auto_home -nobrowse,hidefromfinder
/Network/Servers -fstab -nobrowse,hidefromfinder

/- -static

Thanks in advance,
--
Eric Browning
Systems Administrator
801-984-7623

Hi, Eric. I'm not sure if it would help, but there has been some talk on the FreeBSD dev lists in the past couple of months about raising the FreeBSD dirhash settings to improve performance http://lists.freebsd.org/pipermail/freebsd-stable/2013-August/074964.html

One suggestion is to increase vfs.ufs .dirhash_reclaimage from 5 to 60, and another is to raise the vfs.ufs.dirhash_maxmem, perhaps by a factor of 10 from its default.

I found another suggestion relayed from James Gosling regarding file locking over NFS: https://blogs.oracle.com/jag/entry/nfs_on_snow_leopard

It would be interesting to hear if any particular change improves performance.

Guy

Eric Browning
Systems Administrator
801-984-7623

Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-kernel mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden

References:
	>FreeBSD 9.2 NFS server and Mac 10.8/7/6 clients s-l-o-w finder (From: Eric Browning <email@hidden>)

Prev by Date: FreeBSD 9.2 NFS server and Mac 10.8/7/6 clients s-l-o-w finder
Next by Date: Programmatic Interface to pf (firewall)
Previous by thread: FreeBSD 9.2 NFS server and Mac 10.8/7/6 clients s-l-o-w finder
Next by thread: Programmatic Interface to pf (firewall)
Index(es):
- Date
- Thread