FC4 - NFS - Never had problems like this before

gmp · #1 9th July 2005, 06:38 AM

I want to start of by saying that we've never had NFS issues like this and we've put in many hours today trying to resolve this without any success... it's quite frustrating but I know one of the "really smart guys" here may have the solution for us. We had this all happily working before FC4 and now that we've upgraded I think we need to go back to linux admin school for a refresher...

In a nutshell we're having problems getting one machine to mount an NFS filesystem from another when everything appears to be configured correctly.

For the record: The firewalls are disabled on BOTH machines.

Here are the gorey details:

We have two machines, taxi and ignatowski.

On taxi we have a filesystem mounted at /mnt/x14 which we would like to export as an nfs filesystem and mount over on ignatowski at the same location, /mnt/x14.

Host files are configured.
/etc/exports on taxi is configured.
/etc/fstab on ignatowski is configured.

When we go to mount the filesystem we get:

mount: taxi:/mnt/x14 failed, reason given by server: Permission denied.

Now I've googled and searched all over and it seems this is a fairly common issue but none of the threads or discussions I have found indicate that anyone has found any sort of resolution for this so if I can get an answer that leads to a solution I will post everything we do to solve it for the sake of kharma.

Anyway... let's get the preliminaries out of the way...

Code:

[root@taxi ~]# ping ignatowski
PING ignatowski.gmp-vfx.com (100.100.100.21) 56(84) bytes of data.
64 bytes from ignatowski.gmp-vfx.com (100.100.100.21): icmp_seq=0 ttl=64 time=0.119 ms
64 bytes from ignatowski.gmp-vfx.com (100.100.100.21): icmp_seq=1 ttl=64 time=0.100 ms
64 bytes from ignatowski.gmp-vfx.com (100.100.100.21): icmp_seq=2 ttl=64 time=0.106 ms

okay... that works..

Code:

[root@taxi ~]# ssh ignatowski
root@ignatowski's password:
Last login: Fri Jul  8 21:27:17 2005
[root@ingnatowski ~]# ping taxi
PING taxi.gmp-vfx.com (100.100.100.20) 56(84) bytes of data.
64 bytes from taxi.gmp-vfx.com (100.100.100.20): icmp_seq=0 ttl=64 time=0.140 ms
64 bytes from taxi.gmp-vfx.com (100.100.100.20): icmp_seq=1 ttl=64 time=0.114 ms
64 bytes from taxi.gmp-vfx.com (100.100.100.20): icmp_seq=2 ttl=64 time=0.108 ms

okay.. that seems to work.

so authentication/host resolution is not an issue... or at least it appears not to be an issue.

next we'll take a look at the exports file:

Code:

#/etc/exports
/mnt/x14  ignatowski.gmp-vfx.com(rw,no_root_squash,sync)

and then have a look at the output of the following:

Code:

[root@taxi ~]# exportfs -ra
[root@taxi ~]# showmount -e
Export list for taxi.gmp-vfx.com:
/mnt/x14 ignatowski.gmp-vfx.com
[root@taxi ~]# cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/mnt/x14        ignatowski.gmp-vfx.com(rw,no_root_squash,sync,wdelay)

<SSH'ing over to ignatowski....>

Code:

[root@taxi ~]# ssh ignatowski
root@ignatowski's password:
Last login: Fri Jul  8 22:25:10 2005 from taxi.gmp-vfx.com

[root@ingnatowski ~]# cat/etc/fstab
-bash: cat/etc/fstab: No such file or directory
[root@ingnatowski ~]# cat /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
/dev/devpts             /dev/pts                devpts  gid=5,mode=620  0 0
/dev/shm                /dev/shm                tmpfs   defaults        0 0
/dev/proc               /proc                   proc    defaults        0 0
/dev/sys                /sys                    sysfs   defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

taxi:/mnt/x14           /mnt/x14                nfs     rw              0 0

/dev/fd0                /media/floppy           auto    pamconsole,exec,noauto,managed 0 0
/dev/hda                /media/cdrecorder       auto    pamconsole,exec,noauto,managed 0 0

[root@ingnatowski ~]# mount -a
mount: taxi:/mnt/x14 failed, reason given by server: Permission denied

Quite frankly I'm totally stumped... just so I know I'm not losing my mind we'll check to see that the daemons are running on both machines:

<ON IGNATOWSKI>

Code:

[root@ingnatowski ~]# ps -aef | grep portmap
rpc       2176     1  0 21:26 ?        00:00:00 portmap
root      7225  7171  0 22:34 pts/3    00:00:00 grep portmap
[root@ingnatowski ~]# ps -aef | grep nfs
root      2736     1  0 21:27 ?        00:00:00 [nfsd]
root      2737     1  0 21:27 ?        00:00:00 [nfsd]
root      2738     1  0 21:27 ?        00:00:00 [nfsd]
root      2739     1  0 21:27 ?        00:00:00 [nfsd]
root      2740     1  0 21:27 ?        00:00:00 [nfsd]
root      2741     1  0 21:27 ?        00:00:00 [nfsd]
root      2742     1  0 21:27 ?        00:00:00 [nfsd]
root      2743     1  0 21:27 ?        00:00:00 [nfsd]
root      7229  7171  0 22:34 pts/3    00:00:00 grep nfs
[root@ingnatowski ~]# ps -aef | grep rpc
rpc       2176     1  0 21:26 ?        00:00:00 portmap
rpcuser   2194     1  0 21:26 ?        00:00:00 rpc.statd
root      2239     1  0 21:26 ?        00:00:00 rpc.idmapd
root      2705     1  0 21:27 ?        00:00:00 rpc.rquotad
root      2745    15  0 21:27 ?        00:00:00 [rpciod/0]
root      2746    15  0 21:27 ?        00:00:00 [rpciod/1]
root      2747    15  0 21:27 ?        00:00:00 [rpciod/2]
root      2748    15  0 21:27 ?        00:00:00 [rpciod/3]
root      2750     1  0 21:27 ?        00:00:00 rpc.mountd
root      7234  7171  0 22:35 pts/3    00:00:00 grep rpc
[root@ingnatowski ~]# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  32768  status
    100024    1   tcp  32769  status
    100007    2   udp    798  ypbind
    100007    1   udp    798  ypbind
    100007    2   tcp    801  ypbind
    100007    1   tcp    801  ypbind
    100011    1   udp    774  rquotad
    100011    2   udp    774  rquotad
    100011    1   tcp    789  rquotad
    100011    2   tcp    789  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32771  nlockmgr
    100021    3   udp  32771  nlockmgr
    100021    4   udp  32771  nlockmgr
    100021    1   tcp  32783  nlockmgr
    100021    3   tcp  32783  nlockmgr
    100021    4   tcp  32783  nlockmgr
    100005    1   udp    816  mountd
    100005    1   tcp    828  mountd
    100005    2   udp    816  mountd
    100005    2   tcp    828  mountd
    100005    3   udp    816  mountd
    100005    3   tcp    828  mountd
    300019    1   tcp    830  amd
    300019    1   udp    831  amd
  [root@ingnatowski ~]# exit
logout
Connection to ignatowski closed.

<OVER ON TAXI>

Code:

[root@taxi ~]# ps -ef | grep nfsd
root      4680     1  0 22:14 ?        00:00:00 [nfsd]
root      4681     1  0 22:14 ?        00:00:00 [nfsd]
root      4682     1  0 22:14 ?        00:00:00 [nfsd]
root      4683     1  0 22:14 ?        00:00:00 [nfsd]
root      4684     1  0 22:14 ?        00:00:00 [nfsd]
root      4685     1  0 22:14 ?        00:00:00 [nfsd]
root      4686     1  0 22:14 ?        00:00:00 [nfsd]
root      4687     1  0 22:14 ?        00:00:00 [nfsd]
root      5043  3625  0 22:36 pts/2    00:00:00 grep nfsd
[root@taxi ~]# ps -ef | grep portmap
rpc       2205     1  0 21:18 ?        00:00:00 portmap
root      5047  3625  0 22:36 pts/2    00:00:00 grep portmap
[root@taxi ~]# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  32768  status
    100024    1   tcp  32769  status
    100004    2   udp    784  ypserv
    100004    1   udp    784  ypserv
    100004    2   tcp    787  ypserv
    100004    1   tcp    787  ypserv
 600100069    1   udp    792  fypxfrd
 600100069    1   tcp    794  fypxfrd
    100007    2   udp    803  ypbind
    100007    1   udp    803  ypbind
    100007    2   tcp    806  ypbind
    100007    1   tcp    806  ypbind
    300019    1   tcp    921  amd
    300019    1   udp    922  amd
    100011    1   udp    619  rquotad
    100011    2   udp    619  rquotad
    100011    1   tcp    635  rquotad
    100011    2   tcp    635  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100021    1   udp  32795  nlockmgr
    100021    3   udp  32795  nlockmgr
    100021    4   udp  32795  nlockmgr
    100021    1   tcp  32856  nlockmgr
    100021    3   tcp  32856  nlockmgr
    100021    4   tcp  32856  nlockmgr
    100005    1   udp    640  mountd
    100005    1   tcp    652  mountd
    100005    2   udp    640  mountd
    100005    2   tcp    652  mountd
    100005    3   udp    640  mountd
    100005    3   tcp    652  mountd

Have I lost my mind??? I've been setting up NFS for a decade and I've never had this problem before.

HEeeeeLLLLLP!

gmp

gmp · #2 9th July 2005, 05:42 PM

Nothing? Nobody?

I guess that means it's not something obvious I overlooked.

Hmmm....

SlowJet · #3 9th July 2005, 06:03 PM

I've never setup NFS before but I read through your stuuf and read the how to's and help on the gui tool.

There is only 3 things I could even guess at.

1 no_root_squash - maybe you need a real NFS user account(s) with chown nfs:nfs for running it and then some real group type permissions set due to how SELinux is setup?
2. SElinux is blocking something - tail /var/log/audit/audit.log and other logs (messages, nfs log?)
3. There are some some bugs outstanding - update everything for FC4, get the new nfstools, policy, network?

Other than that, it seems to be ok.

SJ

gmp · #4 9th July 2005, 06:06 PM

SELinux is not enabled and the firewall is not enabled either.

I am also not using the hosts.allow or hosts.deny (since they are a big freaking security risk and anyone with half a brain has a real firewall for that crap anyway).

g

SlowJet · #5 9th July 2005, 06:18 PM

Well, I thoht you might say something like that so if it isn't permisions or a bug, it must be Karma.

SJ

eidt: check this out "nfsnobody group and user id 65534"

gmp · #6 9th July 2005, 07:09 PM

Now that doesn't really help now does it?

g

SlowJet · #7 9th July 2005, 07:14 PM

No, I think you are the reident NFS tester for FC4.

SJ

jcliburn · #8 9th July 2005, 07:30 PM

This is a total shot in the dark, but... On the client you might try forcing NFS3 using nfsvers=3 in the mount options, just to see if it makes any difference. See the nfs manpage for details (as if you haven't already).

gmp · #9 9th July 2005, 07:51 PM

jcliburn,

good idea but I hhad tried that earlier this morning...

Here's what our fstab on ignatowski lookes like:

Code:

taxi:/mnt/x11           /mnt/x11                nfs     rw,vers=3,hard  0 0
taxi:/mnt/x12           /mnt/x12                nfs     rw,vers=3,hard  0 0
taxi:/mnt/x14           /mnt/x14                nfs     rw,vers=3,hard  0 0
taxi:/mnt/x15           /mnt/x15                nfs     rw,vers=3,hard  0 0

jcliburn · #10 9th July 2005, 08:12 PM

I got it working on my 2 FC4 boxes, kite and osprey. NFS server is kite, client is osprey.

On kite (server)

Code:

[root@kite ~]# cat /etc/exports
/opt            osprey(rw,sync)
[root@kite ~]# showmount -e
Export list for kite:
/opt osprey

on osprey (client)

Code:

[root@osprey ~]# mount -t nfs kite:/opt /kite/opt -o nfsvers=3
[root@osprey ~]# cd /kite/opt
[root@osprey opt]# ls
java  rpm  src  THIS_FILE_ON_KITE

It takes a good minute or two for the mount to take place, but it eventually completes.

gmp · #11 9th July 2005, 08:27 PM

I'm doing the same here.

On taxi - the nfs server:

Code:

[root@taxi ~]# cat /etc/exports
/mnt/x14  ignatowski(rw,sync)
[root@taxi ~]# showmount -e
Export list for taxi.gmp-vfx.com:
/mnt/x14 ignatowski.gmp-vfx.com

but I get a different end result...

on ignatowski - the nfs client:

Code:

[root@ingnatowski ~]# mount -t nfs taxi:/mnt /mnt -o nfsvers=3
mount: taxi:/mnt failed, reason given by server: Permission denied

I'm absolutely dumbfounded...

jcliburn · #12 9th July 2005, 08:35 PM

What happens if you change your mount command to look like this:

Code:

mount -t nfs taxi:/mnt/x14 /mnt -o nfsvers=3

gmp · #13 9th July 2005, 08:38 PM

same thing:

Code:

[root@ingnatowski ~]# mount -t nfs taxi:/mnt/x14 /mnt/x14 -o nfsvers=3
mount: taxi:/mnt/x14 failed, reason given by server: Permission denied
[root@ingnatowski ~]# mount -t nfs taxi:/mnt/x14 /mnt -o nfsvers=3
mount: taxi:/mnt/x14 failed, reason given by server: Permission denied

and just as another sanity check:

Code:

[root@ingnatowski ~]# cd /mnt
[root@ingnatowski mnt]# ls -ltrh
total 20K
drwxrwxrwx  2 root root 4.0K Jul  8 15:37 x15
drwxrwxrwx  2 root root 4.0K Jul  8 15:37 x12
drwxrwxrwx  2 root root 4.0K Jul  8 15:37 x11
drwxrwxrwx  2 root root 4.0K Jul  8 15:37 x10
drwxrwxrwx  2 root root 4.0K Jul  9 10:35 x14

jcliburn · #14 9th July 2005, 08:39 PM

Wow. Are you absolutely certain your firewalls are down? (Mine are.) iptables -L on both machines shows no active rules?

gmp · #15 9th July 2005, 08:42 PM

aw crap.. you're not going to believe this:

Code:

[root@taxi ~]# iptables -L
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination



[root@ingnatowski mnt]# iptables -L
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Does this mean they are actually up or is this what I should be seeing.