autofs on foreign host fails to mount F16 NFS share

[KevinDrums]

I have a Mandriva 2010.2 system which can NFS mount all of my other Mandriva systems using autofs to do that. From a command line all I need do is, for instance

ls /net/machine1/home/

will list the contents of /home on the host machine1. On my F16 system I can run

ls /net/F16machine/home

and get a listing of /home on the F16 system itself. I believe that NFS is working because I can run

sudo mount -t nfs F16machine:/home /mnt/F16mount

on a Mandriva machine and then list the contents of /mnt/F16mount which will show the contents of /home on F16machine.

I can get autofs on the F16machine to negotiate the NFS mount to my Mandriva machines just fine.

But, when I try to get autofs to negotiate the NFS mount from a Mandriva machine to the F16machine, then

ls /net/F16machine/home

just times out, and I get a "No such file or directory" error message on command line. At that time I also get

Apr 11 23:41:58 e129939 automount[19018]: lookup_mount: exports lookup failed for F16machine
Apr 11 23:41:58 e129939 automount[19018]: update_negative_cache: key "F16machine" not found in map.

in /var/log/messages on the host where I ran the 'ls' command.

Any ideas where/what I should troubleshoot next?

For what it's worth, I started on Linux with Red Hat 4.0, moved to Mandrake 7.0 for its better (at the time) support for multiple desktops, and stuck with Mandriva for their good admin tools and multimedia (audio production) capability. I recently found out that an audio Fedora SIG has worked to get a lot of the CCRMA audio add on packages into the Fedora repositories. This is good timing as Mandriva's business shuffle is leaving me unimpressed. If I can get Fedora to work, then I'm likely to stick with it for a while. But, I'm having a little trouble understanding the F16 way of dealing with the firewall. SELinux is completely new to me, as is systemctl. I'll keep working at it, and right now "it" is "autofs".

Thanks much.....

[KevinDrums]

On a Mandriva machine, in /etc/autofs/autofs.conf I set LOGGING="debug" and restarted autofs. On the Fedora 16 machine, in /etc/sysconfig/autofs I set LOGGING="debug", also restarting autofs.

Next on the Mandriva machine I ran 'ls /net/f16machine/'. Nothing appeared in f16machine:/var/log/messages. The following appeared in mdvhost:/var/log/messages

    Apr 12 13:16:29 mdvhost automount[9306]: lookup_mount: exports lookup failed for f16machine
    Apr 12 13:16:29 mdvhost automount[9306]: update_negative_cache: key "f16machine" not found in map.
    Apr 12 13:19:16 mdvhost automount[24056]: Starting automounter version 5.0.5, master map auto.master
    Apr 12 13:19:16 mdvhost automount[24056]: using kernel protocol version 5.01
    Apr 12 13:19:16 mdvhost automount[24056]: lookup_nss_read_master: reading master files auto.master
    Apr 12 13:19:16 mdvhost automount[24056]: parse_init: parse(sun): init gathered global options: (null)
    Apr 12 13:19:16 mdvhost automount[24056]: lookup_read_master: lookup(file): read entry /net
    Apr 12 13:19:16 mdvhost automount[24056]: lookup_read_master: lookup(file): read entry +auto.master
    Apr 12 13:19:16 mdvhost automount[24056]: lookup_nss_read_master: reading master files auto.master
    Apr 12 13:19:16 mdvhost automount[24056]: parse_init: parse(sun): init gathered global options: (null)
    Apr 12 13:19:16 mdvhost automount[24056]: lookup(file): failed to read included master map auto.master
    Apr 12 13:19:16 mdvhost automount[24056]: master_do_mount: mounting /net
    Apr 12 13:19:16 mdvhost automount[24056]: automount_path_to_fifo: fifo name /var/run/autofs.fifo-net
    Apr 12 13:19:16 mdvhost automount[24056]: lookup_nss_read_map: reading map hosts (null)
    Apr 12 13:19:16 mdvhost automount[24056]: parse_init: parse(sun): init gathered global options: (null)
    Apr 12 13:19:16 mdvhost automount[24056]: mounted indirect on /net with timeout 300, freq 75 seconds
    Apr 12 13:19:16 mdvhost automount[24056]: st_ready: st_ready(): state = 0 path /net
    Apr 12 13:19:16 mdvhost automount[24056]: ghosting enabled
    Apr 12 13:19:35 mdvhost automount[24056]: handle_packet: type = 3
    Apr 12 13:19:35 mdvhost automount[24056]: handle_packet_missing_indirect: token 1330, name f16machine, request pid 24444
    Apr 12 13:19:35 mdvhost automount[24056]: attempting to mount entry /net/f16machine
    Apr 12 13:19:35 mdvhost automount[24056]: lookup(hosts): can't find path in hosts map /net/f16machine
    Apr 12 13:19:35 mdvhost automount[24056]: lookup_mount: lookup(hosts): lookup failed - update exports list
    Apr 12 13:19:35 mdvhost automount[24056]: lookup_mount: lookup(hosts): fetchng export list for f16machine
    Apr 12 13:19:38 mdvhost automount[24056]: lookup_mount: exports lookup failed for f16machine
    Apr 12 13:19:38 mdvhost automount[24056]: update_negative_cache: key "f16machine" not found in map.
    Apr 12 13:19:38 mdvhost automount[24056]: dev_ioctl_send_fail: token = 1330
    Apr 12 13:19:38 mdvhost automount[24056]: failed to mount /net/f16machine
    Apr 12 13:19:38 mdvhost automount[24056]: handle_packet: type = 3
    Apr 12 13:19:38 mdvhost automount[24056]: handle_packet_missing_indirect: token 1331, name f16machine, request pid 24444
    Apr 12 13:19:38 mdvhost automount[24056]: attempting to mount entry /net/f16machine
    Apr 12 13:19:38 mdvhost automount[24056]: dev_ioctl_send_fail: token = 1331
    Apr 12 13:19:38 mdvhost automount[24056]: failed to mount /net/f16machine

Then on the Fedora machine I ran 'ls /net/f16machine/' and the following appeared in f16machine:/var/log/messages

    Apr 12 13:24:17 f16machine autofs[16865]: Starting automount: [  OK  ]
    Apr 12 13:24:23 f16machine automount[16876]: handle_packet: type = 3
    Apr 12 13:24:23 f16machine automount[16876]: handle_packet_missing_indirect: token 1, name f16machine, request pid 16886
    Apr 12 13:24:23 f16machine automount[16876]: attempting to mount entry /net/f16machine
    Apr 12 13:24:23 f16machine automount[16876]: lookup_mount: lookup(hosts): fetchng export list for f16machine
    Apr 12 13:24:23 f16machine automount[16876]: rpc_get_exports_proto
    Apr 12 13:24:23 f16machine automount[16876]: lookup_mount: lookup(hosts): f16machine -> "/projects" "f16machine:/projects" "/nobackup" "f16machine:/nobackup" "/music" "f16machine:/music" "/home" "f16machine:/home"
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): expanded entry: "/projects" "f16machine:/projects" "/nobackup" "f16machine:/nobackup" "/music" "f16machine:/music" "/home" "f16machine:/home"
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): gathered options:
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): dequote(""/projects"") -> /projects
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): gathered options:
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): dequote(""f16machine:/projects"") -> f16machine:/projects
    Apr 12 13:24:23 f16machine automount[16876]: add_offset_entry: parse(sun): added multi-mount offset /projects -> f16machine:/projects
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): dequote(""/nobackup"") -> /nobackup
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): gathered options:
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): dequote(""f16machine:/nobackup"") -> f16machine:/nobackup
    Apr 12 13:24:23 f16machine automount[16876]: add_offset_entry: parse(sun): added multi-mount offset /nobackup -> f16machine:/nobackup
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): dequote(""/music"") -> /music
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): gathered options:
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): dequote(""f16machine:/music"") -> f16machine:/music
    Apr 12 13:24:23 f16machine automount[16876]: add_offset_entry: parse(sun): added multi-mount offset /music -> f16machine:/music
    Apr 12 13:24:23 f16machine automount[16876]: parse_mount: parse(sun): dequote(""/home"") -> /home
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): gathered options:
    Apr 12 13:24:23 f16machine automount[16876]: parse_mapent: parse(sun): dequote(""f16machine:/home"") -> f16machine:/home
    Apr 12 13:24:23 f16machine automount[16876]: add_offset_entry: parse(sun): added multi-mount offset /home -> f16machine:/home
    Apr 12 13:24:23 f16machine automount[16876]: mount_multi_triggers: mount offset /net/f16machine/home at /net/f16machine
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: calling mount -t autofs -s  -o fd=13,pgrp=16876,minproto=5,maxproto=5,offset automount /net/f16machine/home
    Apr 12 13:24:23 f16machine automount[16876]: mounted offset on /net/f16machine/home with timeout 300, freq 75 seconds
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: mounted trigger /net/f16machine/home at /net/f16machine/home
    Apr 12 13:24:23 f16machine automount[16876]: mount_multi_triggers: mount offset /net/f16machine/music at /net/f16machine
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: calling mount -t autofs -s  -o fd=13,pgrp=16876,minproto=5,maxproto=5,offset automount /net/f16machine/music
    Apr 12 13:24:23 f16machine automount[16876]: mounted offset on /net/f16machine/music with timeout 300, freq 75 seconds
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: mounted trigger /net/f16machine/music at /net/f16machine/music
    Apr 12 13:24:23 f16machine automount[16876]: mount_multi_triggers: mount offset /net/f16machine/nobackup at /net/f16machine
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: calling mount -t autofs -s  -o fd=13,pgrp=16876,minproto=5,maxproto=5,offset automount /net/f16machine/nobackup
    Apr 12 13:24:23 f16machine automount[16876]: mounted offset on /net/f16machine/nobackup with timeout 300, freq 75 seconds
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: mounted trigger /net/f16machine/nobackup at /net/f16machine/nobackup
    Apr 12 13:24:23 f16machine automount[16876]: mount_multi_triggers: mount offset /net/f16machine/projects at /net/f16machine
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: calling mount -t autofs -s  -o fd=13,pgrp=16876,minproto=5,maxproto=5,offset automount /net/f16machine/projects
    Apr 12 13:24:23 f16machine automount[16876]: mounted offset on /net/f16machine/projects with timeout 300, freq 75 seconds
    Apr 12 13:24:23 f16machine automount[16876]: mount_autofs_offset: mounted trigger /net/f16machine/projects at /net/f16machine/projects
    Apr 12 13:24:23 f16machine automount[16876]: dev_ioctl_send_ready: token = 1
    Apr 12 13:24:23 f16machine automount[16876]: mounted /net/f16machine

I didn't gain any new insight from that exercise. Maybe someone reading this post will see something that might help me?

Thanks....

[KevinDrums]

I got the user & group stuff working by specifying the Domain in /etc/idmapd.conf on the Mandriva machine. Leaving Domain commented out, and going with the default seems fine on the Fedora machine. The result of this is that a manual NFS mount allows me to see the correct user and group of remote files. But, I still can't automount (autofs) the Fedora 16 x86_64 machine from the Mandriva 2010.2 i686 machine. When I try to 'ls /net/f16machine/' I get the following in /var/log/messages:

Apr 12 18:40:21 mdvhost automount[17954]: \
  st_expire: state 1 path /net
Apr 12 18:40:21 mdvhost automount[17954]: \
  expire_proc: exp_proc = 3062889328 path /net
Apr 12 18:40:21 mdvhost automount[17954]: \
  expire_cleanup: got thid 3062889328 path /net stat 0
Apr 12 18:40:21 mdvhost automount[17954]: \
  expire_cleanup: sigchld: exp 3062889328 finished, switching from 2 to 1
Apr 12 18:40:21 mdvhost automount[17954]: \
  st_ready: st_ready(): state = 2 path /net
Apr 12 18:40:34 mdvhost automount[17954]: \
  handle_packet: type = 3
Apr 12 18:40:34 mdvhost automount[17954]: \
  handle_packet_missing_indirect: token 1374, name f16machine, \
  request pid 27782
Apr 12 18:40:34 mdvhost automount[17954]: \
  attempting to mount entry /net/f16machine
Apr 12 18:40:34 mdvhost automount[17954]: \
  lookup_mount: lookup(program): f16machine -> key is 'f16machine'
Apr 12 18:40:34 mdvhost automount[17954]: \
  lookup_mount: lookup(program): looking up f16machine
Apr 12 18:40:34 mdvhost automount[17954]: \
  >> clnt_create: RPC: Port mapper failure - Unable to receive: \
  errno 113 (No route to host)
Apr 12 18:40:34 mdvhost automount[17954]: \
  lookup(program): lookup for f16machine failed
Apr 12 18:40:34 mdvhost automount[17954]: \
  dev_ioctl_send_fail: token = 1374
Apr 12 18:40:34 mdvhost automount[17954]: \
  failed to mount /net/f16machine
Apr 12 18:40:34 mdvhost automount[17954]: \
  handle_packet: type = 3
Apr 12 18:40:34 mdvhost automount[17954]: \
  handle_packet_missing_indirect: token 1375, name f16machine, \
  request pid 27782
Apr 12 18:40:34 mdvhost automount[17954]: \
  attempting to mount entry /net/f16machine
Apr 12 18:40:34 mdvhost automount[17954]: \
  dev_ioctl_send_fail: token = 1375
Apr 12 18:40:34 mdvhost automount[17954]: \
  failed to mount /net/f16machine
Apr 12 18:41:36 mdvhost automount[17954]: \
  st_expire: state 1 path /net
Apr 12 18:41:36 mdvhost automount[17954]: \
  expire_proc: exp_proc = 3062889328 path /net
Apr 12 18:41:36 mdvhost automount[17954]: \
  expire_cleanup: got thid 3062889328 path /net stat 0
Apr 12 18:41:36 mdvhost automount[17954]: \
  expire_cleanup: sigchld: exp 3062889328 finished, switching from 2 to 1
Apr 12 18:41:36 mdvhost automount[17954]: \
  st_ready: st_ready(): state = 2 path /net

The "No route to host" message has got to be a good clue. But, why does the manual NFS mount work? Why does SSH and ping work? It's just autofs that can't find the route. Any idea folks?

Some other NFS threads had a little controversy as to whether NFS is broken on modern (3-6month old releases) 64-bit machines. I just installed my Fedora machine a wee ago. Maybe it's worth a try to install the i686 version and see if this clears up?

Thanks....

[KevinDrums]

The problem was on my firewall. The solution is to open ports 111 and 20048 for both udp and tcp. Troubleshooting information below.

Automounting to the Fedora 16 machine doesn't work yet.

mandriva1> showmount --exports f16machine
rpc mount export: RPC: Unable to receive; errno = No route to host

mandriva1> rpcinfo -p f16machine
rpcinfo: can't contact portmapper: RPC: Remote system error - No route to host

Automounting between Mandriva machines has been working for a while.

mandriva1> showmount --exports mandriva2
Export list for mandriva2:
/spare 192.168.1.0/24
/projects 192.168.1.0/24
/nobackup 192.168.1.0/24
/music 192.168.1.0/24
/media 192.168.1.0/24
/home 192.168.1.0/24

mandriva1> rpcinfo -p mandriva2
program vers proto port
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 4001 status
100024 1 tcp 4001 status
100021 1 udp 4002 nlockmgr
100021 3 udp 4002 nlockmgr
100021 4 udp 4002 nlockmgr
100021 1 tcp 4002 nlockmgr
100021 3 tcp 4002 nlockmgr
100021 4 tcp 4002 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 2 tcp 2049 nfs_acl
100227 3 tcp 2049 nfs_acl
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 2 udp 2049 nfs_acl
100227 3 udp 2049 nfs_acl
100005 1 udp 33675 mountd
100005 1 tcp 59528 mountd
100005 2 udp 33675 mountd
100005 2 tcp 59528 mountd
100005 3 udp 33675 mountd
100005 3 tcp 59528 mountd

I opened port 111 for tcp and udp on the f16machine firewall. Then I got:

mandriva1> rpcinfo -p f16machine
program vers proto port
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100011 1 udp 875 rquotad
100011 2 udp 875 rquotad
100011 1 tcp 875 rquotad
100011 2 tcp 875 rquotad
100024 1 udp 4001 status
100024 1 tcp 4001 status
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 2 tcp 2049 nfs_acl
100227 3 tcp 2049 nfs_acl
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 2 udp 2049 nfs_acl
100227 3 udp 2049 nfs_acl
100021 1 udp 4002 nlockmgr
100021 3 udp 4002 nlockmgr
100021 4 udp 4002 nlockmgr
100021 1 tcp 4002 nlockmgr
100021 3 tcp 4002 nlockmgr
100021 4 tcp 4002 nlockmgr
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd

But, still I had no luck.

mandriva1> showmount --exports f16machine
rpc mount export: RPC: Unable to receive; errno = No route to host

I tried opening all of the ports listed in rpcinfo on the firewall. That worked. Then I backed off to opening only ports 111 and 20048, both for tcp and udp. Now I can see the mount points.

mandriva1> showmount --exports f16machine
Export list for f16machine:
/projects 192.168.1.0/24
/nobackup 192.168.1.0/24
/music 192.168.1.0/24
/home 192.168.1.0/24

What's better is that I can mount the mount points!

Ahhh, nice. I hope this helps someone else!

Cheers.