Missing LVM entries in /dev/disk/by-uuid

[jeff1969]

I don't really know what I'm doing here, but I'll describe the situation:

• Intel RAID card supporting 2 RAIDS
  • A 2 disk RAID 0 for the system
  • A 5 disk 10 TB RAID 5 for storage
• LVM laid on top of the RAID 0 to partition it
• All has been well for months
• updated (yum update), didn't track what got upgraded
• On reboot, partitions aren't found. I end up in a rescue session
• From there, i can vgchange -ay, then mount -a, then startx and all is well

Now to the topic of the post. In rebuggin, it seems none of the missing partitions are represented in /dev/disk/by-uuid. Does that matter? From what I've read, it seems LVm is supposed to manage that, not udev?

any help appreciated,
~Jeff

[mkudro]

Strange situation..but for starters:
- check /var/log/yum.log for things that got updated/installed.
- Check /etc/fstab if everything is ok in there, compare uuid in fstab to ones you get from "lvdisplay |egrep 'Path|UUID'"
- anything encrypted? if so check /etc/crypttab too.
- just for kicks check if lvm is enabled "systemctl list-unit-files |grep lvm"

Post back if you find anything strange ... or not.

[jeff1969]

Originally Posted by mkudro

Strange situation..but for starters:
- check /var/log/yum.log for things that got updated/installed.

See attached list.

Originally Posted by mkudro

- Check /etc/fstab if everything is ok in there, compare uuid in fstab to ones you get from "lvdisplay |egrep 'Path|UUID'"

Ahhh....maybe here's the problem...here's (a slightly modified) lvdisplay result (where HOSTNAME is usually my real hostname):

  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME00/LogVol00_bigraid
  LV UUID                mYzfDf-h7gw-UWp8-ojYw-oDaY-BmTI-bnq03I
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol02_tmp
  LV UUID                ECqDnu-U3yd-meob-lHl9-WhMP-1Lxc-TLYncA
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol03_usrlocal
  LV UUID                aY1VXm-Ryhc-EA6M-h4lI-uCKk-X0ao-NX0tRe
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol00_var
  LV UUID                HbGJEh-KhJm-JL1e-HDaC-sLxd-qGZB-VdzUGM
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol01_opt
  LV UUID                wXqj42-9qaQ-QHY7-HNAj-OWmb-5z1q-vCQP50
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol04_home
  LV UUID                IwS9nW-Q4Uv-UDoe-hwyB-6Kyg-RZ74-kHmiEc
  --- Logical volume ---
  LV Name                /dev/vg_HOSTNAME/LogVol05_slash
  LV UUID                gpR427-HTeO-zmMe-1yJC-tvQP-mOLd-q4dcsN

But here's /etc/fstab:

/dev/mapper/vg_HOSTNAME-LogVol05_slash /                       ext4    defaults        1 1
/dev/mapper/vg_HOSTNAME-LogVol04_home /home                   ext4    defaults        1 2
/dev/mapper/vg_HOSTNAME-LogVol01_opt /opt                    ext4    defaults        1 2
/dev/mapper/vg_HOSTNAME-LogVol02_tmp /tmp                    ext4    defaults        1 2
/dev/mapper/vg_HOSTNAME-LogVol03_usrlocal /usr/local              ext4    defaults        1 2
/dev/mapper/vg_HOSTNAME-LogVol00_var /var                    ext4    defaults        1 2
/dev/mapper/vg_HOSTNAME00-LogVol00_bigraid    /bigraid      xfs   defaults 1 2 
UUID=ee284287-bc5e-4566-8689-2d444bc5d34a /boot                   ext4    defaults        1 2

/boot *does* get mounted and shows up in /dev/disk/by-uuid. This used to work, but maybe broke in the kernel upgrade? I'll try modifying fstab to use the UUIDs and try a reboot tonight (time to get to work now)

Originally Posted by mkudro

- anything encrypted? if so check /etc/crypttab too.

Nope

Originally Posted by mkudro

- just for kicks check if lvm is enabled "systemctl list-unit-files |grep lvm"

lvm2-monitor.service                        enabled

Post back if you find anything strange ... or not.[/QUOTE]

Thanks for the input!

[jeff1969]

Nope....Failed to boot because those UUID devices don't exist.

My understanding is very limited. But I see:

ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Dec 13  2012 08fe1d0e-c4d4-4165-bf88-707aee183270 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Dec 13  2012 38a03b62-876e-4437-95d9-32fdc0eccd05 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Dec 13  2012 4E20B7C020B7AD79 -> ../../sda3
lrwxrwxrwx 1 root root 10 Dec 13  2012 68B30E4C50BCF13B -> ../../sdc2
lrwxrwxrwx 1 root root 10 Dec 13  2012 ee284287-bc5e-4566-8689-2d444bc5d34a -> ../../sda2

I read these links should be managed by LVM for my LVM devices, not udev, so it seems the issue is that LVM isn't properly managing the links at startup?

The missing partitions are dm-1, dm-2,etc...and I do see those mounted:

Dec 13 07:28:42 flux kernel: [    0.665799] device-mapper: ioctl: 4.23.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com
Dec 13 07:28:42 flux kernel: [    2.644876] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [    2.827716] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [    4.735511] EXT4-fs (dm-0): re-mounted. Opts: (null)
Dec 13 07:28:42 flux kernel: [  124.491836] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [  124.559635] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [  124.624426] EXT4-fs (dm-2): warning: maximal mount count reached, running e2fsck is recommended
Dec 13 07:28:42 flux kernel: [  124.624622] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [  124.681713] EXT4-fs (dm-3): warning: maximal mount count reached, running e2fsck is recommended
Dec 13 07:28:42 flux kernel: [  124.682597] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [  124.735994] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null)
Dec 13 07:28:42 flux kernel: [  125.107399] XFS (dm-1): Mounting Filesystem
Dec 13 07:28:42 flux kernel: [  125.997977] XFS (dm-1): Ending clean mount
Dec 13 07:28:42 flux kernel: [  129.384341] EXT4-fs (dm-0): re-mounted. Opts: (null)

We're into things I'm not familiar with here...device mapper, lvm low-level stuff....thanks for the help
~Jeff

[mkudro]

hm...i smell something going wrong in initramfs area....
could you please try rebuilding initramfs:

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

then rebuild grub:

grub2-mkconfig -o /boot/grub2/grub.cfg

and reboot.

[jeff1969]

Thanks. As I recall, when i hit this on monday night I did the first step, but I believe I may have omitted rebuilding grub. I'll try i again when I'm home tonight, and go through the whole process this time. I'll report back when done.

Cheers,
~jeff

---------- Post added at 07:00 PM ---------- Previous post was at 08:28 AM ----------

Okay...when i tried this, i get:

[root@flux ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.6.7-4.fc16.x86_64
Found initrd image: /boot/initramfs-3.6.7-4.fc16.x86_64.img
Found linux image: /boot/vmlinuz-3.6.6-1.fc16.x86_64
Found initrd image: /boot/initramfs-3.6.6-1.fc16.x86_64.img
Found linux image: /boot/vmlinuz-3.6.2-1.fc16.x86_64
Found initrd image: /boot/initramfs-3.6.2-1.fc16.x86_64.img

Found Windows 7 (loader) on /dev/sda3
done

Notice it doesn't find my Fedora installation. No change in behavior when i try to reboot.

Looking for relevant info, this looks strange to me:

ls -l /dev/mapper
total 0
crw------- 1 root root  10, 236 Dec 13 20:04 control
brw-rw---- 1 root disk 253,   1 Dec 13 20:04 vg_flux00-LogVol00_bigraid
brw-rw---- 1 root disk 253,   4 Dec 13 20:04 vg_flux-LogVol00_var
brw-rw---- 1 root disk 253,   5 Dec 13 20:04 vg_flux-LogVol01_opt
brw-rw---- 1 root disk 253,   2 Dec 13 20:04 vg_flux-LogVol02_tmp
brw-rw---- 1 root disk 253,   3 Dec 13 20:04 vg_flux-LogVol03_usrlocal
brw-rw---- 1 root disk 253,   6 Dec 13 20:04 vg_flux-LogVol04_home
lrwxrwxrwx 1 root root        7 Dec 13 20:04 vg_flux-LogVol05_slash -> ../dm-0

Now, slash is the one that mounts (along with /boot, which lives on it's own disk, and isn't part of this). In the mapper, it's a link to the devicemapper entry, and the others arent. Seems different at least.

[mkudro]

This is your Fedora installation...no worries here...

Found linux image: /boot/vmlinuz-3.6.7-4.fc16.x86_64
Found initrd image: /boot/initramfs-3.6.7-4.fc16.x86_64.img

Bu this look indeed strange... your /dev/mapper shows your logical volums as special devices

ls -l /dev/mapper
total 0
crw------- 1 root root  10, 236 Dec 13 20:04 control
brw-rw---- 1 root disk 253,   1 Dec 13 20:04 vg_flux00-LogVol00_bigraid
brw-rw---- 1 root disk 253,   4 Dec 13 20:04 vg_flux-LogVol00_var
brw-rw---- 1 root disk 253,   5 Dec 13 20:04 vg_flux-LogVol01_opt
brw-rw---- 1 root disk 253,   2 Dec 13 20:04 vg_flux-LogVol02_tmp
brw-rw---- 1 root disk 253,   3 Dec 13 20:04 vg_flux-LogVol03_usrlocal
brw-rw---- 1 root disk 253,   6 Dec 13 20:04 vg_flux-LogVol04_home
lrwxrwxrwx 1 root root        7 Dec 13 20:04 vg_flux-LogVol05_slash -> ../dm-0

I think it's time to investigate this ... could you please run this script and attach generated file (/root/fedora_gone_crazy) to thread:

#!/usr/bin/env bash
logFile="/root/fedora_gone_crazy"
for  i in "df -h" "fdisk -l" "pvscan" "vgscan" "lvscan" "vgdisplay" "lvdisplay" "cat /etc/fstab" "ls -al /dev/mapper" "ls -l /dev/dm-*" "cat /proc/partitions" "mount"; do
     printf "running $i \n\n\n"
     echo "# Output of $i" >> $logFile
     $i >> $logFile
     echo -e "\n\n\n\n\n\n"  >> $logFile
done

[jeff1969]

Below is output...file is attached.

Thank you!!

BTW, I've downloaded the F17 image, and am ready to upgrade this weekend if appropriate

sudo /tmp/fedora_gone_crazy.sh
running df -h 


running fdisk -l 



WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/mapper/vg_flux-LogVol05_slash doesn't contain a valid partition table
Disk /dev/mapper/vg_flux00-LogVol00_bigraid doesn't contain a valid partition table
Disk /dev/mapper/vg_flux-LogVol02_tmp doesn't contain a valid partition table
Disk /dev/mapper/vg_flux-LogVol03_usrlocal doesn't contain a valid partition table
Disk /dev/mapper/vg_flux-LogVol00_var doesn't contain a valid partition table
Disk /dev/mapper/vg_flux-LogVol01_opt doesn't contain a valid partition table
Disk /dev/mapper/vg_flux-LogVol04_home doesn't contain a valid partition table
running pvscan 


running vgscan 


running lvscan 


running vgdisplay 


running lvdisplay 


running cat /etc/fstab 


running ls -al /dev/mapper 


running ls -l /dev/dm-* 


running cat /proc/partitions 


running mount

[mkudro]

I don't get it...everything looks ok...well except /dev/mapper...i mean, normally i would expect to see symlinks for lvm devices or major/minor values...bot not both at the same time. Well at least i never encountered this before.
For the rest it looks alright...i really would say rebuilding initramfs would solve this...but we know that it doesn't .

Could you please try removing latest kernel (kernel-3.6.7-4.fc16.x86_64) and trying again. I really want to know where is it going wrong.
I understand that you would rather reinstall F17 and forget about this problem, but it's bugging me know too...so...common, couple of days of testing wont kill us

[jeff1969]

I'm not in a huge rush to upgrade, I just wanted to say I'm ready to if we can't get anywhere with this. As long as I can get the system up and running manually, i'm happy to beat my head on this to learn something.

So you'd like me to just try a 'yum downgrade kernel'? I'm happy to try anything, i just want to get the syntax right before I totally muck something up

Thanks,
~Jeff

[mkudro]

Hi Jeff, lets try this:
before we remove anything let's boot system into older kernel...i assume you have standard installation (meaning you haven't modified /etc/yum.conf) so you should have 3 kernels on the system at any given time.
So reboot, in grub menu with arrow-down key choose previous kernel an boot into it.
Let's see how is that going to play out.

[jeff1969]

I tried that Monday when things started going wrong (first thing I tried). Same behavior.

I'll confirm tonight though.

---------- Post added at 08:02 PM ---------- Previous post was at 12:32 PM ----------

Yes...as expected...none of the previous kernels help

---------- Post added at 10:35 PM ---------- Previous post was at 08:02 PM ----------

Here's something else odd. I had no sound (hadn't realized it all week). After a little digging, I found (with the help of getfacl) that /dev/snd/pcmC0D0c (my soundcard) wasn't read/write for me. A cuick chmod, and presto, i have sound. But again, this worked before the upgrade.

[jeff1969]

Okay...I'm upgrading to F17 tonight. I've jst found that also my CDROm /dev entries are horked up (0600), and I'm done messing with this.

Thanks for the help, mkudro, but I'm moving on

---------- Post added at 11:00 PM ---------- Previous post was at 09:23 PM ----------

And following the upgrade, I'm back in business