how to fix device mapping for guest virtual machine (qemu/KVM)

globe5 · #1 8th December 2012, 03:35 PM

Hi all,

I am running Fedora16 as a home server and I use it as s virtualization host. On that machine I have several virtual guests, running Fedora, Oracle Linux and Windows.

host machine: c4.c4.lan
oel6 machine: node01.c4.lan

One of the virtual guests is Oracle Linux 6.0 and I use it mainly for Oracle database server.
It has two LVM VGs - one for / fs and for swap(vg_node01), the other VG is for oracle's home and data filesystems(vg_ora).
The disks containing these VGs are named /dev/sda and /dev/sdb (or at least used to be).

I decided to install Oracle ASM instance on it and presented it 8 logical volumes from the host machine to emulate scsi disks. The procedure I've done this is:

Code:

[root@c4 ~]# lvcreate -L 2G -n virtdisks_node01_asm1 vg_c4
  Logical volume "virtdisks_node01_asm1" created
virsh # attach-disk node01 /dev/mapper/vg_c4-virtdisks_node01_asm1 sdc --persistent
Disk attached successfully

The above I've repeated for the rest of LVs (8 total) and I am able to query them using virsh:

Code:

virsh # domblkinfo node01 sdc
Capacity:       2147483648
Allocation:     2147483648
Physical:       2147483648

The thing is that when the virtual machine got rebooted, the block devices are assigned with different device names (/dev/sd*). However, if I query the qemu's domain (node01) xml config file, I can see that the naming conventions are correct (reflecting how do I want them):

Code:

[root@c4 11.2.0.3]# virsh dumpxml node01
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg_c4/virtmachines'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg_c4/virtmachines_1'/>
      <target dev='hdb' bus='ide'/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm1'/>
      <target dev='sdc' bus='scsi'/>
      <alias name='scsi0-0-2'/>
      <address type='drive' controller='0' bus='0' unit='2'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm2'/>
      <target dev='sdd' bus='scsi'/>
      <alias name='scsi0-0-3'/>
      <address type='drive' controller='0' bus='0' unit='3'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm3'/>
      <target dev='sde' bus='scsi'/>
      <alias name='scsi0-0-4'/>
      <address type='drive' controller='0' bus='0' unit='4'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm4'/>
      <target dev='sdf' bus='scsi'/>
      <alias name='scsi0-0-5'/>
      <address type='drive' controller='0' bus='0' unit='5'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm5'/>
      <target dev='sdg' bus='scsi'/>
      <alias name='scsi0-0-6'/>
      <address type='drive' controller='0' bus='0' unit='6'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm6'/>
      <target dev='sdh' bus='scsi'/>
      <alias name='scsi1-0-0'/>
      <address type='drive' controller='1' bus='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm7'/>
      <target dev='sdi' bus='scsi'/>
      <alias name='scsi1-0-1'/>
      <address type='drive' controller='1' bus='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm8'/>
      <target dev='sdj' bus='scsi'/>
      <alias name='scsi1-0-2'/>
      <address type='drive' controller='1' bus='0' unit='2'/>
    </disk>

However, this is how I see the block devices from virtual guest's perspective:

Code:

[root@node01 ~]# fdisk -l |grep /dev/sd
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/sda: 2147 MB, 2147483648 bytes
/dev/sda1               1        1009     2095662   83  Linux
Disk /dev/sdb: 2147 MB, 2147483648 bytes
/dev/sdb1               1        1009     2095662   83  Linux
Disk /dev/sdc: 2147 MB, 2147483648 bytes
/dev/sdc1               1        1009     2095662   83  Linux
Disk /dev/sdd: 2147 MB, 2147483648 bytes
/dev/sdd1               1        1009     2095662   83  Linux
Disk /dev/sde: 1073 MB, 1073741824 bytes
/dev/sde1               1        1011     1048376+  83  Linux
Disk /dev/sdf: 1073 MB, 1073741824 bytes
/dev/sdf1               1        1011     1048376+  83  Linux
Disk /dev/sdh: 1073 MB, 1073741824 bytes
/dev/sdh1               1        1011     1048376+  83  Linux
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/sdg: 1073 MB, 1073741824 bytes
/dev/sdg1               1        1011     1048376+  83  Linux
Disk /dev/sdj: 21.5 GB, 21474836480 bytes
/dev/sdj1               2       20480    20970496   8e  Linux LVM
Disk /dev/sdi: 21.5 GB, 21474836480 bytes
/dev/sdi1   *           1          64      512000   83  Linux
/dev/sdi2              64        2611    20458496   8e  Linux LVM
Disk /dev/dm-2 doesn't contain a valid partition table
Disk /dev/dm-3 doesn't contain a valid partition table
Disk /dev/dm-4 doesn't contain a valid partition table
Disk /dev/dm-5 doesn't contain a valid partition table
[root@node01 ~]#

I need to make those LVs presented from the host machine as scsi disks to the guests machine with target device names as I've written them into the guest XML file.

I don't know how to achieve that, please help, anyone?

Many thanks in advance!

stevea · #2 8th December 2012, 03:56 PM

I' m confused. It looks like they are showing up properly named - no ?

Quote:

Quote:

Disk /dev/sdg: 1073 MB, 1073741824 bytes
/dev/sdg1 1 1011 1048376+ 83 Linux

globe5 · #3 8th December 2012, 04:56 PM

Quote:

Originally Posted by stevea

I' m confused. It looks like they are showing up properly named - no ?

Hi stevea, thanks for the follow-up.

Some of the devices got their names properly, some are not. For example these:

Code:

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg_c4/virtmachines'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>

The LV '/dev/vg_c4/virtmachines' should have device name /dev/hda, but actually is named as:

Code:

Disk /dev/sdj: 21.5 GB, 21474836480 bytes
/dev/sdj1               2       20480    20970496   8e  Linux LVM

and respectively:

Code:

 <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vg_c4/virtmachines_1'/>
      <target dev='hdb' bus='ide'/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' unit='1'/>
    </disk>

is shown as:

Code:

Disk /dev/sdi: 21.5 GB, 21474836480 bytes
/dev/sdi1   *           1          64      512000   83  Linux
/dev/sdi2              64        2611    20458496   8e  Linux LVM

As you can see I've named the ASM LVs as target device names sdc to sdj (8 total - first four should be 2GB disks(sdc,sdd,sde,sdf) and the rest four 1GB (sdg,sdh,sdi,sdj) and thus I expect my virtual machine to see them that way:

/dev/hda (first 21.5GB disk)
/dev/hdb (second 21.5GB disk)
/dev/sdc - /dev/sdf - four 2GB disks
/dev/sdg - /dev/sdj - four 1GB disks

but this is not the case.

Any ideas?

Thanks!

---------- Post added at 06:56 PM ---------- Previous post was at 06:12 PM ----------

Update

I've been advised not to use scsi emulation with KVM, but instead to use virtio_blk emulation.
So I did and "converted" the disks to virtio as editing the guest machine XML config file for all ASM disks like this:

old:

Code:

<disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm1'/>
      <target dev='sdc' bus='scsi'/>
      <address type='drive' controller='0' bus='0' unit='2'/>
    </disk>

new:

Code:

<disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/mapper/vg_c4-virtdisks_node01_asm1'/>
      <target dev='vdc' bus='virtio'/>
    </disk>

After guest machine reboot this is how the devices got renamed:

Code:

[root@node01 ~]# fdisk -l |grep /dev/
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/sda: 21.5 GB, 21474836480 bytes
/dev/sda1   *           1          64      512000   83  Linux
/dev/sda2              64        2611    20458496   8e  Linux LVM
Disk /dev/sdb: 21.5 GB, 21474836480 bytes
/dev/sdb1               2       20480    20970496   8e  Linux LVM
Disk /dev/vda: 2147 MB, 2147483648 bytes
/dev/vda1               1        1009     2095662   83  Linux
Disk /dev/vdb: 2147 MB, 2147483648 bytes
/dev/vdb1               1        1009     2095662   83  Linux
Disk /dev/vdc: 2147 MB, 2147483648 bytes
/dev/vdc1               1        1009     2095662   83  Linux
Disk /dev/vdd: 2147 MB, 2147483648 bytes
/dev/vdd1               1        1009     2095662   83  Linux
Disk /dev/vde: 1073 MB, 1073741824 bytes
Disk /dev/dm-1 doesn't contain a valid partition table
/dev/vde1               1        1011     1048376+  83  Linux
Disk /dev/vdf: 1073 MB, 1073741824 bytes
/dev/vdf1               1        1011     1048376+  83  Linux
Disk /dev/vdg: 1073 MB, 1073741824 bytes
/dev/vdg1               1        1011     1048376+  83  Linux
Disk /dev/vdh: 1073 MB, 1073741824 bytes
/dev/vdh1               1        1011     1048376+  83  Linux
Disk /dev/dm-0: 4227 MB, 4227858432 bytes
Disk /dev/dm-2 doesn't contain a valid partition table
Disk /dev/dm-3 doesn't contain a valid partition table
Disk /dev/dm-4 doesn't contain a valid partition table
Disk /dev/dm-5 doesn't contain a valid partition table
Disk /dev/dm-1: 16.7 GB, 16710107136 bytes
Disk /dev/dm-2: 5368 MB, 5368709120 bytes
Disk /dev/dm-3: 5368 MB, 5368709120 bytes
Disk /dev/dm-4: 5368 MB, 5368709120 bytes
Disk /dev/dm-5: 5268 MB, 5268045824 bytes
[root@node01 ~]#

Also, you might notice that disks /dev/sda and /dev/sdb (previously /dev/sdj and /dev/sdi) have shown up - without messing up with their <disk> XML directive. Obviously the naming is controlled somewhere else, but not on the XML file or at least I don't know how and where.

Two questions arose observing the above output:

1. Why is the XML directive 'target dev=name' not working?
2. If one decides to add/reorganize/drop a disk, does this mean that he can expect again change in the naming conventions?

Can someone spread some light please?

Thanks in advance!

stevea · #4 8th December 2012, 05:43 PM

Well - I can understand what is happening, but I have no idea how to force the device naming from KVM.

Ultimately KVM just provides virtual hardware for the guest. It cannot force the guest to name the hardware devices in any way that I can imagine. Normally the kernel (guest for example) will examine th disk controllers in PCI bus order and then examine each drive on each contreoller and name (scsi, sata, ide) sda, sdb, sdc, in bus:drive order. The virtio will name the drives vda, vdb, .. .in order.

So I don't believe that kvm has the ability to force the guest to use a specific device name.

It might be interesting to change the 'target dev=' field and see if the guest observes the name or else the 'controller= bus= unit=' order. Id wager the guest ignores target dev=

--

Obviously you can manage this in the target by using mknod in a startup script or probably by configuring udev.

sorry I don't have more,
-S