pxeboot iscsi root.

[stoggy]

Finished installing Fedora 15 x86_64 to a iscsi root. Using pxeboot to install from and to boot the OS for now on.

Worked really easily using the advanced disk option and installing to an iscsi target. Couple gotchas caught me up.

1. The line in /boot/grub/menu.lst uses p7p1 for eth0 so it wouldn't find my nic and boot root. This caused a kernel panic on boot trying for no or empty root= line

2. Messagebus wouldn't start when booting the new OS. So it would hang trying to go to X. In /etc/fstab it had all the local filesystems, (the ones on the iscsi target) marked as _netdev. Once i took these out it boots like a charm.


This is my pxeboot file.

PROMPT 0
MENU TITLE ISCSI BOOT
TIMEOUT 20

Default menu.c32
LABEL Fedora15
        MENU LABEL Fedora15
        kernel pxelinux.cfg/MSsux-boot/vmlinuz-2.6.38.6-27.fc15.x86_64 ro root=/dev/mapper/vg_MSsux-lv_root netroot=iscsi:@192.168.0.2::3260::iqn.2008-09.com.microsoft:MSsux.rootlun iscsi_initiator=iqn.2008-09.com.microsoft:MSsux.rootlun ip=eth0:dhcp ifname=eth0:00:ab:cd:ef:gh:ij rd_LVM_LV=vg_MSsux/lv_root rd_LVM_LV=vg_MSsux/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us noapic acpi=off rhgb quiet
        initrd pxelinux.cfg/MSsux-boot/initramfs-2.6.38.6-27.fc15.x86_64.img

scp'd, once i was at the last screen of the install, the /boot/vmlinuz... /boot/initramfs... and /boot/grub/menu.lst to my iscsi machine and using those to pxeboot with.

When i was still there i should have vi'd the /srv/sysimage/etc/fstab and removed the _netdev mount options from the local filesystems. (keeps messagebus from working subsequently if not)

then put the grub kernel and append line in my pxeboot file. have to change the odd nic names though.

---------- Post added at 03:09 PM ---------- Previous post was at 02:07 PM ----------

I wanted to try Fedora 15 but didnt want to reload my current machine. Gnome 3 is all that gui stuff.

I used this page for the pxeboot setup.
http://linux-sxs.org/internet_serving/pxeboot.html

I used this page for the iscsi setup.
http://www.cyberciti.biz/tips/howto-...nwith-tgt.html


So what i decided to do was create an iscsi target and boot it. The motherboard I wanted to use doesn't support booting from iscsi targets. One of my motherboards does but I can't use it for this. So i decided I would use pxeboot to start the boot process.

Setup tftpboot server for install and later to boot the machine.
a. Install the tftp-server dhcp httpd and syslinux

sudo yum install tftp-server dhcp httpd syslinux

b. setup dhcpd put this in /etc/dhcp/dhcpd.conf

sudo cp /etc/dhcp/dhcpd.conf{,.ORIG}
cat << EOF | sudo tee /etc/dhcp/dhcpd.conf
ddns-update-style interim;
subnet 192.168.0.0 netmask 255.255.255.0 {
	range 192.168.0.10 192.168.0.254;
	default-lease-time 3600;
	max-lease-time 4800;
	option routers 192.168.0.1;
	option domain-name-servers 192.168.0.1;
	option subnet-mask 255.255.255.0;
	option domain-name "llama.net";
	option time-offset -8;
}
EOF

c. setup xinetd. add this line to /etc/hosts.allow

echo "in.tftpd:       192.168.0." | sudo tee -a /etc/hosts.allow

d. change the disable line to "no" in /etc/xinetd.d/tftp. I also changed the server_args line to -s /tftpboot because that is where i like it...

cat << EOF | sudo tee /etc/xinetd.d/tftp
service tftp
{
        socket_type	= dgram
        protocol	= udp
        wait	= yes
        user	= root
        server	= /usr/sbin/in.tftpd
        server_args	= -s /tftpboot/
        disable	= no
        per_source	= 11
        cps		= 100 2
        flags	= IPv4
}
EOF

e. setup http to serve the installation files.

cat << EOF | sudo tee /etc/httpd/pxeboot.conf
#
<Directory "/tftpboot">
	order deny,allow
	deny from all
	allow from 127.0.0.1
	allow from 192.168.0.0/24
</Directory>
Alias           /tftpboot       /var/www/tftpboot/
EOF

f. create tftpboot directories

sudo mkdir -p /tftpboot/pxelinux.cfg/Fedora15-x86_64
sudo cp /usr/share/syslinux/pxelinux.0 /tftpboot/
sudo cp /usr/share/syslinux/menu.c32 /tftpboot/

i. create the httpd directories and mount the Fedora15 Installation DVD iso image

sudo mkdir /var/www/tftpboot
sudo cp ~/Downloads/Fedora-15-x86_64-DVD.iso /var/www/tftpboot
sudo mkdir /var/www/tftpboot/Fedora15
sudo mount -o loop /var/www/tftpboot/Fedora-15-x86_64-DVD.iso /var/www/tftpboot/Fedora15

j. create a default pxeboot image that will boot the installation.

sudo cp /tftpboot/pxelinux.cfg/default{,-ORIG}
cat << EOF | sudo tee /tftpboot/pxelinux.cfg/default
default Fedora15
timeout 10
label Fedora15
	kernel pxelinux.cfg/Fedora15-x86_64/vmlinuz-F15-x86_64
	append initrd=pxelinux.cfg/Fedora15-x86_64/initrd-F15-x86_64.img ramdisk_size=9216 noapic acpi=off
EOF

k. create httpd for Fedora15 Installation DVD iso image

sudo mkdir /var/www/tftpboot
sudo cp ~/Downloads/Fedora-15-x86_64-DVD.iso /var/www/tftpboot
sudo mkdir /var/www/tftpboot/Fedora15
sudo mount -o loop /var/www/tftpboot/Fedora-15-x86_64-DVD.iso /var/www/tftpboot/Fedora15

l. Copy the installation kernel and initrd into place.

sudo cp /var/www/tftpboot/Fedora15/isolinux/vmlinux /tftpboot/pxelinux.cfg/Fedora15-x86_64/vmlinuz-F15-x86_64
sudo cp /var/www/tftpboot/Fedora15/isolinux/initrd.img /tftpboot/pxelinux.cfg/Fedora15-x86_64/initrd-F15-x86_64.img

m. allow connection through iptables.

# allow dhcp requests
sudo /sbin/iptables -I INPUT -p udp --sport 68 --dport 67 -j ACCEPT
# allow tftp requests
sudo /sbin/iptables -I INPUT -p udp -s 192.168.0.0/24 --dport 69 -j ACCEPT
# allow http requests
sudo /sbin/iptables -I INPUT -p tcp -s 192.168.0.0/24 --dport 80 -j ACCEPT

n. start dhcp

sudo /etc/init.d/dhcpd start

o. restart xinetd - for tftp

sudo /etc/init.d/xinetd restart

p. start httpd

sudo /etc/init.d/httpd start

q. pxeboot the client and Install. When you get to the Disk selection at the bottom of the screen click advanced disk devices. It will pop up a menu and you can add an iscsi disk. This part was kind of weird and I think i had to create a disk under the "other" tab. Poke around and you can find it.

Once the installation completes and you are at the last screen with the reboot button in the bottom right. hit ALT+F2 to switch to a console tty. You need to grab your vmlinuz, initrd and grub file. If you do this now it will make life much easier. I just scp'd them to my iscsi server.

{i forget the name of the /mnt/sysimage dir now... so these may be wrong.} You dont need the menu.lst file but the lines you will need are in it. The line is long and the syntax is odd, so I would suggest copy+paste'ing it.

scp  /mnt/sysimage/boot/vmlinuz*  192.168.0.2:
scp  /mnt/sysimage/boot/initramfs*  192.168.0.2:
scp /mnt/sysimage/boot/grub/menu.lst  192.168.0.2:

*** One gotcha that caught me up, is the install will mark the filesystem on the iscsi target with mount options _netdev in the /etc/fstab. This caused messagebus not to start and I got the chance to learn about systemd. I would suggest editing the fstab file now and learning about systemd later, once booted.

This sed command will remove all the ",_netdev" options from the fstab file. You may have some FS's that require these though, up to you. For the install described here you wont. This sed command will save a copy of the fstab file at /mnt/sysimage/etc/fstab.ORIG if you make a mistake.

sed -i.ORIG 's/,_netdev//' /mnt/sysimage/etc/fstab

r. create the new pxeboot file for the system to boot from for now on. You will need the mac address of the client machine or you could create it as the default boot or make a menu or ... I would suggest making a mac specific boot target. Just makes life easier and if their are problems it wont affect your other machines.

Set these variables for your environment and then run the below sed command on your pxeboot server. This will create the pxeboot file to boot from. I got the kernel line below from the grub file I copied during the install. My grub file showed the nic as p7p1 though so i had to edit that to eth0 before this would work. Not sure why it passes swap and var through the initrd though, I haven't tried cleaning up the kernel line yet.

MACADDRESS="Mac address of the boot nic in the machine that will use the iscsi target for root"
ISCSI_TARGET="This is the iscsi target name.  Check the iscsi howto above."
ISCSI_SERVERIP="This is the IP of your iscsi server."

sed -e "s/{MACADDRESS}/$MACADDRESS/g; s/{ISCSI-TARGET}/$ISCSI_TARGET/g; s/{ISCSI-SERVERIP}/$ISCSISERVERIP/g" << EOF | sudo tee /tftpboot/pxelinux.cfg/00-`echo $MACADDRESS | sed -e 's/:/-/g'`
PROMPT 0
MENU TITLE ISCSI BOOT
TIMEOUT 20

Default menu.c32
LABEL Fedora15
        MENU LABEL Fedora15
        kernel pxelinux.cfg/vmlinuz-2.6.38.6-27.fc15.x86_64 ro root=/dev/mapper/vg0-lv_root netroot=iscsi:@{ISCSI-SERVER-IP}::3260::{ISCSI-TARGET} iscsi_initiator={ISCSI-TARGET} ip=eth0:dhcp ifname=eth0:{MACADDRESS} rd_LVM_LV=vg0/lv_root rd_LVM_LV=vg0/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us noapic acpi=off rhgb quiet
        initrd pxelinux.cfg/initramfs-2.6.38.6-27.fc15.x86_64.img
EOF

s. reboot your client system and done.

[lweidig]

This is a great analysis of a workaround for what we see as a serious bug that made it into the release.

This is really a show stopper for us using iscsi root drives in virtualized guests in F15. Hoping a respin will fix this or maybe a 15.1 release before 16. We are sticking with F14 right now and looking into alternatives.

[stoggy]

what are you using for you hypervisor. KVM and XEN can boot iscsi targets.

or you mean the _netdev? yea that one was hard to find. My iscsi is in ext4 in lvm from a centos 5.6 machine so i cant see/get inside the iscsi disk from the iscsi target machine. left me baffled for a bit.


If you are launching a bunch of virt hosts you should create your disks in the ks.cfg file. Then you can specify the mount options and leave out _netdev. If nothing else go to the post config part and sed -i /etc/fstab.


that said i don't think i like gnome3 thus far. I have read that it is really ment for a tablet or phone, I can see why it would be good there, but i got a mouse, honest. Might be better if the "minimized activities" dropped to the 2nd monitor or something but it is kinda slow moving around in it as is, which is really sad because its very snappy. Its still pretty new so i will hold back a finally verdict. x'ing fingers though.

[mainlylinux]

I know this thread is several months old, but I wonder if this is still an issue. I installed Fedora 15 x64 from the Live CD to an iSCSI target and everything looked pretty good but on reboot I get dropped to an emergency shell and it doesn't look like the root filesystem mounted. I tried this on a second computer with the same result.

I downloaded Fedora 16 Alpha Live CD to see if it had been fixed and I had the same issue.

I am using iSCSI boot (iBFT) on an Intel CT/Gigabit PCIe NIC. I just got the newest Intel iSCSI firmware and flashed yesterday, and Windows 7 works fine on this same setup.

I had read that I could try and troubleshoot Dracut using a serial port connection but neither motherboard has serial ports. Anyone else have any experience with a setup like this?

Thanks,

Dan

[stoggy]

what does your kernel line look like in your iscsi's grub.conf file look like?

should be something like this:

kernel pxelinux.cfg/vmlinuz-2.6.38.6-27.fc15.x86_64 ro root=/dev/mapper/vg0-lv_root netroot=iscsi:@{ISCSI-SERVER-IP}::3260::{ISCSI-TARGET} iscsi_initiator={ISCSI-TARGET} ip=eth0:dhcp ifname=eth0:{MACADDRESS} rd_LVM_LV=vg0/lv_root rd_LVM_LV=vg0/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us noapic acpi=off rhgb quiet

[mainlylinux]

It looks like this:


menuentry 'Fedora Linux, with Linux 3.1.0-0.rc3.git0.0.fc16.x86_64' --class fedora --class gnu-linux --class $
load_video
set gfxpayload=keep
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 52b9fdbc-67ce-44e2-93c5-9ec0948a4721
echo 'Loading Linux 3.1.0-0.rc3.git0.0.fc16.x86_64 ...'
linux /vmlinuz-3.1.0-0.rc3.git0.0.fc16.x86_64 root=/dev/mapper/vg_vmhost01-lv_root ro iscsi_firmwar$
echo 'Loading initial ramdisk ...'
initrd /initramfs-3.1.0-0.rc3.git0.0.fc16.x86_64.img
}

---------- Post added at 12:12 AM ---------- Previous post was at 12:06 AM ----------

I didn't mention this, but I'm not pxe booting. My NIC has iSCSI firmware which passes iBFT information about the target.

Dan

[stoggy]

oh F16 is using lilo? Yuck. Don't forget to re-run lilo!!!


You need to check this line:

linux /vmlinuz-3.1.0-0.rc3.git0.0.fc16.x86_64 root=/dev/mapper/vg_vmhost01-lv_root ro iscsi_firmwar$


you have to specify the iscsi initiator, netroot, ifname, and ip on the kernel line so it can boot. My guess is you are not doing so or something is wrong so it cant find the iscsi target. That or your iscsi target has something wrong but considering you already installed, probably not.


# IP::PORT::iscsi target
netroot=iscsi:@192.168.0.2::3260::iqn.2111-11.net.bob:bobbins.rootlun

# iscsi target
iscsi_initiator=iqn.2111-11.net.bob:bobbins.rootlun

# NIC:IP/dhcp
ip=eth3:dhcp

# NIC to use to get dhcp or use IP with
ifname=eth3:00:A9:B876:E5


the other stuff on the kernel line should be normal stuff. The options above were correct in F15 and CentOS 5. I haven't tried F16 or RHEL6 yet. You could uncompress the initrd and see what the variables are if they have changed. Just walk through the boot order in the initrd.

[mainlylinux]

Thanks for the advice. I'll try and update my grub.cfg later tonight and see if that helps. Grub.cfg says "don't modify" but since the system won't boot, I guess I have no other choice right?

Dan

[stoggy]

yep, its not passing the correct root to the initrd.

the stuff you pasted above looks like lilo though.

[mainlylinux]

it seems like the installer should build grub.cfg differently, doesn't it?

[stoggy]

If you mean that the installer should build it differently then yea, i guess. It doesn't add any of the iscsi options that you will need to boot the iscsi target that you installed on.

I saw the iscsi setup in the gui but you cant use that if you want to load to an iscsi root, it is to late in the setup. You have to specify the iscsi root target on the kernel line when you boot setup too.

There are no iscsi options for disks in the pxecfg file, at least not when i was first doing this. So if you use the iscsi setup page in the gui setup you will double mount your iscsi target and everything hangs. I don't think that page is available on the cui setup.

---------- Post added at 07:23 AM ---------- Previous post was at 07:05 AM ----------

hmm... i re-read my own walk through. LOL...

I guess I am wrong. I did use the advanced disk services to add my iscsi target. Yea the boot disk is created wrong.

That is what the post section is for in kickstart.

[stoggy]

I tried last night with Fedora 16. I keep getting an error when trying to scan for iscsi targets on the pxebooting machine. Something about not iterable, Capable of being iterated or repeated, thats kind of what i thought it ment. Submitted a bug. Haven't tried on console install yet. May have to do a straight ks install. the error looks like gui stuff, "pasted below". More to come...If i dont reply back in 6 hours call the police. <* that is a joke, in case any bed wett'ers are reading this.


libreport version: 2.0.6
executable: /usr/bin/python
hashmarkername: anaconda
kernel: 3.1.0-7.fc16.x86_64
product: Fedora
reason: TypeError: 'NoneType' object is not iterable
time: Mon Nov 21 02:04:17 2011
version: 16

anaconda-tb-ul5pld: Text file, 128736 bytes

description:
:The following was filed automatically by anaconda:
:anaconda 16.25 exception report
:Traceback (most recent call first):
: File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/iscsi.py", line
247, in discover
: return [n for n in found_nodes if n not in self.nodes]
: File "/usr/lib64/python2.7/site-packages/pyanaconda/partIntfHelpers.py",
line 465, in drive_iscsi_addition
: found_nodes = anaconda.storage.iscsi.discover(**discovery_dict)
: File "/usr/lib64/python2.7/site-packages/pyanaconda/iw/advanced_storage.py",
line 411, in addIscsiDrive
: login_ok_nodes = pih.drive_iscsi_addition(anaconda, wizard)
: File "/usr/lib64/python2.7/site-packages/pyanaconda/iw/advanced_storage.py",
line 477, in addDrive
: rc = addIscsiDrive(anaconda)
: File "/usr/lib64/python2.7/site-packages/pyanaconda/iw/filter_gui.py", line
454, in _add_advanced_clicked
: if not addDrive(self.anaconda):
:TypeError: 'NoneType' object is not iterable