lxc aka linux containers and fedora

[LaKing]

Hi folks.

I'm playing with LXC aka linux containers.

CAUTION So close to the kernel, if something gets messed up, the host may reboot, get unresponsive, hang, or go down completely ...

Fedora Core 16
At the time of this writing, yum install lxc installs lxc 0.7.4. , and the current kernel version is 3.1.1 on FC16.

if you install lxc with yum now, Cgroup namespace will be missing, as it is not on the most recent lxc version.
I filled a bug report. Bug 756248

CentOS 6
Ha s a 2.x kernel which is fine, lxc is not in the default repository's tho.
Can host Fedora 14 Containers.

..therefore, we need to compile it in all editions.

Host Installation
Uncompress the 0.7.5 lxc tarball, in the lxc directory ...

yum groupinstall "Development Tools"
yum install libcap-devel libvirt

./configure
make
make install

lxc-checkconfig

lxc-checkconfig is the command to check lxc against the kernel.

Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-3.1.1-2.fc16.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/local/bin/lxc-checkconfig

.. everything is enabled.

mount should be able to see if there is cgroup mounted or not, FC16 has, CentOS6 does not.
However, in Centos there is a file, cgroup.conf that seems to responsible for mounting, instead of fstab.
To create it, manually run as root:

mkdir -p /cgroup
mount none -t cgroup /cgroup

echo 'none /cgroup cgroup defaults 0 0' >> /etc/fstab

[LaKing]

Prepare configuration

 gedit /usr/local/lib/lxc/templates/lxc-fedora

Navigate to line 230. In the function copy_configuration is a cat to the config file, and a cat to the fstab file of the container. The fstab's EOF file is missing, so the script itself gets into the fstab.
To correct this, simply write EOF to line 230.

    cat <<EOF > $config_path/fstab
+proc            $rootfs_path/proc         proc    nodev,noexec,nosuid 0 0
+devpts          $rootfs_path/dev/pts      devpts defaults 0 0
+sysfs           $rootfs_path/sys          sysfs defaults  0 0
EOF

If already here, at line 112, add the releasever variable, which is missing.

    YUM="yum --installroot $INSTALL_ROOT -y --nogpgcheck --releasever=$release"

About /etc/rc.sysinit and /etc/rc.d/rc.sysinit ... these are not part of the default fedora install any more, but the script refers to them on line 68.
On line 113 to the PKG_LIST variable for yum, add initscripts legacy, and whatever is appropriate: mc openssh-server ...

Save that lxc-template, or save as /root/lxc-fedora

To fix a configuration path error, link /var/lib/lxc to /usr/local/var/lib/lxc

cd /usr/local
mkdir var
cd var
mkdir lib
cd lib
ln -s  /var/lib/lxc

Container Installation from scratch.
Execute, the saved fedora-lxc template. X marks the release. Eg 14 for Fedora 14. fcX will be the name of the container.

./lxc-fedora -R X -n fcX

/var/lib/lxc/... is the default location of the containers.

[LaKing]

Container start

lxc-start -n fcX

In case there is no container created or it's destroyed, an error might show up:

lxc-start: no configuration file for '/sbin/init' (may crash the host)

lxc-create creates an empty container.
The config file should be given as an argument explicitly!
Starting an empty container will reboot the host.I thought lxc-fedora is a replacement, or a special version of lxc-create,
.. well, it is not, if the symlink in /usr/local/var/lib/lxc is not set up properly.


Destroying the container is the proper way to delete the rootfs.

lxc-destroy -n fcX

NOTE: FC16 will report the error message about udev-post, FC14 for example, wont. udev wont work in containers anyway.

The following commands should work now properly. The -n argument should never be omitted, always name the container.

lxc-start -n fcX -l DEBUG -o /path-to/fcX.log

lxc-console -n fcX

lxc-stop -n fcX   

lxc-info -n fcX
 _______________
  state: RUNNING
  pid: 5678

.. where fcX is your container name.

Note: If there is only a message in the console like:

lxc-start: inherited fd 7 on pipe:[35622]

.. comes from the console, mc might be running.

[LaKing]

Shutdown -P now in lxc-console

"Specifically, you do not want /dev/initctl from the host to be mapped into your container, unless you want your containers to be able to shut down or reboot your controlling host."

.. that is true.

Simply delete /dev/initctl from the container rootfs.

[LaKing]

File 'sharing'

As far as I see ...
The /etc/fstab file, in a 'fresh' fsroot does not exist. It should be created, so that there are no (or much less) error messages, since lxc-fedora didnt create it. (modify it if you like to have it automatically done.)

The config has an fstab entry, which is processed, ... however, even if the fstab directive is commented out in the config, the necessary dev, proc, sysfs (...) mounts wont be missing.
That means, the whole fstab can be full of mounts.

Within a container, the command mount will always return

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

.. which seem to come fom the containers /etc/mtab dynamically.


Note that the lxc.mount.entry directive works also, as additive entry.

lxc.mount.entry = /what /var/lib/lxc/fcX/rootfs/where none ro,bind 0 0

Networking.

yum Install libvirt

on the host

The host can be set up as bridge. Network Manager should be turned off, and the interfaces eth0, [eth1, ..] and br0 configured.

The container config should use the network br0, have IP assigned. ... and in the container, should be also the networking-script to set up eth0 - which is only a virtual interface, connected to the br0 of the host.
/etc/resolv.conf can be mapped from the host, or should contain a copy of it, in order to have DNS requests served.

.. if everything works fine, ping should work in the container.

The networking part is done by libvirt.

[LaKing]

CentOs 6 Containers

The lxc-fedora template can be modified to create CentOs Containers.
Simply run a search and replace, replace "fedora" with "centos" and modify the rpm package name and URL. Line 114.

RELEASE_URL="http://ftp.funet.fi/pub/mirrors/centos.org/$release.0/os/$arch/Packages/centos-release-$release-0.el6.centos.5.$arch.rpm"
...
rpm --root $INSTALL_ROOT -ivh $INSTALL_ROOT/centos-release-$release-0.el6.centos.5.$arch.rpm

Save as lxc-centos. Invoke with the name and release variables.

CentOs 6 containers run on CentOs 6.

[schorschi]

To establish LXC in a stronger, enterprise architecture context? Would it not be a best practice to route all container traffic through a separate nic? And maybe allow use of bridge to the separate nic? Using libvirt is fine, but NAT is considered unsafe by many secuirty entities.

1) so host platform uses eth0
2) containers use eth1
3) br0 on eth1, so containers are visible to external entities? eliminates need for use of NAT?

[LaKing]

In a complete bridged setup, on a public machine, you will need static IP addressing for each container. ..

.. at the moment I don't see a real reason to use seperate NIC's for host/containers.

[LaKing]

Ramez Hanna @ lxc-users mailing list wrote:

here is is how o got f16 to work
* use the shipped fedora template to create the container
* chroot into the container rootfs
* touch /etc/fstab
* ln -s /dev/null /etc/systemd/system/udev.service
* unlink /etc/systemd/system/default.target
* ln -s /lib/systemd/system/multi-user.taget /etc/systemd/system/default.target
if you want to setup a getty
* ln -s /lib/systemd/system/getty@.service
/etc/systemd/system/getty.target.wants/getty@tty1.service
* exit the chroot

if you had installed sshd in the rootfs then ssh is ready you can just ssh in

the problem i am facing right now is that i am unable to stop systemd
from mounting /dev
which leads to not possible to access the lxc-console because the
container is using tty* from the host and not the ones created by lxc
which also means that if you pick a higher tty (above the ones used by
your host and allow it in the cgroup conf) then you can access your
container's tty using the ctrl-alt-Fx keys

any one wants to contribute or comment please do
i will start working on the template now and soon send patches

[LaKing]

From: Ramez Hanna

* fix cached rootfs update
* fix rootfs path
* add handling of systemd (aka >f15)

---
templates/lxc-fedora.in | 33 +++++++++++++++++++++++++--------
1 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/templates/lxc-fedora.in b/templates/lxc-fedora.in
index e7f42a6..1873373 100644
--- a/templates/lxc-fedora.in
+++ b/templates/lxc-fedora.in
@@ -69,11 +69,6 @@ EOF
127.0.0.1 localhost $name
EOF

- sed -i 's|.sbin.start_udev||' ${rootfs_path}/etc/rc.sysinit
- sed -i 's|.sbin.start_udev||' ${rootfs_path}/etc/rc.d/rc.sysinit
- chroot ${rootfs_path} chkconfig udev-post off
- chroot ${rootfs_path} chkconfig network on
-
dev_path="${rootfs_path}/dev"
rm -rf $dev_path
mkdir -p $dev_path
@@ -99,6 +94,21 @@ EOF

return 0
}
+configure_fedora_init()
+{
+ sed -i 's|.sbin.start_udev||' ${rootfs_path}/etc/rc.sysinit
+ sed -i 's|.sbin.start_udev||' ${rootfs_path}/etc/rc.d/rc.sysinit
+ chroot ${rootfs_path} chkconfig udev-post off
+ chroot ${rootfs_path} chkconfig network on
+}
+
+configure_fedora_systemd()
+{
+ unlink ${rootfs_path}/etc/systemd/system/default.target
+ touch ${rootfs_path}/etc/fstab
+ chroot ${rootfs_path} ln -s /dev/null //etc/systemd/system/udev.service
+ chroot ${rootfs_path} ln -s /lib/systemd/system/multi-user.target /etc/systemd/system/default.target
+}

download_fedora()
{
@@ -170,7 +180,8 @@ copy_fedora()

update_fedora()
{
- chroot $cache/rootfs yum -y update
+ YUM="yum --installroot $cache/rootfs -y --nogpgcheck"
+ $YUM update
}

install_fedora()
@@ -353,7 +364,7 @@ if [ "$(id -u)" != "0" ]; then
fi


-rootfs_path=$path/$name/rootfs
+rootfs_path=$path/rootfs
config_path=$default_path/$name
cache=$cache_base/$release

@@ -362,7 +373,7 @@ revert()
echo "Interrupted, so cleaning up"
lxc-destroy -n $name
# maybe was interrupted before copy config
- rm -rf $path/$name
+ rm -rf $path
rm -rf $default_path/$name
echo "exiting..."
exit 1
@@ -388,6 +399,12 @@ if [ $? -ne 0 ]; then
exit 1
fi

+type /bin/systemd >/dev/null 2>&1
+if [ $? -ne 0 ]; then
+ configure_fedora_init
+else
+ configure_fedora_systemd
+fi

if [ ! -z $clean ]; then
clean || exit 1
--
1.7.7.6

[LaKing]

lxc 0.8.0-rc1 is out.

The releasever problem still presist.