lxc

set up

arch

archwiki_setup redhat_guide linux_containers_guide

  • install lxc 
    pacman -S lxc dnsmasq
  • or on debian 
    apt-get install lxc dnsmasq-base uidmap acl libpam-cgfs
echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf
reboot
  • add the following line to '/etc/pam.d/system-login'
  • (debian '/etc/pam.d/login') 
    session    optional    pam_cgfs.so    -c freezer,memory,name=systemd,unified
  • create '/etc/default/lxc-net' config 
    # Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
# containers.  Set to "false" if you'll use virbr0 or another existing
# bridge, or mavlan to your host's NIC.
USE_LXC_BRIDGE="true"

# If you change the LXC_BRIDGE to something other than lxcbr0, then
# you will also need to update your /etc/lxc/default.conf as well as the
# configuration (/var/lib/lxc/<container>/config) for any containers
# already created using the default config to reflect the new bridge
# name.
# If you have the dnsmasq daemon installed, you'll also have to update
# /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
# Uncomment the next line if you'd like to use a conf-file for the lxcbr0
# dnsmasq.  For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have
# container 'mail1' always get ip address 10.0.3.100.
#LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

# Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc
# domain.  You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR)
# to your system dnsmasq configuration file (normally /etc/dnsmasq.conf,
# or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager).
# Once these changes are made, restart the lxc-net and network-manager services.
# 'container1.lxc' will then resolve on your host.
#LXC_DOMAIN="lxc"
  • add the following lines to '/etc/lxc/default.conf' 
    lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
  • start lxc-net 
    systemctl restart lxc-net
  • check that lxcbr0 bridge has been created 
    ip a s lxcbr0
  • create '/etc/subuid' 
    pyratebeard:100000:65536
  • create '/etc/subgid' 
    pyratebeard:100000:65536
  • create '/etc/lxc/lxc-usernet' for allowing user to create network devices 
    pyratebeard veth lxcbr0 10
    • veth - virtual ethernet
    • lxcbr0 - network bridge
    • 10 - number of devices allowed
  • create local dirs 
    mkdir ~/.{config,cache}/lxc
mkdir ~/.local/share
  • create '~/.config/lxc/default.conf' 
    lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
  • make '~/.local/share' executable and set acls 
    chmod +x ~/.local/share
setfacl -m u:100000:x /home/pyratebeard
setfacl -m u:100000:x /home/pyratebeard/.local

create container

lxc-create -t download -n <name>
# or
lxc-create -n <name> -t download -- --dist alpine --release 3.13 --arch amd64
lxc-start -d -n <name>
lxc-attach -n <name>
or 
vi ~/.local/share/lxc/powerzone/rootfs/etc/shadow
	# remove `!` from root user
lxc-start -n powerzone
lxc-console -n powerzone
  • python module for script api 5

alpine linux config

/sbin/apk update
/sbin/apk upgrade
passwd
busybox adduser pyratebeard
busybox adduser pyratebeard wheel
apk add doas vim openssh zsh
echo "permit nopass pyratebeard" | tee -a /etc/doas.d/doas.conf
ln -s /bin/zsh /usr/bin/zsh
/sbin/rc-update add sshd
/sbin/rc-service sshd start
/sbin/rc-status
logout (ctrl-a q to exit console)

debian config

passwd
apt-get install openssh-server python3
vi /etc/ssh/sshd_config
	PermitRootLogin yes
systemctl reload sshd

alpine services

add files to /etc/init.d/ 

#!/sbin/openrc-run
name="test"
command="echo hello"

known errors

  • systemd containers fail to start 
    Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.
    • '/sys/fs/cgroup/systemd' dir doesn't exist
    • to fix, create dir, mount cgroup, set permissions lxc-users group post 
      sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
sudo chown pyratebeard:users /sys/fs/cgroup/systemd
  • keyserver not found on lxc-create
  • wait_ondaemonized_startL 833 no such file or directory
    • lxc-start in foreground gives segmentation fault 
      lxc-start -n test /bin/sh
  • unable to start on debian 11 (error "895 received container state aborting" - in forground mode "1365 numerical result out of range")
    • use unpriv start 
      lxc-unpriv-start -n <name>

moving containers

so answer 

lxc-stop -n $NAME
cd ~/.local/share/lxc/$NAME
sudo tar --numeric-owner -czvf ../$NAME.tgz ./*
chown pyratebeard: ../$NAME.tgz
rsync -avh $NAME.tgz user@hostname:.local/share/lxc/
ssh user@hostname
mkdir ~/.local/share/lxc/$NAME
cd ~/.local/share/lxc/$NAME
sudo tar --numeric-owner -xzvf ../$NAME.tgz .
  • tried this between wht-rht-obj and fka
    • container runs (after adding user gid to /etc/subgid)
    • no ip address though. veth is created but ip4 not given
  • check dir/file permissions
    • .local/share/lxc/$NAME = 755 100000:100000
    • .local/share/lxc/$NAME/rootfs/* = 100000:100000
    • .local/share/lxc/$NAME/config = pyratebeard:users

example

setting up multiple websites behind haproxy

  • install openzfs
  • start lx daemon 
    sudo apt install zfsutils-linux
sudo lxd init
  • answer questions
  • launch containers 
    lxc launch ubuntu:18.04 subdomain1
lxc launch ubuntu:18.04 subdomain2
lxc launch ubuntu:18.04 haproxy
lxc list

gollum haproxy log pastebin radicale site stagit znc ftp

debian test

  • debian 10 (aws instance)
    • 'admin' user
  • apt-get install lxc dnsmasq-base uidmap
  • follow setup (see own wiki)
  • building debian containers works well
  • ansible playbook runs using proxyjump in ssh config
  • attempting to run haproxy in container
  • iptables rules for prerouting
    • sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 80 -j DNAT --to-destination <haproxy_ip>:80
    • sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 443 -j DNAT --to-destination <haproxy_ip>:443
    • sudo iptables -L -n -t nat
    • sudo apt-get install iptables-persistent

  • haproxy container

    • apt-get install haproxy
    • add the following to the 'global' section ... maxconn 2048 ... tune.ssl.default-dh-param 2048
    • add the following to the 'defaults' section ... option forwardfor option http-server-close ...
    • create frontend frontend http_frontend bind *:80 acl infratuxture hdr(host) -i penguin.renre.com #acl anotherlxc hdr(host) -i anotherdomain.renre.com use_backend penguin if infratuxture #use_backend anotherdomain if anotherlxc

    • create backend

      backend penguin
        balance leastconn
        http-request set-header X-Client-IP %[src]
        server penguin 10.0.3.162:80 check

#backend anotherdomain
#       balance leastconn
#       http-request set-header X-Client-IP %[src]
#       server anotherdomain an.oth.er.ip:80 check
  • infratuxture container
    • apt-get install git lighttpd
    • pull git repo in html dir cd /var/www/html git clone https://git.renre.com/infrastructure/linux-patching.github.io.git .

bindmount

  • mount a dir on lxc, add follwoing to container conf 
    mp0: /path/on/host,mp=/mount/path/on/container

uid/gid mapping

  • in lxc conf 
    lxc.idmap: u 0 100000 1005
lxc.idmap: g 0 100000 1005
lxc.idmap: u 1005 1005 1
lxc.idmap: g 1005 1005 1
lxc.idmap: u 1006 101006 64530
lxc.idmap: g 1006 101006 64530
  • explanation taken from itsembedded

The format of the lxc.idmap configuration lines are , where selects whether the mapping is for user id’s or group id’s.

Below is an explanation of what each mapping combination does:

  • (u/g) 0 10000 1000 - map 1000 ID’s starting from 0, to ID’s starting at 100000. This means that the ROOT UID/GID 0:0 on the guest will be mapped to 100000:100000 on the host, 1:1 will be mapped to 100001:1000001, and so on.

  • (u/g) 1000 1000 1 - map the UID/GID pair 1000:1000 to 1000:1000 on the host. The number 1 is there to specify we’re only mapping a single ID, and not a range.

  • (u/g) 1001 101000 64535 - map 64535 ID’s starting at 1001, to ID’s starting at 101000. This means that UID/GID pair 1001:1001 on the guest will be mapped to 101000:101000, 1002:1002 to 101001:101001, all the way to finally 65535:65535 to 165534:165534.

mounting zfs dataset in lxc container

  • requires uid/gid mapping
  • this example is for using the www-data user with nextcloud
  • on host 
    zfs set acltype=posixacl pool/dataset
setfacl -m u:100033:rwx /path/to/dataset
  • add mount point as above
  • on container check acl 
    getfacl /path/to/mount

Table of Contents