WARNING don't copy commands on this page blindly, you can easily end up messing your install. e.g. rm /sbin/init in the host system and not the container rootfs would not be nice.The aim of this is to create and Ubuntu build server with
libc,
python,
nodejs,
build-essentials, all at exactly the same version as Ubuntu but
without systemd, using
apt to install everything.
My goal is to run LXC containers with as little RAM overhead as possible. The container host I'm running on has 500MB of RAM, I'm quite tight with resources.
TL;DR
- create lxinitd based lxc container
- run the ubuntu lxc template script
- revert lxinitd's /sbin/init and /etc/rc.local
- add core services to /etc/rc.local replacing systemd inits
Step 1 Create the lxinitd based server.
Install
lxinitd.
Create a container called "ci".
lxc-create -t lxinitd -n ci -f /etc/lxc/default.confAt this point I then added some mounts, and I remove the
/lib /lib64 and
/usr/lib mounts defined in
/var/lib/lxc/ci/config.
In this case I want the container to have its own copy of all libs installed via .debs, by default, lxinitd mount libs from the container host.
I then eddited lxinitd's boot script
$rootfs/etc/rc.local, you can have lxc do the networking but I prefer to do it myself with
ip, I'm not using Ubuntu/Debian style
/etc/network/interfaces or
dhcp.
My
$rootfs/etc/rc.local looks like this
#!/bin/lxinitd
respawn /sbin/getty -L tty1 115200 vt100
/sbin/ip addr add $ip_address/24 dev eth0
/sbin/ip route add default via 10.0.3.1You may want to do something different with your base lxc container.
Step 2 Overlay the Ubuntu container bootstrap
First backup
$rootfs/etc/rc.local since debootstrapping will overwrite it.
Remove the
$rootfs/sbin/init symlink to lxinitd.
mv $rootfs/etc/rc.local $rootfs/etc/rc.local.tmp
rm $rootfs/sbin/initTake a copy of
/usr/share/lxc/templates/lxc-ubuntu so it can be edited an run independently of lxc.
The main change to make to this script is to permanently disable running systemd services when apt installs applications.
The default
lxc-ubuntu template already disables running systemd servers, but it enables it again for the final configuration.
echo exit 101 > $rootfs/usr/sbin/policy-rc.d
chmod +x $rootfs/usr/sbin/policy-rc.dBy leaving and executable
$rootfs/usr/sbin/policy-rc.d in the filesystem apt knows not to try to run commands during installations, this is necessary when you are installing in a chroot.
Edit the parts of the
lxc-ubuntu script which remove
policy-rc.d.
With
policy-rc.d changes made, run the script as follows, add any of the packages you need in the base system. I'm adding sshd here but you don't need todo that.
./lxc-ubuntu.sh --name ci --rootfs /var/lib/lxc/ci/rootfs --path /var/lib/lxc/ci --packages openssh-server
Step 3 Revert the lxinitd init process
The next step is to restore lxinit's init, first remove
/sbin/init (which will be a symlink to
/bin/systemd), symlink
/sbin/init to
/bin/lxinitd, then copy back our
rc.local.
cd $rootfs/sbin
rm init
ln -s ../bin/lxinitd init
mv $rootfs/etc/rc.local.tmp $rootfs/etc/rc.local
Step 4 Configure systemd services replacements
To be honest I thought this stage would be painful, perhaps impossible. But it was very easy.
systemd peeps decided to own DNS resolving, which is arguably not the remit of an init system, this container does not have
systemd-resolved running.
The fix is trivial, just put a real DNS server in
/etc/resolv.conf, this one is the OpenDNS public server.
echo 'nameserver 208.67.222.222' > etc/resolv.confObviously if you have a local DNS or dnsmasq running on the container host or local network, use that.
Ensure that
/etc/hostname and
/etc/hosts and
lxc.utsname report the same string for the local hostname.
Configure syslog, crond and sshd, by adding a couple of lines to
/etc/rc.local.
mkdir -p /var/run/sshd
service /var/run/crond.pid /usr/sbin/cron -f
service /var/run/rsyslogd.pid /usr/sbin/rsyslogd
service /var/run/sshd.pid /usr/sbin/sshd -DI really like the fact that setting up services in lxinitd is a one-liner.
And that was it. Nothing more was needed to replace the init system entirely. systemd is installed and all its many libs are there on the filesystem but they don't boot.
lxc-start -n ciruns lxinitd and boots services in the container and systemd does not get a look-in.
The ps list of this container looks like this.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 208 4 ? Ss 19:42 0:00 /sbin/init
root 28 0.0 0.0 26072 2456 ? S 19:42 0:00 /usr/sbin/cron -f
root 30 0.0 0.0 65516 5580 ? S 19:42 0:00 /usr/sbin/sshd -D
syslog 31 0.0 0.0 256400 2752 ? Ssl 19:42 0:00 /usr/sbin/rsyslogdi.e. all things that I'minterested inand nothing else.
Equivalent Ubuntu base container ps list looks like this.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.7 154448 3752 ? Ss May24 0:32 /sbin/init
root 38 0.0 0.8 79480 4420 ? Ss May24 0:09 /lib/systemd/systemd-journald
systemd+ 42 0.0 0.1 74464 644 ? Ss May24 0:07 /lib/systemd/systemd-networkd
message+ 66 0.0 0.3 47448 1700 ? Ss May24 0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 69 0.0 0.1 31208 712 ? Ss May24 0:04 /usr/sbin/cron -f
systemd+ 75 0.0 0.4 65696 2072 ? Ss May24 0:08 /lib/systemd/systemd-resolved
root 89 0.0 0.2 72136 1024 ? Ss May24 0:00 /usr/sbin/sshd -D
ci 93 0.0 0.1 79980 828 ? Ss May24 0:00 /lib/systemd/systemd --user
ci 94 0.0 0.3 103780 1624 ? S May24 0:00 (sd-pam)
root 96 0.0 0.0 15876 136 pts/0 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/0 115200,38400,9600 vt220
root 97 0.0 0.0 15876 132 pts/2 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/2 115200,38400,9600 vt220
root 98 0.0 0.0 15876 132 pts/1 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 vt220
root 99 0.0 0.0 15876 140 pts/1 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/1 115200,38400,9600 vt220
root 100 0.0 0.0 15876 136 pts/3 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/3 115200,38400,9600 vt220
syslog 3619 0.0 0.1 256536 664 ? Ssl May24 0:01 /usr/sbin/rsyslogd -n
root 11670 0.0 0.4 65796 2216 ? Ss May25 0:03 /lib/systemd/systemd-logindI dont have dbus, systemd-journald, systemd-networkd, systemd-resolved, systemd-logind, & 5 agetty instances.
on the systemd based lxc container I have ~100 systemd units active, on my lxinitd based container I have
linci> sudo systemctl
Failed to connect to bus: No such file or directoryNext step is to have one instance of rsyslog in the container host system, there is no real need to have it running inside each container.
In a container there is often no need to run cron, sshd or rsyslog, in which case, the base container is taking only 4k of RAM (RSS 4), yet this is a full OS Ubuntu and you can
apt install whatever
I add a other services to the base container to support my builds,
xtomp, nginx, fcgiwrap, ngircd, ii, tsp. All of which are installed
with
apt install xxx
and a oneliner in
/etc/rc.local.
Clearly lots of things in the official repos are going to be broken . You could not boot Gnome3 off an lxc / lxinitd based container. I doubt such trickery would work with a RedHat sponsored distro.
For an Ubuntu like buildbox this is perfect.