The aim of this is to create and Ubuntu build server with libc, python, nodejs, build-essentials, all at exactly the same version as Ubuntu but without systemd, using apt to install everything.
My goal is to run LXC containers with as little RAM overhead as possible. The container host I'm running on has 500MB of RAM, I'm quite tight with resources.
TL;DR
- create lxinitd based lxc container
- run the ubuntu lxc template script
- revert lxinitd's /sbin/init and /etc/rc.local
- add core services to /etc/rc.local replacing systemd inits
Step 1 Create the lxinitd based server.
Install lxinitd.
Create a container called "ci".
lxc-create -t lxinitd -n ci -f /etc/lxc/default.conf
At this point I then added some mounts, and I remove the /lib /lib64 and /usr/lib mounts defined in /var/lib/lxc/ci/config.
In this case I want the container to have its own copy of all libs installed via .debs, by default, lxinitd mount libs from the container host.
I then eddited lxinitd's boot script $rootfs/etc/rc.local, you can have lxc do the networking but I prefer to do it myself with ip, I'm not using Ubuntu/Debian style /etc/network/interfaces or dhcp.
My $rootfs/etc/rc.local looks like this
#!/bin/lxinitd
respawn /sbin/getty -L tty1 115200 vt100
/sbin/ip addr add $ip_address/24 dev eth0
/sbin/ip route add default via 10.0.3.1
You may want to do something different with your base lxc container.
Step 2 Overlay the Ubuntu container bootstrap
Remove the $rootfs/sbin/init symlink to lxinitd.
mv $rootfs/etc/rc.local $rootfs/etc/rc.local.tmp
rm $rootfs/sbin/init
Take a copy of /usr/share/lxc/templates/lxc-ubuntu so it can be edited an run independently of lxc.
The main change to make to this script is to permanently disable running systemd services when apt installs applications.
The default lxc-ubuntu template already disables running systemd servers, but it enables it again for the final configuration.
echo exit 101 > $rootfs/usr/sbin/policy-rc.d
chmod +x $rootfs/usr/sbin/policy-rc.d
By leaving and executable $rootfs/usr/sbin/policy-rc.d in the filesystem apt knows not to try to run commands during installations, this is necessary when you are installing in a chroot.
Edit the parts of the lxc-ubuntu script which remove policy-rc.d.
With policy-rc.d changes made, run the script as follows, add any of the packages you need in the base system. I'm adding sshd here but you don't need todo that.
./lxc-ubuntu.sh --name ci --rootfs /var/lib/lxc/ci/rootfs --path /var/lib/lxc/ci --packages openssh-server
Step 3 Revert the lxinitd init process
cd $rootfs/sbin
rm init
ln -s ../bin/lxinitd init
mv $rootfs/etc/rc.local.tmp $rootfs/etc/rc.local
Step 4 Configure systemd services replacements
To be honest I thought this stage would be painful, perhaps impossible. But it was very easy.
systemd peeps decided to own DNS resolving, which is arguably not the remit of an init system, this container does not have systemd-resolved running.
The fix is trivial, just put a real DNS server in /etc/resolv.conf, this one is the OpenDNS public server.
echo 'nameserver 208.67.222.222' > etc/resolv.conf
Obviously if you have a local DNS or dnsmasq running on the container host or local network, use that.
Ensure that /etc/hostname and /etc/hosts and lxc.utsname report the same string for the local hostname.
Configure syslog, crond and sshd, by adding a couple of lines to /etc/rc.local.
mkdir -p /var/run/sshd
service /var/run/crond.pid /usr/sbin/cron -f
service /var/run/rsyslogd.pid /usr/sbin/rsyslogd
service /var/run/sshd.pid /usr/sbin/sshd -D
I really like the fact that setting up services in lxinitd is a one-liner.
And that was it. Nothing more was needed to replace the init system entirely. systemd is installed and all its many libs are there on the filesystem but they don't boot.
lxc-start -n ci
runs lxinitd and boots services in the container and systemd does not get a look-in.
The ps list of this container looks like this.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 208 4 ? Ss 19:42 0:00 /sbin/init
root 28 0.0 0.0 26072 2456 ? S 19:42 0:00 /usr/sbin/cron -f
root 30 0.0 0.0 65516 5580 ? S 19:42 0:00 /usr/sbin/sshd -D
syslog 31 0.0 0.0 256400 2752 ? Ssl 19:42 0:00 /usr/sbin/rsyslogd
i.e. all things that I'minterested inand nothing else.
Equivalent Ubuntu base container ps list looks like this.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.7 154448 3752 ? Ss May24 0:32 /sbin/init
root 38 0.0 0.8 79480 4420 ? Ss May24 0:09 /lib/systemd/systemd-journald
systemd+ 42 0.0 0.1 74464 644 ? Ss May24 0:07 /lib/systemd/systemd-networkd
message+ 66 0.0 0.3 47448 1700 ? Ss May24 0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 69 0.0 0.1 31208 712 ? Ss May24 0:04 /usr/sbin/cron -f
systemd+ 75 0.0 0.4 65696 2072 ? Ss May24 0:08 /lib/systemd/systemd-resolved
root 89 0.0 0.2 72136 1024 ? Ss May24 0:00 /usr/sbin/sshd -D
ci 93 0.0 0.1 79980 828 ? Ss May24 0:00 /lib/systemd/systemd --user
ci 94 0.0 0.3 103780 1624 ? S May24 0:00 (sd-pam)
root 96 0.0 0.0 15876 136 pts/0 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/0 115200,38400,9600 vt220
root 97 0.0 0.0 15876 132 pts/2 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/2 115200,38400,9600 vt220
root 98 0.0 0.0 15876 132 pts/1 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 vt220
root 99 0.0 0.0 15876 140 pts/1 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/1 115200,38400,9600 vt220
root 100 0.0 0.0 15876 136 pts/3 Ss+ May24 0:00 /sbin/agetty --noclear --keep-baud pts/3 115200,38400,9600 vt220
syslog 3619 0.0 0.1 256536 664 ? Ssl May24 0:01 /usr/sbin/rsyslogd -n
root 11670 0.0 0.4 65796 2216 ? Ss May25 0:03 /lib/systemd/systemd-logind
I dont have dbus, systemd-journald, systemd-networkd, systemd-resolved, systemd-logind, & 5 agetty instances.
on the systemd based lxc container I have ~100 systemd units active, on my lxinitd based container I have
linci> sudo systemctl
Failed to connect to bus: No such file or directory
Next step is to have one instance of rsyslog in the container host system, there is no real need to have it running inside each container.
In a container there is often no need to run cron, sshd or rsyslog, in which case, the base container is taking only 4k of RAM (RSS 4), yet this is a full OS Ubuntu and you can
apt install whatever
I add a other services to the base container to support my builds, xtomp, nginx, fcgiwrap, ngircd, ii, tsp. All of which are installed with
apt install xxx
and a oneliner in /etc/rc.local.
Clearly lots of things in the official repos are going to be broken . You could not boot Gnome3 off an lxc / lxinitd based container. I doubt such trickery would work with a RedHat sponsored distro.
For an Ubuntu like buildbox this is perfect.
