Recently I ran into a situation where log messages from a
service implemented in Golang were missing
from a Linux system. To be able to easily troubleshoot this problem and quickly
iterate over possible solutions I utilized Docker and
systemd inside a Docker container.
The Golang service in question sometimes generates a lot of log output.
Sometimes up to 50,000 log messages in just a few seconds. And sometimes
e.g. the last 20,000 log messages of the service were just missing
from the logs managed by
journald / systemd.
To be able to reproduce this problem I implemented 2 simple
programs (one written in C, the other writen in Go). My
first guess was that this had something to do with
buffered / unbuffered output streams.
Running systemd inside a Docker container
So to be able to easily troubleshoot this problem and quickly
iterate over possible solutions I utilized Docker. To start
systemd inside a Docker container a few pre-requisites have
to be met:
systemdhas to be installed inside the container of course. It provides e.g. the
/sbin/initbinary. Using the
fedora:34Docker image this can be achieved by installing
httpdfor example (the Apache web server).
cgroupfile system has to be mounted inside the container (
-v /sys/fs/cgroup:/sys/fs/cgroup:ro). We do this in read-only mode.
/runmount points have to be present inside the container (
--tmpfs /tmp --tmpfs /run).
So if you just want to get
systemd and the Apache web server
up and running inside a Docker container just clone
the source code repository located here and execute the
following command inside the folder
Build the container image:
docker build . -t sysd
Start a container:
docker run --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 9090:80 --name sysd --rm sysd
To get a shell inside the container execute the following command in a second shell:
docker exec -it sysd bash
Now you may also open the URL http://localhost:9090/ in your web browser to view he page served by the Apache web server running as a child process of
systemdinside the container.
If you want to play around with the integration of
C or Golang programs with
journalctl you can take a closer
look at the
goprinter programs, their
systemd service files and the
To build the
cprinter program change into it’s folder and execute
the following command:
gcc -Wall cprinter.c -o cprinter
To build the
goprinter program change into it’s folder and execute
the following command:
Next start the container using the following command:
docker run --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 9090:80 --name sysd --rm \ -v $(pwd)/cprinter/cprinter:/appl/cprinter \ -v $(pwd)/cprinter/cprinter.service:/etc/systemd/system/cprinter.service \ -v $(pwd)/goprinter/goprinter:/appl/goprinter \ -v $(pwd)/goprinter/goprinter.service:/etc/systemd/system/goprinter.service \ -v $(pwd)/journald.conf:/etc/systemd/journald.conf \ sysd
After the container is up and running you can then get a shell inside the container and test the generation of log messages using the following commands:
[root@08e24ab748c8 /]# systemctl start cprinter [root@08e24ab748c8 /]# journalctl -u cprinter
At first I thought this had something to do
with buffered / unbuffered output streams, but in the end
it all boiled down to the default log message rate limits enforced
journald. Writing to stdout or stderr using the Go stdlib is
never buffered by the way (no matter if using fmt.Printf or log.Logger).
Also see here.
The following programs / tools were used while writing this blog post:
- Kubuntu 21.04
- Docker 20.10.2
- Go 1.16.3
A note about Netcup (advertisement)
Netcup is a German hosting company. Netcup offers inexpensive, yet powerfull web hosting packages, KVM-based root servers or dedicated servers for example. Using a coupon code from my Netcup coupon code web app you can even save more money (6$ on your first purchase, 30% off any KVM-based root server, ...).