Recently I ran into a situation where log messages from a service implemented in Golang were missing from a Linux system. To be able to easily troubleshoot this problem and quickly iterate over possible solutions I utilized Docker and ran systemd inside a Docker container.

Introduction

The Golang service in question sometimes generates a lot of log output. Sometimes up to 50,000 log messages in just a few seconds. And sometimes e.g. the last 20,000 log messages of the service were just missing from the logs managed by journald / systemd. To be able to reproduce this problem I implemented 2 simple programs (one written in C, the other writen in Go). My first guess was that this had something to do with buffered / unbuffered output streams.

Running systemd inside a Docker container

So to be able to easily troubleshoot this problem and quickly iterate over possible solutions I utilized Docker. To start systemd inside a Docker container a few pre-requisites have to be met:

1. systemd has to be installed inside the container of course. It provides e.g. the /sbin/init binary. Using the fedora:34 Docker image this can be achieved by installing httpd for example (the Apache web server).

2. The cgroup file system has to be mounted inside the container (-v /sys/fs/cgroup:/sys/fs/cgroup:ro). We do this in read-only mode.

3. The /tmp and /run mount points have to be present inside the container (--tmpfs /tmp --tmpfs /run).

So if you just want to get systemd and the Apache web server up and running inside a Docker container just clone the source code repository located here and execute the following command inside the folder running-systemd-inside-a-docker-container:

1. Build the container image: docker build . -t sysd

2. Start a container: docker run --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 9090:80 --name sysd --rm sysd

3. To get a shell inside the container execute the following command in a second shell: docker exec -it sysd bash

4. Now you may also open the URL http://localhost:9090/ in your web browser to view he page served by the Apache web server running as a child process of systemd inside the container.

Troubleshooting journald

If you want to play around with the integration of C or Golang programs with journalctl you can take a closer look at the cprinter and goprinter programs, their corresponding systemd service files and the journald config file.

To build the cprinter program change into it’s folder and execute the following command:

gcc -Wall cprinter.c -o cprinter

To build the goprinter program change into it’s folder and execute the following command:

go build

Next start the container using the following command:

docker run --tmpfs /tmp --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 9090:80 --name sysd --rm \
  -v $(pwd)/cprinter/cprinter:/appl/cprinter \
  -v $(pwd)/cprinter/cprinter.service:/etc/systemd/system/cprinter.service \
  -v $(pwd)/goprinter/goprinter:/appl/goprinter \
  -v $(pwd)/goprinter/goprinter.service:/etc/systemd/system/goprinter.service \
  -v $(pwd)/journald.conf:/etc/systemd/journald.conf \
  sysd

After the container is up and running you can then get a shell inside the container and test the generation of log messages using the following commands:

[root@08e24ab748c8 /]# systemctl start cprinter
[root@08e24ab748c8 /]# journalctl -u cprinter

Summary

At first I thought this had something to do with buffered / unbuffered output streams, but in the end it all boiled down to the default log message rate limits enforced by journald. Writing to stdout or stderr using the Go stdlib is never buffered by the way (no matter if using fmt.Printf or log.Logger). Also see here.

Environment

The following programs / tools were used while writing this blog post:

• Kubuntu 21.04
• Docker 20.10.2
• Go 1.16.3