Podman Quadlet Troubleshooting on Linux
Podman Quadlet is a feature to generate systemd services for rootless or rootful containers in Podman. This is not a primer for it, but a troubleshooting reference.
We've been using it more and more, so this will be our reference from now on. (I'll make it easier to follow when I have more time).
Most commands have been used and tested on Ubuntu Server 24.04 LTS. I’ll focus on rootless, but add the rootful version as well.
Your Variables #
Use the following form to insert your names to make it easier to copy and paste:
I try to follow the Podman Quadlet process - feel free to start at the top or jump in where the problems begin
random-app.container
(Quadlet file in one of the search paths)
>>>
podman-user-generator / podman-system-generator
(runs on boot and on every daemon-reload)
>>>
random-app.service
(generated systemd unit)
>>>
ExecStart=/usr/bin/podman run ...
(started by systemd)
>>>
conmon + OCI runtime (crun/runc)
>>>
container process
Paths, extensions and permissions #
Make sure the Quadlet files are saved in the correct directories:
[...] meaning duplicate named quadlets found under /run take precedence over ones in /etc, as well as those in /usr
# rootless
$XDG_RUNTIME_DIR/containers/systemd/
$XDG_CONFIG_HOME/containers/systemd/ or ~/.config/containers/systemd/
/etc/containers/systemd/users/${UID}
/etc/containers/systemd/users/
/usr/share/containers/systemd/users/${UID}
/usr/share/containers/systemd/users/
---
# rootful
/run/containers/systemd/
/etc/containers/systemd/
/usr/share/containers/systemd/
Side note: Quadlet files in /etc/containers/systemd/users/ are picked up by every user - use /etc/containers/systemd/users/${UID}/ to limit it to a single user. The /usr/share/containers/systemd/users/ paths are only supported by recent Podman versions and are not available on Ubuntu 24.04 (Podman 4.9.3). man podman-systemd.unit shows the paths of your installed version.
Accepted file extensions:
.container
.volume
.network
.kube
.image
.pod # Podman 5.x - not on Ubuntu 24.04
.build # Podman 5.x - not on Ubuntu 24.04
.artifact # recent Podman 5.x - not on Ubuntu 24.04
Permissions of Quadlet files
There is no required permission mode, but there are some things to consider:
- The relevant systemd manager must be able to read the file.
- Every parent directory must be traversable.
- Untrusted users must not be able to modify the file.
600 for files, 700 for dirs should do the trick.
Basic Troubleshooting #
Reload the systemd config with systemctl --user daemon-reload and restart the service with systemctl --user restart random-app.service after every change!
Podman Quadlet package installed and ready #
Quadlet was added in Podman 4.4 - Ubuntu 22.04 LTS ships Podman 3.4.4, so there is no Quadlet at all. Ask me how I know...
# Check Podman version - Quadlet requires 4.4+
podman --version
# Check package
dpkg -L podman | grep -E 'podman-(system|user)-generator'
# Make sure the generators are there - both are symlinks to /usr/libexec/podman/quadlet
# rootful
ls -l /usr/lib/systemd/system-generators/podman-system-generator
# rootless
ls -l /usr/lib/systemd/user-generators/podman-user-generator
# Check cgroup version - we need `v2`
podman info --format '{{.Host.CgroupsVersion}}'
Dry-run Quadlet generator #
# Run it read-only to test function
# rootless
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
# limit dry-run to a single directory
QUADLET_UNIT_DIRS="$HOME/.config/containers/systemd" \
/usr/lib/systemd/system-generators/podman-system-generator \
--user --dryrun
# rootful
sudo /usr/lib/systemd/system-generators/podman-system-generator --dryrun
# only show errors, plus additional checks of the generated unit
# rootless
systemd-analyze --user --generators=true verify random-app.service
# rootful
sudo systemd-analyze --generators=true verify random-app.service
This will output all services and configs generated - and the errors for every file that has been skipped:
quadlet-generator[1305164]: converting "random-app.container": unsupported key 'FooBar' in group 'Container' in /home/user/.config/containers/systemd/random-app.container
quadlet-generator[1305164]: processing encountered some errors
Trigger unit generation #
Translates the Quadlet files to .service files
# rootless
systemctl --user daemon-reload
# rootful
sudo systemctl daemon-reload
The service name depends on the type of quadlet file:
random-app.container > random-app.service
random-app.kube > random-app.service
random-app.volume > random-app-volume.service
random-app.network > random-app-network.service
random-app.pod > random-app-pod.service
random-app.image > random-app-image.service
random-app.build > random-app-build.service
random-app.artifact > random-app-artifact.service
Newer Podman versions allow overriding the name with ServiceName= in the Quadlet file.
Make sure generated service exists #
# rootless
systemctl --user list-unit-files | grep random-app
systemctl --user list-unit-files --type=service | grep random-app
systemctl --user cat random-app.service
systemctl --user show random-app.service \
-p FragmentPath \
-p SourcePath \
-p LoadState \
-p UnitFileState
# rootful
sudo systemctl list-unit-files | grep random-app
sudo systemctl cat random-app.service
sudo systemctl show random-app.service \
-p FragmentPath \
-p SourcePath \
-p LoadState \
-p UnitFileState
The systemctl cat is useful to check the ExecStart= command that will eventually run the container. SourcePath points to your Quadlet file, FragmentPath to the generated service, and UnitFileState=generated is expected.
Side note: those service files are not permanent! They are (re)created in /run/user/$UID/systemd/generator/ (rootful: /run/systemd/generator/) on every daemon-reload and on boot - editing them directly is pointless.
If service is missing, check:
- Quadlet files in wrong directory
- syntax error in file
- keys or file types from a newer Podman version than the installed one - the generator skips the whole file. Happens quickly when copying from the
latestdocs to Ubuntu 24.04 (Podman 4.9.3) - wrong service name (happens way too often)
- generator missing
- run
daemon-reloadagain
Side note: not sure where to put it: services can't be enabled via systemctl - the generator applies the [Install] section of the Quadlet file itself. Trying it anyway fails with an error ending in ... is transient or generated.
Check the dependency lists:
# rootless
systemctl --user list-dependencies default.target
# rootful
sudo systemctl list-dependencies multi-user.target
Enable lingering #
Makes sure that services and containers can run without active session as non-root user - required to start service/container after reboot.
# Check if enabled
loginctl show-user "$USER" -p Linger -p State
# Output:
# State=active
# Linger=yes
# Enable it
sudo loginctl enable-linger "$USER"
Rootless: user session and user namespace #
systemctl --user needs the user's systemd instance and D-Bus session. Switching users with su or sudo -u doesn't provide them and results in:
Failed to connect to bus: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined
# Log in as the user directly via SSH, or
# open a proper login session (package `systemd-container` on Ubuntu)
sudo machinectl shell appuser@
# or control the user manager from an admin account
sudo systemctl --user -M appuser@ status random-app.service
Rootless containers need subordinate UID/GID ranges:
# Check if ranges exist for the user
grep "^$USER:" /etc/subuid /etc/subgid
# Add them if missing, then let Podman pick them up
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 "$USER"
podman system migrate
Side note: Quadlet doesn't support User=, Group= or DynamicUser= in the [Service] section to run a container rootless - put the Quadlet file into the user's search path instead.
Start and status of service #
# rootless
systemctl --user start random-app.service
systemctl --user status random-app.service --no-pager -l
# rootful
sudo systemctl start random-app.service
sudo systemctl status random-app.service --no-pager -l
Check systemd logs #
# Check logs of the Quadlet generator
# rootless
journalctl --user -b | grep -i quadlet
# rootful
sudo journalctl -b | grep -i quadlet
# Check systemd service specific logs
journalctl --user -xeu random-app.service
journalctl --user -u random-app.service -b --no-pager
journalctl --user -u random-app.service -b -n 100 -o short-precise
# Live logs
journalctl --user -fu random-app.service
# Systemd results and exit status
systemctl --user show random-app.service \
-p LoadState \
-p ActiveState \
-p SubState \
-p Result \
-p ExecMainCode \
-p ExecMainStatus \
-p NRestarts
# Output of a healthy service:
Result=success # result of last/current run
NRestarts=0 # number of automatic restarts by systemd
ExecMainCode=0 # how the main process ended: 0 = still running, 1 = exited, 2 = killed by signal, 3 = core dumped
ExecMainStatus=0 # exit code - or signal number if killed
LoadState=loaded # systemd found and loaded the generated service
ActiveState=active # broad status
SubState=running # more detailed service state
Common Result= values of a failed service:
exit-code- process exited with a non-zero code, checkExecMainStatusand the logssignal- process was killed by a signaltimeout- startup took longer thanTimeoutStartSec(default 90s), often slow image pullsoom-kill- killed by the OOM killerstart-limit-hit- (re)started too often in a short time, see below
Debugging toolbox #
Run the container command manually
systemctl --user cat random-app.service shows the full ExecStart= line. Running it in the shell prints errors directly instead of hiding them in the journal:
# stop the service first, otherwise --replace removes the running container
systemctl --user stop random-app.service
# copy the ExecStart= line, replace %N with the service name and remove -d, --cgroups=split and --sdnotify=conmon
/usr/bin/podman run --name systemd-random-app --replace --rm --network systemd-random-app -v systemd-random-app:/data --publish 8080:80 docker.io/library/nginx:latest
Drop-in files
Quadlet files support drop-ins like systemd units, e.g. random-app.container.d/10-override.conf next to the Quadlet file - or container.d/ for all .container files. Worth a look if a setting doesn't seem to apply. The generated service contains the merged result.
ls -la ~/.config/containers/systemd/*.d/
podman quadlet (Podman 5.6+, not on Ubuntu 24.04)
podman quadlet list
podman quadlet print random-app.container
Start limit hit
After too many failed (re)starts, systemd refuses to start the service ("Start request repeated too quickly"). Fix the problem, then reset the counter:
systemctl --user reset-failed random-app.service
systemctl --user start random-app.service
Check Podman container status and logs #
Quadlet names the container systemd-<name> by default - unless ContainerName= is set in the Quadlet file. Same goes for volumes, networks and pods.
podman ps -a --filter name=random-app
podman inspect systemd-random-app
podman logs systemd-random-app
podman events --since 10m
podman events --since 10m --filter container=systemd-random-app
No output from podman logs? Check the log driver - the logs might only end up in the journal:
podman inspect --format '{{.HostConfig.LogConfig.Type}}' systemd-random-app
Some startup failures:
- can't pull/access image
- big/slow images; extend systemd startup timeout (default is 90s)
- can be changed with:
[Service] TimeoutStartSec=900in container file - source - or pre-pull the image with
podman pullor an.imageQuadlet - problems with bind-mount permissions - rootless containers run in a user namespace, so UIDs inside the container don't match the ones on the host
UserNS=keep-idmaps your user into the containerpodman unshare chown 1000:1000 /path/to/dirchanges the ownership from inside the user namespace - use the UID/GID the app runs as inside the container- on SELinux hosts (Fedora/RHEL): add
:Z(private) or:z(shared) toVolume= - network port already used or privileged port (rootless normally can't bind ports below
1024) - allow it with
sudo sysctl net.ipv4.ip_unprivileged_port_start=80and persist it in/etc/sysctl.d/ - a referenced Quadlet is missing or failed -
Network=random-app.networkorVolume=random-app.volume:/dataadd a dependency onrandom-app-network.serviceorrandom-app-volume.service
Container not starting after reboot #
- make sure
lingeringis enabled for the user - make sure the correct
[Install]config is set - rootless:
WantedBy=default.target# starts with all other services - rootful:
WantedBy=multi-user.target - or both
- network not ready at boot, so the image pull fails
- newer Podman versions add a dependency on
network-online.target(rootful) orpodman-user-wait-network-online.service(rootless) automatically - check with
systemctl --user cat random-app.service | grep network-online - if missing: pre-pull the image once with
podman pull- it stays in local storage, and Podman only pulls missing images by default