Podman Quadlet is a feature to generate systemd services for rootless or rootful containers in Podman. This is not a primer for it, but a troubleshooting reference.

We've been using it more and more, so this will be our reference from now on. (I'll make it easier to follow when I have more time).

Most commands have been used and tested on Ubuntu Server 24.04 LTS. I’ll focus on rootless, but add the rootful version as well.


Your Variables #

Use the following form to insert your names to make it easier to copy and paste:


I try to follow the Podman Quadlet process - feel free to start at the top or jump in where the problems begin

random-app.container
    (Quadlet file in one of the search paths)
>>>
podman-user-generator / podman-system-generator
    (runs on boot and on every daemon-reload)
>>>
random-app.service
    (generated systemd unit)
>>>
ExecStart=/usr/bin/podman run ...
    (started by systemd)
>>>
conmon + OCI runtime (crun/runc)
>>>
container process

Paths, extensions and permissions #

Make sure the Quadlet files are saved in the correct directories:

[...] meaning duplicate named quadlets found under /run take precedence over ones in /etc, as well as those in /usr

# rootless

$XDG_RUNTIME_DIR/containers/systemd/

$XDG_CONFIG_HOME/containers/systemd/ or ~/.config/containers/systemd/

/etc/containers/systemd/users/${UID}

/etc/containers/systemd/users/

/usr/share/containers/systemd/users/${UID}

/usr/share/containers/systemd/users/

---

# rootful

/run/containers/systemd/

/etc/containers/systemd/

/usr/share/containers/systemd/

Source

Side note: Quadlet files in /etc/containers/systemd/users/ are picked up by every user - use /etc/containers/systemd/users/${UID}/ to limit it to a single user. The /usr/share/containers/systemd/users/ paths are only supported by recent Podman versions and are not available on Ubuntu 24.04 (Podman 4.9.3). man podman-systemd.unit shows the paths of your installed version.

Accepted file extensions:

.container
.volume
.network
.kube
.image
.pod         # Podman 5.x - not on Ubuntu 24.04
.build       # Podman 5.x - not on Ubuntu 24.04
.artifact    # recent Podman 5.x - not on Ubuntu 24.04

Source

Permissions of Quadlet files

There is no required permission mode, but there are some things to consider:

  • The relevant systemd manager must be able to read the file.
  • Every parent directory must be traversable.
  • Untrusted users must not be able to modify the file.

600 for files, 700 for dirs should do the trick.


Basic Troubleshooting #

Reload the systemd config with systemctl --user daemon-reload and restart the service with systemctl --user restart random-app.service after every change!


Podman Quadlet package installed and ready #

Quadlet was added in Podman 4.4 - Ubuntu 22.04 LTS ships Podman 3.4.4, so there is no Quadlet at all. Ask me how I know...

# Check Podman version - Quadlet requires 4.4+
podman --version

# Check package
dpkg -L podman | grep -E 'podman-(system|user)-generator'

# Make sure the generators are there - both are symlinks to /usr/libexec/podman/quadlet
# rootful
ls -l /usr/lib/systemd/system-generators/podman-system-generator
# rootless
ls -l /usr/lib/systemd/user-generators/podman-user-generator

# Check cgroup version - we need `v2`
podman info --format '{{.Host.CgroupsVersion}}'

Dry-run Quadlet generator #

# Run it read-only to test function
# rootless
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun

# limit dry-run to a single directory
QUADLET_UNIT_DIRS="$HOME/.config/containers/systemd" \
    /usr/lib/systemd/system-generators/podman-system-generator \
    --user --dryrun

# rootful
sudo /usr/lib/systemd/system-generators/podman-system-generator --dryrun

# only show errors, plus additional checks of the generated unit
# rootless
systemd-analyze --user --generators=true verify random-app.service

# rootful
sudo systemd-analyze --generators=true verify random-app.service

This will output all services and configs generated - and the errors for every file that has been skipped:

quadlet-generator[1305164]: converting "random-app.container": unsupported key 'FooBar' in group 'Container' in /home/user/.config/containers/systemd/random-app.container
quadlet-generator[1305164]: processing encountered some errors

Trigger unit generation #

Translates the Quadlet files to .service files

# rootless
systemctl --user daemon-reload

# rootful
sudo systemctl daemon-reload

The service name depends on the type of quadlet file:

random-app.container > random-app.service
random-app.kube      > random-app.service
random-app.volume    > random-app-volume.service
random-app.network   > random-app-network.service
random-app.pod       > random-app-pod.service
random-app.image     > random-app-image.service
random-app.build     > random-app-build.service
random-app.artifact  > random-app-artifact.service

Newer Podman versions allow overriding the name with ServiceName= in the Quadlet file.

Source


Make sure generated service exists #

# rootless
systemctl --user list-unit-files | grep random-app
systemctl --user list-unit-files --type=service | grep random-app
systemctl --user cat random-app.service
systemctl --user show random-app.service \
    -p FragmentPath \
    -p SourcePath \
    -p LoadState \
    -p UnitFileState

# rootful
sudo systemctl list-unit-files | grep random-app
sudo systemctl cat random-app.service
sudo systemctl show random-app.service \
    -p FragmentPath \
    -p SourcePath \
    -p LoadState \
    -p UnitFileState

The systemctl cat is useful to check the ExecStart= command that will eventually run the container. SourcePath points to your Quadlet file, FragmentPath to the generated service, and UnitFileState=generated is expected.

Side note: those service files are not permanent! They are (re)created in /run/user/$UID/systemd/generator/ (rootful: /run/systemd/generator/) on every daemon-reload and on boot - editing them directly is pointless.

If service is missing, check:

  • Quadlet files in wrong directory
  • syntax error in file
  • keys or file types from a newer Podman version than the installed one - the generator skips the whole file. Happens quickly when copying from the latest docs to Ubuntu 24.04 (Podman 4.9.3)
  • wrong service name (happens way too often)
  • generator missing
  • run daemon-reload again

Side note: not sure where to put it: services can't be enabled via systemctl - the generator applies the [Install] section of the Quadlet file itself. Trying it anyway fails with an error ending in ... is transient or generated.

Check the dependency lists:

# rootless
systemctl --user list-dependencies default.target

# rootful
sudo systemctl list-dependencies multi-user.target

Enable lingering #

Makes sure that services and containers can run without active session as non-root user - required to start service/container after reboot.

# Check if enabled
loginctl show-user "$USER" -p Linger -p State
# Output:
# State=active
# Linger=yes

# Enable it
sudo loginctl enable-linger "$USER"

Rootless: user session and user namespace #

systemctl --user needs the user's systemd instance and D-Bus session. Switching users with su or sudo -u doesn't provide them and results in:

Failed to connect to bus: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined
# Log in as the user directly via SSH, or
# open a proper login session (package `systemd-container` on Ubuntu)
sudo machinectl shell appuser@

# or control the user manager from an admin account
sudo systemctl --user -M appuser@ status random-app.service

Rootless containers need subordinate UID/GID ranges:

# Check if ranges exist for the user
grep "^$USER:" /etc/subuid /etc/subgid

# Add them if missing, then let Podman pick them up
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 "$USER"
podman system migrate

Side note: Quadlet doesn't support User=, Group= or DynamicUser= in the [Service] section to run a container rootless - put the Quadlet file into the user's search path instead.


Start and status of service #

# rootless
systemctl --user start random-app.service
systemctl --user status random-app.service --no-pager -l

# rootful
sudo systemctl start random-app.service
sudo systemctl status random-app.service --no-pager -l

Check systemd logs #

# Check logs of the Quadlet generator
# rootless
journalctl --user -b | grep -i quadlet

# rootful
sudo journalctl -b | grep -i quadlet

# Check systemd service specific logs
journalctl --user -xeu random-app.service

journalctl --user -u random-app.service -b --no-pager
journalctl --user -u random-app.service -b -n 100 -o short-precise

# Live logs
journalctl --user -fu random-app.service

# Systemd results and exit status
systemctl --user show random-app.service \
    -p LoadState \
    -p ActiveState \
    -p SubState \
    -p Result \
    -p ExecMainCode \
    -p ExecMainStatus \
    -p NRestarts

# Output of a healthy service:
Result=success          # result of last/current run
NRestarts=0             # number of automatic restarts by systemd
ExecMainCode=0          # how the main process ended: 0 = still running, 1 = exited, 2 = killed by signal, 3 = core dumped
ExecMainStatus=0        # exit code - or signal number if killed
LoadState=loaded        # systemd found and loaded the generated service
ActiveState=active      # broad status
SubState=running        # more detailed service state

Common Result= values of a failed service:

  • exit-code - process exited with a non-zero code, check ExecMainStatus and the logs
  • signal - process was killed by a signal
  • timeout - startup took longer than TimeoutStartSec (default 90s), often slow image pulls
  • oom-kill - killed by the OOM killer
  • start-limit-hit - (re)started too often in a short time, see below

Debugging toolbox #

Run the container command manually

systemctl --user cat random-app.service shows the full ExecStart= line. Running it in the shell prints errors directly instead of hiding them in the journal:

# stop the service first, otherwise --replace removes the running container
systemctl --user stop random-app.service

# copy the ExecStart= line, replace %N with the service name and remove -d, --cgroups=split and --sdnotify=conmon
/usr/bin/podman run --name systemd-random-app --replace --rm --network systemd-random-app -v systemd-random-app:/data --publish 8080:80 docker.io/library/nginx:latest

Drop-in files

Quadlet files support drop-ins like systemd units, e.g. random-app.container.d/10-override.conf next to the Quadlet file - or container.d/ for all .container files. Worth a look if a setting doesn't seem to apply. The generated service contains the merged result.

ls -la ~/.config/containers/systemd/*.d/

podman quadlet (Podman 5.6+, not on Ubuntu 24.04)

podman quadlet list
podman quadlet print random-app.container

Start limit hit

After too many failed (re)starts, systemd refuses to start the service ("Start request repeated too quickly"). Fix the problem, then reset the counter:

systemctl --user reset-failed random-app.service
systemctl --user start random-app.service

Check Podman container status and logs #

Quadlet names the container systemd-<name> by default - unless ContainerName= is set in the Quadlet file. Same goes for volumes, networks and pods.

podman ps -a --filter name=random-app
podman inspect systemd-random-app
podman logs systemd-random-app

podman events --since 10m
podman events --since 10m --filter container=systemd-random-app

No output from podman logs? Check the log driver - the logs might only end up in the journal:

podman inspect --format '{{.HostConfig.LogConfig.Type}}' systemd-random-app

Some startup failures:

  • can't pull/access image
  • big/slow images; extend systemd startup timeout (default is 90s)
  • can be changed with: [Service] TimeoutStartSec=900 in container file - source
  • or pre-pull the image with podman pull or an .image Quadlet
  • problems with bind-mount permissions - rootless containers run in a user namespace, so UIDs inside the container don't match the ones on the host
  • UserNS=keep-id maps your user into the container
  • podman unshare chown 1000:1000 /path/to/dir changes the ownership from inside the user namespace - use the UID/GID the app runs as inside the container
  • on SELinux hosts (Fedora/RHEL): add :Z (private) or :z (shared) to Volume=
  • network port already used or privileged port (rootless normally can't bind ports below 1024)
  • allow it with sudo sysctl net.ipv4.ip_unprivileged_port_start=80 and persist it in /etc/sysctl.d/
  • a referenced Quadlet is missing or failed - Network=random-app.network or Volume=random-app.volume:/data add a dependency on random-app-network.service or random-app-volume.service

Container not starting after reboot #

  • make sure lingering is enabled for the user
  • make sure the correct [Install] config is set
  • rootless: WantedBy=default.target # starts with all other services
  • rootful: WantedBy=multi-user.target
  • or both
  • network not ready at boot, so the image pull fails
  • newer Podman versions add a dependency on network-online.target (rootful) or podman-user-wait-network-online.service (rootless) automatically
  • check with systemctl --user cat random-app.service | grep network-online
  • if missing: pre-pull the image once with podman pull - it stays in local storage, and Podman only pulls missing images by default