Mastering Systemd Service Files: A Comprehensive Guide

Systemd service files tell Linux how to start, stop, restart, and supervise your application. If your service starts manually but fails at boot, restarts too aggressively, or writes logs to the wrong place, the unit file is usually where you need to look.

This guide focuses on creating and configuring systemd service unit files from scratch. You will see the core sections, a working Python service example, common troubleshooting commands, and a few security and resource controls worth using in production.

Understanding Systemd Unit Files

Systemd uses unit files to describe various system resources such as services, sockets, devices, mount points, and more. A service unit file, typically ending with the .service extension, defines how systemd should manage a specific daemon or application.

These files are organized into sections, with each section containing key-value pairs representing configuration directives. The primary sections we'll focus on are [Unit], [Service], and [Install].

Anatomy of a Systemd Service File

A typical systemd service file has the following structure:

[Unit]
Description=A brief description of the service.
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my_application --config /etc/my_app.conf
Restart=on-failure
User=myuser
Group=mygroup

[Install]
WantedBy=multi-user.target

Let's break down each section and its common directives:

The [Unit] Section

This section provides metadata about the unit and defines its relationship with other units. It's used for dependencies and ordering.

• Description=: A human-readable name for the service. This is what you'll see in systemctl status output.
• Documentation=: URLs or paths to documentation for the service.
• Requires=: Defines strong dependencies. If a unit listed here fails to start, this unit will also fail to start.
• Wants=: Defines weak dependencies. If a unit listed here fails to start, this unit will still attempt to start.
• Before=: Ensures this unit starts before the units listed.
• After=: Controls ordering only. For example, After=network.target starts this unit after the basic network target, but it does not guarantee external connectivity. Network-dependent services may need After=network-online.target plus the distribution's wait-online service.
• Conflicts=: If a unit listed here is started, this unit will be stopped, and vice-versa.

The [Service] Section

This section configures the behavior of the service itself. It's where you define how to start, stop, and manage the process.

• Type=: Specifies the process startup type. Common values include:

  • simple (default): The main process is the one specified in ExecStart=. Systemd assumes the service is started immediately after the ExecStart= process is forked.
  • forking: The ExecStart= process forks a child, and the parent exits. Systemd considers the service started when the parent exits. You often need to specify PIDFile= with this type.
  • oneshot: Similar to simple, but the process is expected to exit after its work is done. Useful for setup scripts.
  • notify: The daemon sends a notification message to systemd when it has successfully started. This is the preferred type for modern daemons that support it.
  • dbus: The service acquires a D-Bus name.

• ExecStart=: The command to execute to start the service. For most service types, use one ExecStart= command. Multiple ExecStart= lines are valid for Type=oneshot, where they run sequentially.

• ExecStop=: The command to execute to stop the service.

• ExecReload=: The command to execute to reload the service's configuration without restarting.

• Restart=: Defines when the service should be automatically restarted. Common values:

  • no (default): Never restart.
  • on-success: Restart only if the service exits cleanly (exit code 0).
  • on-failure: Restart if the service exits with a non-zero exit code, is terminated by a signal, or times out.
  • on-abnormal: Restart if terminated by a signal or times out.
  • on-abort: Restart only if terminated uncleanly by a signal.
  • always: Always restart, regardless of the exit status.

• RestartSec=: The time to sleep before restarting the service (default is 100ms).

• User=: The user to run the service as.

• Group=: The group to run the service as.

• WorkingDirectory=: The directory to change to before executing the commands.

• Environment=: Sets environment variables for the service.

• EnvironmentFile=: Reads environment variables from a file.

• PIDFile=: Path to the PID file (often used with Type=forking).

• StandardOutput= / StandardError=: Controls where stdout and stderr go, such as journal, null, or inherit. On common systemd-based distributions, service output normally lands in the journal unless the manager defaults have been changed.

The [Install] Section

This section defines how the unit should be enabled or disabled, typically by creating symbolic links.

• WantedBy=: Specifies the target that should "want" this service when it's enabled. Common values:
  • multi-user.target: For services that should start when the system reaches a multi-user command-line state.
  • graphical.target: For services that should start when the system reaches a graphical login state.

Creating Your First Systemd Service File

Let's create a simple service file for a hypothetical Python script named my_app.py located at /opt/my_app/my_app.py.

1. Create the service file:

Service files for custom applications are typically placed in /etc/systemd/system/. Let's name our file my_app.service.

# Create the directory if it doesn't exist
sudo mkdir -p /etc/systemd/system/

# Create the service file using a text editor
sudo nano /etc/systemd/system/my_app.service

2. Add the following content to my_app.service:

[Unit]
Description=My Custom Python Application
After=network.target

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/my_app/
ExecStart=/usr/bin/python3 /opt/my_app/my_app.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

Explanation of the example:

• Description: Clearly identifies our application.
• After=network.target: Ensures the network is available before starting.
• Type=simple: Assumes my_app.py is the main process and doesn't fork.
• User=appuser, Group=appgroup: Specifies the user and group the application should run as. Make sure these users and groups exist on your system and have appropriate permissions. You might need to create them:

  sudo groupadd appgroup
  sudo useradd -r -g appgroup appuser
  sudo chown -R appuser:appgroup /opt/my_app/
  

• WorkingDirectory: Sets the context for the script.
• ExecStart: The command to run the Python script. Ensure /usr/bin/python3 is the correct path to your Python interpreter and that the script is executable.
• Restart=on-failure: If the script crashes, systemd will try to restart it.
• WantedBy=multi-user.target: This service will be started automatically when the system boots into a multi-user environment.

3. Reload the systemd manager configuration:

After creating or modifying a service file, you must tell systemd to reload its configuration.

sudo systemctl daemon-reload

4. Enable and Start the Service:

• Enable: This makes the service start automatically on boot.

  sudo systemctl enable my_app.service
  

• Start: This starts the service immediately.

  sudo systemctl start my_app.service
  

5. Check the Service Status:

To verify if your service is running and to see any potential errors:

sudo systemctl status my_app.service

If there are issues, the status command will often show error messages or logs from journald.

6. Viewing Logs:

Systemd integrates with journald for logging. You can view logs for your service using:

sudo journalctl -u my_app.service

You can also follow logs in real-time:

sudo journalctl -f -u my_app.service

Other Useful Commands

• Stop the service: sudo systemctl stop my_app.service
• Restart the service: sudo systemctl restart my_app.service
• Reload configuration (if supported by the app): sudo systemctl reload my_app.service
• Disable auto-start on boot: sudo systemctl disable my_app.service

Advanced Configuration and Best Practices

Security Considerations

• Run services as non-root users: Always specify User= and Group= unless absolutely necessary. This follows the principle of least privilege.
• Isolate services: Consider sandboxing features like PrivateTmp=true, ProtectSystem=strict, ProtectHome=true, and NoNewPrivileges=true. Test them with your application because they can block legitimate file writes.
  • PrivateTmp=true: Gives the service its own private /tmp and /var/tmp directories.
  • ProtectSystem=strict: Makes most of the filesystem read-only for the service. Use ReadWritePaths= for directories the service must write to.
  • NoNewPrivileges=true: Prevents the service from gaining new privileges.

Handling Complex Startups

• Type=forking with PIDFile=: For older applications that fork, ensure PIDFile= points to the correct file.
• Type=notify: If your application supports it, this is the most robust way for systemd to know when it's truly ready.
• ExecStartPre= and ExecStartPost=: Commands to run before and after ExecStart=. Useful for setup or cleanup tasks.

Resource Control

Systemd allows you to limit resource usage:

• CPUWeight=: Relative CPU weight for the service.
• MemoryMax=: Maximum memory the service can use.
• IOWeight=: Relative I/O weight where supported by the kernel and cgroup setup.

Example:

[Service]
# ... other directives ...
MemoryMax=512M
CPUWeight=50

Timers vs. Cron

Systemd timers offer a modern alternative to traditional cron jobs. They are more flexible and integrate better with systemd's logging and dependency management.

• Cron: Scheduled tasks defined in crontab files.
• Systemd Timers (.timer units): These units schedule .service units. You define a .timer file that specifies when a corresponding .service file should run.

Example:

To run a script daily at 3 AM:

1. my_script.service: The service to run.

   [Unit]
   Description=My daily script
   
   [Service]
   Type=oneshot
   ExecStart=/opt/my_scripts/run_daily.sh
   User=scriptuser
   

2. my_script.timer: The timer that schedules the service.

   [Unit]
   Description=Run my daily script once a day
   
   [Timer]
   # Run at 03:00 every day
   OnCalendar=*-*-* 03:00:00
   # Run soon after boot if the scheduled time was missed while the machine was off.
   Persistent=true
   
   [Install]
   WantedBy=timers.target
   

To use this:

• Place both files in /etc/systemd/system/.
• Run sudo systemctl daemon-reload.
• Enable and start the timer: sudo systemctl enable my_script.timer and sudo systemctl start my_script.timer.

Timers offer advantages like Persistent=true (runs missed jobs upon boot), calendar events (like hourly, daily, weekly), and better integration with journalctl.

Troubleshooting Common Issues

• Service not starting: Check systemctl status <service_name> and journalctl -u <service_name>. Look for typos, incorrect paths, missing dependencies, or permission errors.
• Incorrect Type=: If a service fails immediately or hangs, the Type= might be wrong. Try simple or forking and ensure PIDFile is correct if using forking.
• Permission denied: Ensure the User= and Group= specified have read/write access to necessary files and directories.
• Environment variables: If your application relies on specific environment variables, ensure they are set correctly using Environment= or EnvironmentFile=.
• Dependencies: Verify that After=, Wants=, and Requires= match what you mean. After= orders startup; it does not pull another unit in by itself.

Before enabling a new unit on a production host, run:

sudo systemd-analyze verify /etc/systemd/system/my_app.service

This catches many syntax and directive mistakes before you rely on the service at boot.

Key Takeaway

Write the smallest service file that accurately describes your app, then add restart policy, logging, security restrictions, and resource limits deliberately. After every change, run systemctl daemon-reload, verify the unit, and check systemctl status plus journalctl -u before you trust it in production.