How to Write and Manage Custom Systemd Unit Files Effectively

Custom systemd unit files are what turn a command that works in your terminal into a service the operating system can start, stop, restart, log, and supervise. The difference matters. A command in a shell inherits your environment and dies when the session ends. A service has an explicit user, working directory, restart policy, dependencies, resource limits, and logs.

This is the practical path I use for small internal APIs, workers, sidecar scripts, and one-off daemons: write the simplest correct unit, run it as a dedicated user, let logs go to the journal, and only add advanced directives when a real requirement appears.

Understanding Systemd Unit Files

Systemd manages various system resources, known as units, which are defined by configuration files. These units include services (.service), mount points (.mount), devices (.device), sockets (.socket), and more. For managing applications and background processes, the .service unit type is the most common and relevant.

Systemd unit files are plain text files typically stored in specific directories. The primary locations, in order of precedence, are:

/etc/systemd/system/: This is the recommended location for custom unit files and overrides, as they take precedence over system defaults and persist across system updates.
/run/systemd/system/: Used for runtime-generated unit files.
/usr/lib/systemd/system/: Contains unit files provided by installed packages. Do not modify files in this directory directly.

By placing your custom unit files in /etc/systemd/system/, you ensure they are properly recognized and managed by systemd.

Anatomy of a `.service` Unit File

A systemd .service unit file is structured into several sections, each denoted by [SectionName], containing various directives (key-value pairs). The three main sections for a service unit are [Unit], [Service], and [Install].

Let's break down the most crucial directives you'll use:

[Unit] Section

This section contains generic options about the unit, its description, and dependencies.

Description: A human-readable string describing the service. This appears in systemctl status output.
```
Description=My Custom Python Web Application
```
Documentation: A URL pointing to documentation for the service (optional).
```
Documentation=https://example.com/docs/my-app
```
After: Specifies that this unit should start after the listed units. This helps manage startup order. For web applications, you might want to ensure networking is up.
```
After=network.target
```
Requires: A strong dependency. If the required unit fails to start, this unit will not start. If the required unit is stopped, this unit may also be stopped.
```
Requires=docker.service
```
Wants: A weaker dependency. If the wanted unit fails or is not found, this unit still attempts to start. This is usually a better default than Requires.
```
Wants=syslog.target
```

[Service] Section

This section defines the execution parameters for your service, including how it starts, stops, and behaves.

Type: Defines the process startup type. Critical for how systemd monitors your service.
- simple (default): The ExecStart command is the main process of the service. Systemd considers the service started immediately after ExecStart is invoked. It expects the process to run indefinitely in the foreground.
- forking: The ExecStart command forks a child process and the parent exits. Systemd considers the service started once the parent process exits. Use this if your application daemonizes itself.
- oneshot: The ExecStart command is a one-time process that exits when it's done. Useful for scripts that perform a task and terminate (e.g., a backup script).
- notify: Similar to simple, but the service tells systemd when it is ready. This requires application support for systemd notifications.
- idle: The ExecStart command is executed only when all jobs are finished, delaying execution until the system is mostly idle.
```
Type=simple
```
ExecStart: The command to execute when the service starts. This is the most important directive in this section. Always use the absolute path to your executable or script.
```
ExecStart=/usr/bin/python3 /opt/my_app/app.py
```
ExecStop: The command to execute when the service is stopped (optional). If not specified, systemd sends SIGTERM to the processes.
```
ExecStop=/usr/bin/pkill -f 'my_app/app.py'
```
ExecReload: The command to execute to reload the service's configuration (optional).
```
ExecReload=/bin/kill -HUP $MAINPID
```
User: The user account under which the service's processes will run. Essential for security; avoid root unless absolutely necessary.
```
User=myappuser
```
Group: The group account under which the service's processes will run.
```
Group=myappgroup
```
WorkingDirectory: The working directory for the executed commands.
```
WorkingDirectory=/opt/my_app
```
Restart: Defines when the service should be automatically restarted.
- no (default): Never restart.
- on-success: Restart only if the service exits cleanly.
- on-failure: Restart only if the service exits with a non-zero status code or is killed by a signal.
- always: Always restart the service, regardless of exit status.
```
Restart=on-failure
```
RestartSec: How long to wait before restarting the service (e.g., 5s for 5 seconds).
```
RestartSec=5s
```
Environment: Sets environment variables for the executed commands.
```
Environment="APP_ENV=production" "DEBUG=false"
```
EnvironmentFile: Reads environment variables from a file. Each line should be KEY=VALUE.
```
EnvironmentFile=/etc/default/my_app
```
LimitNOFILE: Sets the maximum number of open file descriptors allowed for the service (e.g., 100000). Important for high-concurrency applications.
```
LimitNOFILE=65536
```

[Install] Section

This section defines how the service is enabled to start automatically at boot time.

WantedBy: Specifies the target unit that "wants" this service. When the target unit is enabled, this service will be symlinked into its .wants directory, effectively making it start with the target.
- multi-user.target: The standard target for most server services, indicating a system with non-graphical multi-user logins.
- graphical.target: For services that require a graphical environment.
```
WantedBy=multi-user.target
```
RequiredBy: Similar to WantedBy, but a stronger dependency. If the target is enabled, this unit is also enabled, and if this unit fails, the target will also fail.

For most custom services intended to run in the background on a server, Type=simple and WantedBy=multi-user.target are the right starting point. If the application already daemonizes itself, either disable that behavior or use Type=forking with care. A foreground process is easier for systemd to supervise.

Step-by-Step: Creating and Managing a Custom Systemd Service

Let's create a practical example: a simple Python HTTP server that serves files from a specified directory. We'll set it up as a systemd service.

Step 1: Prepare Your Application/Script

First, create the application script. For this example, we'll use a simple Python HTTP server. Create a directory for your application, e.g., /opt/my_app, and place app.py inside it.

# /opt/my_app/app.py

import http.server
import socketserver
import os

PORT = int(os.environ.get("PORT", 8000))
DIRECTORY = os.environ.get("DIRECTORY", os.getcwd())

class Handler(http.server.SimpleHTTPRequestHandler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, directory=DIRECTORY, **kwargs)

print(f"Serving directory {DIRECTORY} on port {PORT}")

with socketserver.TCPServer(("", PORT), Handler) as httpd:
    print("Server started.")
    httpd.serve_forever()

Create the directory and file:

sudo mkdir -p /opt/my_app
sudo nano /opt/my_app/app.py

(Paste the Python code)

Make sure the script is executable (optional for python3 command, but good practice):

sudo chmod +x /opt/my_app/app.py

Consider creating a dedicated user for your service for security reasons:

sudo useradd --system --no-create-home myappuser

Set appropriate ownership for your application directory:

sudo chown -R myappuser:myappuser /opt/my_app

Step 2: Create the Unit File

Now, create the systemd unit file for our Python application. We'll name it my_app.service.

sudo nano /etc/systemd/system/my_app.service

Paste the following content:

# /etc/systemd/system/my_app.service

[Unit]
Description=My Custom Python HTTP Server
Documentation=https://github.com/example/my_app
After=network.target

[Service]
Type=simple
User=myappuser
Group=myappuser
WorkingDirectory=/opt/my_app
Environment="PORT=8080" "DIRECTORY=/var/www/html"
ExecStart=/usr/bin/python3 /opt/my_app/app.py
Restart=on-failure
RestartSec=10s
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Note: We've set StandardOutput=journal and StandardError=journal to direct the service's output to the systemd journal, making it easy to view logs with journalctl.

If your app needs secrets, avoid putting them directly in the unit file. Use an environment file with restrictive permissions, a secret manager, or distribution-specific credential support. Unit files are often readable by more people than you expect.

Step 3: Place the Unit File

As instructed, we've placed the unit file in /etc/systemd/system/. This is where custom unit files should reside.

Step 4: Reload Systemd Daemon

After creating or modifying a unit file, systemd needs to be informed of the changes. This is done by reloading the systemd daemon:

sudo systemctl daemon-reload

Step 5: Start the Service

Now you can start your service:

sudo systemctl start my_app.service

Step 6: Check Service Status and Logs

Verify that your service is running correctly:

systemctl status my_app.service

Example output (truncated):

● my_app.service - My Custom Python HTTP Server
     Loaded: loaded (/etc/systemd/system/my_app.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-10-26 10:30:00 UTC; 5s ago
       Docs: https://github.com/example/my_app
   Main PID: 12345 (python3)
      Tasks: 1 (limit: 1100)
     Memory: 6.5M
        CPU: 45ms
     CGroup: /system.slice/my_app.service
             └─12345 /usr/bin/python3 /opt/my_app/app.py

Oct 26 10:30:00 yourhostname python3[12345]: Serving directory /var/www/html on port 8080
Oct 26 10:30:00 yourhostname python3[12345]: Server started.

To view the service's logs, use journalctl:

journalctl -u my_app.service -f

This command shows logs for my_app.service and -f (follow) will show new logs in real-time.

You can also test the server from your browser or curl on http://localhost:8080 (assuming /var/www/html exists and contains some files).

Step 7: Enable the Service for Autostart

To make your service start automatically every time the system boots, you need to enable it:

sudo systemctl enable my_app.service

This command creates a symbolic link from /etc/systemd/system/multi-user.target.wants/my_app.service to /etc/systemd/system/my_app.service.

Step 8: Stop and Disable the Service

To stop a running service:

sudo systemctl stop my_app.service

To prevent a service from starting automatically on boot (while leaving it enabled to be started manually):

sudo systemctl disable my_app.service

If you want to remove the service entirely, disable it first, then stop it, and finally delete the .service file from /etc/systemd/system/ and run sudo systemctl daemon-reload.

Step 9: Updating a Service

If you modify your app.py script or the my_app.service unit file, you'll need to update systemd and restart the service:

Edit /opt/my_app/app.py or /etc/systemd/system/my_app.service.
If you modified the unit file, run sudo systemctl daemon-reload.
Restart the service: sudo systemctl restart my_app.service.

Safer Patterns for Real Services

A unit that works is not always a unit you want to maintain for years. These patterns prevent common mistakes:

Run in the foreground. Let systemd supervise the main process. Avoid nohup, screen, tmux, background &, or application daemon modes inside ExecStart.
Keep ExecStart direct. If you need shell features such as pipes or variable expansion, call /bin/sh -c '...' intentionally. Otherwise, run the executable directly.
Use a dedicated user. A service that only needs to read /opt/my_app and bind to an unprivileged port should not run as root.
Reload after unit edits. sudo systemctl daemon-reload is required when the unit file changes.
Separate code deploys from unit changes. If only Python code changed, restart the service. If the unit changed, reload systemd first.

Troubleshooting a New Unit

If the service fails, start with:

systemctl status my_app.service
journalctl -u my_app.service -n 100 --no-pager
systemctl cat my_app.service

Common failures are usually plain:

status=203/EXEC often means the executable path is wrong, the file is missing, or the file is not executable.
Permission denied usually means the service user cannot read a file, enter a directory, write logs, or bind the requested port.
address already in use means another process owns the port. Check with sudo ss -tulpen | grep ':8080'.
A service that starts manually but fails under systemd often depends on environment variables, a different working directory, or files in your home directory.

You can test the command as the service user:

sudo -u myappuser /usr/bin/python3 /opt/my_app/app.py

That is not a perfect reproduction of systemd's environment, but it catches obvious application errors before you chase unit-file details.

A More Production-Friendly Variant

For a long-running internal service, I would usually add a few guardrails:

[Unit]
Description=My Custom Python HTTP Server
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=myappuser
Group=myappuser
WorkingDirectory=/opt/my_app
EnvironmentFile=-/etc/my_app/my_app.env
ExecStart=/usr/bin/python3 /opt/my_app/app.py
Restart=on-failure
RestartSec=10s
TimeoutStopSec=30s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/opt/my_app /var/log/my_app
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

ProtectSystem=full makes much of the system read-only to the service, so add ReadWritePaths= only for directories the app genuinely needs to write. Test hardening one directive at a time. Security options are useful, but a service that cannot read its config or write its data will fail at startup.

Best Practices and Troubleshooting

Absolute Paths: Always use absolute paths for ExecStart, WorkingDirectory, and any other file paths within your unit file. Relative paths can lead to unexpected behavior.
Dedicated Users: Run services under non-privileged, dedicated user accounts (e.g., myappuser) to enhance security and limit potential damage in case of compromise.
Clear Logging: Utilize StandardOutput=journal and StandardError=journal to direct service output to the systemd journal. Use journalctl -u <service_name> to view logs.
Dependencies: Carefully consider After, Wants, and Requires to ensure your service starts in the correct order relative to its dependencies (e.g., networking, databases).
Testing Changes: Before enabling a service to start on boot, thoroughly test it by starting and stopping it manually. Check its status and logs.
Resource Limits: Use directives like LimitNOFILE, LimitNPROC, and MemoryMax when the service has known limits or failure modes.
Environment Variables: Use Environment= or EnvironmentFile= for configuration values that might change or vary between environments, rather than hardcoding them in the unit file or script.
Error Handling in Scripts: Ensure your application scripts handle errors gracefully. A non-zero exit code will trigger Restart=on-failure.

Warning: Avoid modifying unit files directly in /usr/lib/systemd/system/. Any changes will likely be overwritten by package updates. Use /etc/systemd/system/ for custom units or overrides.

A good custom unit is boring in the best way: the command is explicit, the user is unprivileged, the restart behavior is intentional, and the logs are easy to find. Once that is solid, systemd timers, socket activation, and deeper cgroup controls are natural next steps.