Understanding Systemd Units: A Deep Dive into Service Configuration

Systemd unit files are the small text files that decide how services start, what they depend on, which user they run as, and what happens when they fail. If you have ever wondered why systemctl restart myapp.service works for one app but not another, the answer is usually in the unit file.

This guide focuses on .service units because they are the ones administrators and developers edit most often. The same system also manages sockets, timers, mounts, devices, paths, and targets, but service files are where most operational mistakes show up.

What are Systemd Unit Files?

Systemd unit files are simple text files that contain configuration directives for a specific unit. A unit represents a resource managed by systemd. The most common type is the service unit, which defines how to start, stop, restart, and manage a background process or application.

Unit files are organized into sections, each denoted by square brackets ([]). The most important sections for service units are:

[Unit]: Contains metadata about the unit, dependencies, and ordering.
[Service]: Defines the behavior of the service itself, including how to execute it.
[Install]: Specifies how the unit should be enabled or disabled, typically linking it to target units.

Systemd looks for unit files in several standard directories, with the most common being:

/etc/systemd/system/: For locally configured units, overriding default ones.
/usr/lib/systemd/system/: For units installed by packages on many distributions.
/lib/systemd/system/: Used by some Debian-family systems for package-provided units.

When you need to inspect a unit, avoid guessing the path. Use:

systemctl cat nginx.service
systemctl show -p FragmentPath nginx.service

systemctl cat is especially useful because it shows the base unit plus any drop-in overrides. That is the version systemd is actually using.

Anatomy of a `.service` Unit File

Let's break down a typical .service unit file to understand its components.

The `[Unit]` Section

This section provides descriptive information and defines the relationships between units.

Description=: A human-readable description of the service.
Documentation=: URLs or paths to documentation for the service.
After=: Specifies that this unit should start after the listed units have finished starting.
Requires=: Similar to After=, but also makes the listed units mandatory. If a required unit fails to start, this unit will also fail.
Wants=: A weaker form of dependency. This unit will attempt to start its wanted units, but their failure won't prevent this unit from starting.
Conflicts=: Specifies units that cannot run concurrently with this unit.

Example [Unit] section:

[Unit]
Description=My Custom Web Server
Documentation=https://example.com/docs/my-web-server
After=network.target

This indicates that our custom web server should start after the network is available.

One common trap: After= controls order, not requirement. If you write After=postgresql.service, systemd starts your service after PostgreSQL when both are part of the transaction, but it does not automatically pull PostgreSQL in. If your app truly needs PostgreSQL to be started by the same transaction, use Wants=postgresql.service or, for a hard dependency, Requires=postgresql.service as well.

Even then, dependencies are not health checks. After=network.target does not guarantee DNS works, a remote API is reachable, or a database is accepting connections. Your application still needs sensible retry behavior.

The `[Service]` Section

This is where the core logic for running the service resides.

Type=: Defines the process startup type. Common types include:
- simple (default): The main process is the one started by ExecStart=. Systemd considers the service started immediately after the ExecStart= process is forked.
- forking: Used for traditional daemons that fork off a child process and exit. Systemd waits for the parent process to exit.
- oneshot: For tasks that execute a single command and then exit.
- notify: The service sends a notification to systemd when it has finished starting up.
- dbus: For services that acquire a D-Bus name.
ExecStart=: The command to execute to start the service.
ExecStop=: The command to execute to stop the service.
ExecReload=: The command to execute to reload the service's configuration without restarting it.
Restart=: Defines when the service should be restarted. Options include no (default), on-success, on-failure, on-abnormal, on-watchdog, on-abort, and always.
RestartSec=: The time to wait before restarting the service.
User= / Group=: The user and group under which the service should run.
WorkingDirectory=: The working directory for the executed processes.
Environment= / EnvironmentFile=: Sets environment variables for the service.

Example [Service] section:

[Service]
Type=simple
ExecStart=/usr/local/bin/my-web-server --config /etc/my-web-server.conf
User=www-data
Group=www-data
Restart=on-failure
RestartSec=5

This configuration starts our web server, runs it as the www-data user and group, and automatically restarts it if it fails, with a 5-second delay.

Type= deserves extra care. Many broken units use Type=forking because an old init script used daemon mode. For a modern application that stays in the foreground, Type=simple is usually correct. If your process forks into the background but systemd is not told how to identify the real main process, status reporting and restarts can become misleading.

For a one-time job, use Type=oneshot and often RemainAfterExit=yes if the completed action should count as active. For example, a unit that prepares a firewall rule or mounts a special resource may exit successfully but still represent a state you care about.

The `[Install]` Section

This section is used when enabling or disabling a unit. It defines how the unit integrates with systemd's target units.

WantedBy=: Specifies the target(s) that should "want" this unit when it's enabled. For services that should start at boot, multi-user.target is commonly used.

Example [Install] section:

[Install]
WantedBy=multi-user.target

When you run systemctl enable my-custom-service.service, systemd creates a symbolic link from /etc/systemd/system/multi-user.target.wants/ to your service file, ensuring it starts when the system reaches the multi-user runlevel.

If a unit has no [Install] section, it may still be perfectly valid. It just cannot be enabled directly with systemctl enable unless another install mechanism exists. Some units are meant to be pulled in by dependencies, sockets, timers, or targets instead of being enabled by hand.

Creating and Managing Custom Service Units

Let's walk through the process of creating a custom service unit.

Step 1: Create the Unit File

Create a new file in /etc/systemd/system/ with a .service extension. For our example, let's create /etc/systemd/system/my-app.service.

[Unit]
Description=My Custom Application Service
After=network.target

[Service]
Type=simple
ExecStart=/opt/my-app/bin/run-app --port 8080
User=appuser
Group=appgroup
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Important Considerations:

Ensure the ExecStart command points to an executable script or binary that is accessible and has execute permissions.
Create the specified User and Group if they don't exist (sudo useradd -r -s /bin/false appuser, sudo groupadd appgroup, sudo usermod -a -G appgroup appuser).
Make sure the application can be started and stopped correctly using the specified commands.

Before you put a command in ExecStart=, run it manually as the same user if possible:

sudo -u appuser /opt/my-app/bin/run-app --port 8080

This catches missing execute bits, missing directories, bad relative paths, and permission problems before systemd is involved. Once it works manually, move it into the unit and let systemd handle supervision.

Step 2: Reload Systemd Configuration

After creating or modifying a unit file, you must tell systemd to reload its configuration.

sudo systemctl daemon-reload

This command scans for new or changed unit files and updates systemd's internal state.

Step 3: Enable and Start the Service

To start the service immediately and configure it to start on boot:

sudo systemctl enable my-app.service  # Creates symlinks for boot startup
sudo systemctl start my-app.service   # Starts the service now

Step 4: Manage the Service

Use systemctl commands to manage your service:

Check status:
```
sudo systemctl status my-app.service
```
This will show if the service is active, its process ID, recent log entries, and more.
Stop the service:
```
sudo systemctl stop my-app.service
```
Restart the service:
```
sudo systemctl restart my-app.service
```
Reload the service (if ExecReload= is defined):
```
sudo systemctl reload my-app.service
```
Disable the service (prevent from starting on boot):
```
sudo systemctl disable my-app.service
```

Step 5: View Logs with `journalctl`

Systemd integrates tightly with journald for logging. You can view logs for your service using journalctl:

View logs for a specific service:
```
sudo journalctl -u my-app.service
```
Follow logs in real-time:
```
sudo journalctl -f -u my-app.service
```
View logs since the last boot:
```
sudo journalctl -b -u my-app.service
```

Best Practices and Tips

Use Type=notify for modern applications: If your application supports it, Type=notify provides better integration with systemd, allowing it to accurately track the service's readiness.
Run services as non-root users: Always specify User= and Group= in the [Service] section to minimize security risks.
Define dependencies carefully: Use After=, Requires=, and Wants= to ensure services start in the correct order and that critical dependencies are met.
Leverage Restart=: Configure appropriate restart policies to ensure service availability.
Keep unit files simple: For complex startup sequences, consider using wrapper scripts invoked by ExecStart= rather than complex commands directly in the unit file.
Use systemctl cat <unit>: To view the full content of a unit file as systemd sees it, including any overrides.
Use systemctl edit <unit>: This command opens an editor to create an override file for an existing unit, which is a cleaner way to modify default unit files than editing them directly.

Editing Existing Units Safely

Do not edit package-owned units in /usr/lib/systemd/system/ or /lib/systemd/system/ unless you are debugging a disposable machine. Package upgrades can replace those files. Use an override instead:

sudo systemctl edit nginx.service

That creates a drop-in under /etc/systemd/system/nginx.service.d/. For example, to add a restart policy:

[Service]
Restart=on-failure
RestartSec=5s

Some directives can be specified more than once. Others need to be cleared before replacement. ExecStart= is the classic example:

[Service]
ExecStart=
ExecStart=/usr/local/bin/my-nginx-wrapper

The blank ExecStart= line resets the previous value. Without it, systemd may reject the unit or keep more commands than you intended.

After any unit or drop-in change, use the same review loop:

sudo systemctl daemon-reload
systemctl cat my-app.service
sudo systemctl restart my-app.service
journalctl -u my-app.service -n 50 --no-pager

Unit files are not hard once you separate the three jobs: [Unit] describes relationships, [Service] describes process behavior, and [Install] describes enablement. Most real-world debugging is just finding which of those jobs was configured with the wrong assumption.

A Realistic Service File Walkthrough

Here is a small but realistic service for a Python web application:

[Unit]
Description=Inventory API
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=inventory
Group=inventory
WorkingDirectory=/srv/inventory-api
EnvironmentFile=/etc/inventory-api/env
ExecStart=/srv/inventory-api/.venv/bin/gunicorn app:app --bind 127.0.0.1:9000
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

There are several quiet decisions in that file. The service runs as inventory, not root. The command uses an absolute path to the virtual environment's gunicorn, so it does not depend on an interactive shell's PATH. The app binds to localhost because a reverse proxy will expose it publicly. The environment file lives outside the unit so deployment can update configuration without rewriting package-owned service metadata.

The dependency lines are intentionally modest. After=postgresql.service controls order if PostgreSQL is part of the same startup transaction. It does not prove the database is ready for connections, and it does not replace application retry logic. network-online.target can help on systems that correctly implement network readiness, but it is not a universal guarantee that every remote dependency is reachable.

If this service fails, the first checks are predictable:

systemctl status inventory-api.service
journalctl -u inventory-api.service -b --no-pager
systemctl cat inventory-api.service
sudo -u inventory /srv/inventory-api/.venv/bin/gunicorn app:app --bind 127.0.0.1:9000

The last command is not something you leave running in production. It is a diagnostic check that asks, "Can the configured user run this command at all?" If it cannot import the app, read the environment file, or write to its log directory, systemd will not fix that for you.

Resource and Security Directives You Will See Often

Many production units include hardening or resource controls. A few common examples:

[Service]
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
MemoryMax=512M
CPUQuota=80%

These directives can be very useful, but they can also break assumptions. PrivateTmp=true gives the service a private /tmp, so another process may not see files it writes there. ProtectHome=true can block access to /home, /root, and /run/user. ProtectSystem=full makes much of the system read-only from the service's point of view. If an app suddenly cannot write where it used to, inspect the hardening settings before blaming the application.

Resource limits have the same tradeoff. MemoryMax= can stop one service from consuming the whole machine, but if the value is too low, the service may be killed under normal load. Check the journal for out-of-memory messages and compare the limit with real usage before raising or removing it.

The Most Useful Debugging Commands

Keep these close when working with service units:

systemctl status my-app.service
systemctl cat my-app.service
systemctl show my-app.service
systemd-analyze verify /etc/systemd/system/my-app.service
journalctl -u my-app.service -b --no-pager

systemctl show is verbose, but it exposes the properties systemd calculated after parsing the unit. That can reveal a surprising value inherited from a default, a drop-in, or a reset directive. systemd-analyze verify catches some syntax and dependency errors before you restart a service. It is not a replacement for testing the application, but it catches enough mistakes to be worth running.