Troubleshooting Systemd: Understanding Service Dependencies and Ordering Directives

Most confusing systemd dependency bugs come from mixing up two separate ideas: requirement and order. Requires= and Wants= answer "what other unit should be pulled in?" After= and Before= answer "when should this unit start relative to another one?" If you only remember one thing from this guide, remember that After=postgresql.service does not start PostgreSQL for you. It only says that if both units are being started, your unit should wait until PostgreSQL's start job has run.

That distinction is the source of a lot of "works when I start it manually, fails after reboot" incidents. A web app starts before the database socket is accepting connections. A worker starts before /mnt/jobs is mounted. A service waits for network.target and still fails because an IP address has not been assigned yet. These are not exotic systemd problems. They are ordinary startup assumptions that need to be made explicit.

Core Dependency Directives: Requires and Wants

Systemd uses two primary directives to define direct dependencies between units: Requires and Wants. These directives are placed within the [Unit] section of a unit file (e.g., a .service file).

Requires=

The Requires= directive establishes a strong dependency. If unit A Requires= unit B, systemd starts B when A is started. If B cannot be started, A's start job fails too. If B is explicitly stopped while A is running, A is normally stopped as well because the required relationship no longer holds.

Example:

Consider a web application service (myapp.service) that critically depends on a database service (mariadb.service). The myapp.service unit file might include:

[Unit]
Description=My Web Application
Requires=mariadb.service

[Service]
ExecStart=/usr/bin/myapp

[Install]
WantedBy=multi-user.target

In this scenario, if mariadb.service fails to start or is manually stopped, systemd will also stop myapp.service. If you try to start myapp.service and mariadb.service is not running, systemd will attempt to start mariadb.service first. If mariadb.service fails, myapp.service will not start.

Wants=

The Wants= directive defines a weaker, optional dependency. If unit A Wants= unit B, systemd will try to start unit B when starting unit A, but unit A will still activate even if unit B fails to start or is not running. This is useful for services that benefit from another service but can function independently, perhaps with reduced features or a warning.

Example:

Suppose a monitoring agent (monitoring-agent.service) can run without a specific logging service (app-logger.service) but would ideally have it available. The monitoring-agent.service unit file might look like this:

[Unit]
Description=Monitoring Agent
Wants=app-logger.service

[Service]
ExecStart=/usr/bin/monitoring-agent

[Install]
WantedBy=multi-user.target

Here, systemd will attempt to start app-logger.service when monitoring-agent.service is activated. However, if app-logger.service fails to start, monitoring-agent.service will still proceed to start successfully.

Requires= vs. Wants=

• Requires=: Strong dependency. If the required unit fails, the dependent unit fails or stops.
• Wants=: Weak dependency. The dependent unit attempts to start the wanted unit but proceeds even if it fails.

It's important to note that Requires= implies Wants=. If a unit requires another, it also implicitly wants it.

Ordering Directives: After and Before

While Requires and Wants define what needs to be running, After and Before define when units should be started relative to each other. These directives control the sequence of operations during the system boot process or when units are activated on demand. They are often used in conjunction with dependency directives.

After=

The After= directive specifies that the current unit's start job should be ordered after the listed units' start jobs. It does not, by itself, pull those units into the transaction. It also does not prove that the dependency is logically ready for your application; it only uses systemd's view of the unit's activation state.

Example:

A network-dependent service (custom-network-app.service) should start only after the network is fully configured. This is typically handled by ensuring it starts after the network target (network.target).

[Unit]
Description=Custom Network Application
Requires=network.target
After=network.target

[Service]
ExecStart=/usr/bin/custom-network-app

[Install]
WantedBy=multi-user.target

In this configuration, systemd will order custom-network-app.service after network.target if both are part of the same transaction. For services that need an address, DNS, or a route to another host, network-online.target is often closer to the intent, but only if the distribution's wait-online service is enabled and correctly configured.

Before=

The Before= directive specifies that the current unit should be started before the units listed in Before=. This is useful for services that need to be stopped after others during shutdown, or started before certain services to provide an environment for them.

Example:

Imagine a scenario where a mail server (postfix.service) needs to be running before any user-facing services that might send emails. You might use Before= to ensure postfix.service starts early.

[Unit]
Description=Postfix Mail Transfer Agent
# ... other directives like Conflicts=
Before=user-session.target

[Service]
ExecStart=/usr/lib/postfix/master

[Install]
WantedBy=multi-user.target

This setup attempts to start postfix.service before anything that is part of user-session.target begins its startup. Similarly, during shutdown, postfix.service would be among the last to be stopped if it has a corresponding After=user-session.target.

After= vs. Before=

• After=: Guarantees that the listed units are active before the current unit starts.
• Before=: Guarantees that the current unit starts before the listed units.

After= and Before= are complementary. If unit A says Before=B, the ordering is A first, then B. If unit B says After=A, the result is the same. You usually only need to express the relationship in one unit file. When editing your own service, it is usually clearer to say what your service needs to come after, because that keeps the reasoning local.

Combining Directives for Robust Configurations

In real-world scenarios, you'll often combine these directives to create complex dependency graphs. The multi-user.target is a common target that signifies the system is ready for multi-user operations. Many services are configured to be WantedBy=multi-user.target and After=multi-user.target (or more precisely, After=basic.target and After=getty.target etc. that multi-user.target depends on).

A Common Pattern:

A service that requires a database and should start after the network is configured might look like this:

[Unit]
Description=My Application Service
Requires=mariadb.service
Wants=other-optional-service.service
After=network.target mariadb.service

[Service]
ExecStart=/usr/local/bin/my_app

[Install]
WantedBy=multi-user.target

Explanation of the pattern:

1. Requires=mariadb.service: Guarantees that mariadb.service must be running for my_app.service to function. If mariadb.service fails, my_app.service will stop.
2. Wants=other-optional-service.service: Attempts to start other-optional-service.service but my_app.service will proceed even if it fails.
3. After=network.target mariadb.service: Orders my_app.service after the network target and the MariaDB start job. If the app needs to connect over TCP to a database on another host, use the appropriate network-online setup instead of assuming network.target means "the network is usable."
4. WantedBy=multi-user.target: When enabled (systemctl enable my_app.service), this directive adds a symbolic link so that my_app.service is started when the system reaches the multi-user.target state.

Advanced Considerations and Best Practices

• WantedBy vs. RequiredBy: Similar to Wants vs. Requires, WantedBy is a weak ordering and RequiredBy is a strong ordering. Most services use WantedBy=multi-user.target.
• Conflicts=: This directive specifies units that should not be running concurrently with the current unit. If the current unit is started, conflicting units will be stopped, and vice versa.
• Transitive Dependencies: Dependencies are transitive. If A requires B, and B requires C, then A indirectly requires C. Systemd handles these chains automatically.
• Condition*= Directives: Use ConditionPathExists=, ConditionFileNotEmpty=, ConditionVirtualization=, etc., to make unit activation conditional based on system state, further enhancing robustness.
• Use systemctl list-dependencies <unit>: This command is invaluable for visualizing the dependency tree of a unit, including direct and indirect dependencies.
• Use systemctl status <unit>: Always check the status of your service after making configuration changes. It will often show reasons for failure, including dependency issues.
• Avoid Circular Dependencies: While systemd tries to resolve them, direct circular dependencies (A Requires B, B Requires A) can lead to startup loops or failures. Carefully design your dependencies to avoid this.

A Practical Debugging Pass

When a dependency bug shows up, start by looking at the transaction systemd actually built:

systemctl status my_app.service
journalctl -b -u my_app.service --no-pager
systemctl list-dependencies my_app.service
systemctl show my_app.service -p Wants -p Requires -p After -p Before -p BindsTo -p PartOf

systemctl status tells you whether systemd failed to start the unit, killed it later, or considers it active even though the application is unhealthy. journalctl -b keeps you inside the current boot, which matters because dependency problems are often boot-only. systemctl show is blunt but useful: it shows the final merged unit properties after drop-ins, vendor files, and generated dependencies have been applied.

If you are not sure where a dependency came from, inspect the full unit:

systemctl cat my_app.service

This shows the packaged unit and any override files under /etc/systemd/system/my_app.service.d/. I have seen production services where the base unit was correct but an old override still contained After=mysql.service from a previous migration. The service was waiting on a unit that no longer existed, and the logs made it look like the application was broken.

For boot timing questions, use:

systemd-analyze critical-chain my_app.service
systemd-analyze blame

critical-chain is better than staring at timestamps because it shows which units delayed the path to your service. blame can be misleading if you treat it as a ranking of "bad" services, but it is helpful when one dependency takes far longer than expected.

Patterns That Hold Up in Production

For a local database dependency, this is a reasonable starting point:

[Unit]
Description=API service
Requires=postgresql.service
After=postgresql.service

[Service]
ExecStart=/usr/local/bin/api
User=api
Restart=on-failure

[Install]
WantedBy=multi-user.target

That says PostgreSQL should be started with the API, and the API start should be ordered after PostgreSQL. It does not guarantee that every migration has completed or that the database accepts the exact credentials your app uses. If that matters, add an application-level readiness check. Systemd can order processes, but it cannot understand your schema state unless you teach your service to check it.

For a mounted path, prefer RequiresMountsFor= over manually naming mount units:

[Unit]
RequiresMountsFor=/srv/uploads

[Service]
ExecStart=/usr/local/bin/upload-worker
User=uploads

Systemd will derive the needed mount units for the path. This is easier to maintain than remembering that /srv/uploads maps to srv-uploads.mount.

For optional helpers, use Wants=:

[Unit]
Wants=metrics-agent.service
After=metrics-agent.service

If the metrics agent fails, the main service can still start. That is usually what you want for logging sidecars, optional exporters, and local notification helpers. Do not use Requires= just because two services are related. Use it only when the dependent service truly cannot do useful work without the other unit.

For tightly coupled services that should stop together, look at BindsTo= and PartOf=. BindsTo= is stronger than Requires= and is useful when a service should disappear if the bound unit disappears, such as a service tied to a specific device unit. PartOf= is often useful for groups: restarting or stopping the parent unit can propagate to child units. These are not first-choice directives, but they solve problems that Requires= cannot express cleanly.

Common Traps

Do not add After=multi-user.target to a normal long-running service that is enabled with WantedBy=multi-user.target. That often creates odd ordering and rarely says what the author intended. Most services are pulled in by multi-user.target; they do not need to start after it has already been reached.

Do not assume network.target means "internet is reachable." It is a synchronization point for network management, not a connectivity test. If your application talks to a remote API, add retry logic inside the application anyway. Boot-time network ordering reduces noise, but it cannot protect you from DNS failure, routing changes, or a remote dependency being down.

Do not hide long sleeps in ExecStartPre=/bin/sleep 30 unless you have no better option. Sleeps make boots slower when the dependency is ready quickly and still fail when the dependency takes longer than expected. A small readiness loop that checks the actual socket, file, or API is usually clearer.

When you change dependency directives, run systemctl daemon-reload, restart the service, and check the journal from the same boot. The fastest fix is usually the one that proves the exact ordering assumption that was wrong.