Securely Managing Environment Variables in Systemd Service Units

Environment variables are convenient, but they are not automatically private. In a systemd service, they can appear in unit files, drop-ins, systemctl show, process inspection tools, crash reports, debug logs, or copied support bundles. That does not mean you can never use them. It means you should be deliberate about what goes into them and who can read the files that define them.

This guide covers the two common directives, Environment= and EnvironmentFile=, then shows how to use drop-ins so local configuration stays separate from package-managed units.

---

The Role of Environment Variables in Systemd

Environment variables provide a straightforward way to configure a service without changing its code. When systemd starts a service, it builds the process environment and applies variables defined in the unit before executing ExecStart=.

Systemd provides two primary directives within the [Service] section of a unit file for managing these variables.

1. Direct Definition: The Environment Directive

This method allows you to define variables directly within the Systemd unit file. This is suitable for non-sensitive configuration parameters that rarely change.

Usage and Syntax

The Environment directive accepts a space-separated list of variable assignments in the format "KEY=VALUE".

# /etc/systemd/system/my-app.service

[Unit]
Description=My Application Service

[Service]
User=myuser
WorkingDirectory=/opt/my-app

# Define variables directly in the unit file
Environment="APP_PORT=8080" "NODE_ENV=production"

ExecStart=/usr/local/bin/my-app --start

[Install]
WantedBy=multi-user.target

Limitations and Security

While convenient, the Environment directive is a poor place for passwords, tokens, or database credentials. Unit files are often stored in configuration management systems, copied into tickets, or readable by operators who need to inspect service behavior but should not see secrets. Use it for values such as ports, feature flags, log levels, and paths.

2. External Configuration: The EnvironmentFile Directive

For larger configurations, loading variables from an external file is usually cleaner. It lets you manage the variable file's permissions independently from the main unit file. It also keeps package-provided units readable while local settings live in /etc.

Usage and Syntax

The EnvironmentFile directive takes an absolute path to a configuration file. Systemd reads this file line-by-line, treating each line as a potential KEY=VALUE assignment.

[Service]
# Load variables from an external file
EnvironmentFile=/etc/config/my-app-settings.conf

ExecStart=/usr/local/bin/my-app --start

Environment File Format

The external file must adhere to a simple shell-like format:

• Lines starting with # are treated as comments.
• Lines starting with an empty variable assignment (VAR=) will clear the variable if it was previously set.
• Variables are defined as KEY=VALUE.
• Quoting the value (KEY="VALUE WITH SPACES") is supported.

# /etc/config/my-app-settings.conf

# Non-sensitive variables
MAX_WORKERS=4
LOG_LEVEL=INFO

# Sensitive variable (requires strict file permissions and careful access control)
DB_PASSWORD=SecureRandomString12345

Avoid shell habits that systemd's environment file parser does not support the way you expect. Do not write export KEY=value. Do not put spaces around the equals sign. If the value contains spaces, quote it. If the value contains literal quotes, backslashes, or newlines, test it before relying on it in production.

Handling Missing Files

By default, if the file specified by EnvironmentFile does not exist, Systemd will fail the service startup. If the environment file is optional, you can prefix the file path with a hyphen (-):

EnvironmentFile=-/etc/config/optional-settings.conf

If the file is prefixed with -, Systemd will ignore errors caused by the file not being present.

Best Practice: Using Drop-in Units for Sensitive Data

Modifying the core unit file (e.g., /usr/lib/systemd/system/my-app.service) is generally discouraged, especially if the file is managed by a package manager. Instead, use drop-in unit files to apply configuration overrides or additions.

This practice matters because it separates vendor defaults from local configuration. It also makes audits easier: the unit says where configuration is loaded from, and the permissions on that file say who can read it.

Step-by-Step Drop-in Configuration

1. Locate/Create the Drop-in Directory

For a service named my-app.service, the drop-in directory must be named my-app.service.d/ and reside in the /etc/systemd/system/ hierarchy.

sudo mkdir -p /etc/systemd/system/my-app.service.d/

2. Create the Configuration Override

Create a file inside the drop-in directory (e.g., secrets.conf). This file only needs the [Service] section and the specific directives you wish to override or add.

# /etc/systemd/system/my-app.service.d/secrets.conf

[Service]
# Load the secure credentials file
EnvironmentFile=/etc/secrets/my-app-credentials.env

3. Secure the External Environment File

This is the most critical security step. Ensure the external file containing the secrets has restrictive permissions. Ideally, it should be owned by root:root and only readable by the root user or the service user itself.

# Create the secrets file
sudo touch /etc/secrets/my-app-credentials.env

# Populate the file with secrets
sudo sh -c 'echo "DB_PASS=S3cr3tP@ssw0rd" >> /etc/secrets/my-app-credentials.env'

# Set restrictive permissions
sudo chmod 600 /etc/secrets/my-app-credentials.env

If the file referenced by EnvironmentFile contains credentials, keep it readable only by the account that needs to manage the service. 0600 root:root is common when systemd reads the file before dropping privileges with User=, but some operational models use a dedicated root-owned group and 0640. The important part is that ordinary users cannot read the file.

Also be honest about the remaining risk. Environment variables are easier to handle than hardcoded command-line arguments, but they are still not a full secrets-management system. For higher-risk credentials, consider a dedicated secret store, short-lived credentials, systemd credentials on newer distributions, or an application-specific mechanism that reads a protected file directly.

Troubleshooting and Verification

After making any changes to unit files or drop-ins, you must reload the Systemd manager configuration.

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

To verify which environment variables have been successfully loaded by Systemd for a running service, use the systemctl show command and specifically query the Environment property:

systemctl show my-app.service --property=Environment

Example Output (showing loaded variables):

Environment=APP_PORT=8080 NODE_ENV=production DB_PASS=S3cr3tP@ssw0rd

That command is useful for debugging, but it is also a reminder: anyone allowed to run the right inspection commands as root can see the values. Do not paste this output into shared chat, tickets, or public bug reports without redacting it.

If the service fails to start, check the service logs using journalctl -xeu my-app.service. Common reasons for failure related to environment variables include:

1. Incorrect file path in EnvironmentFile.
2. Missing file (and the path was not prefixed with -).
3. Incorrect variable syntax in the external environment file (e.g., spaces around the = sign).

Practical Patterns That Work

Scenario	Directive to Use	Location Best Practice	Security Considerations
Static, Non-Sensitive Config	Environment	Direct unit file or drop-in	Low security risk.
Sensitive Credentials (Secrets)	EnvironmentFile	External file, referenced via a drop-in (*.service.d/)	CRITICAL: Environment file must have 0600 permissions.
Modularity & Overrides	EnvironmentFile	Drop-in unit file	Separates configuration from vendor defaults.

By leveraging the EnvironmentFile directive within a dedicated drop-in unit and ensuring strict file permissions, administrators can securely and flexibly manage service configurations, adhering to principles of least privilege and separation of concerns.

For a small internal service, a reasonable setup often looks like this:

# /etc/systemd/system/my-app.service.d/env.conf
[Service]
Environment="APP_ENV=production"
EnvironmentFile=/etc/my-app/runtime.env
EnvironmentFile=-/etc/my-app/local.env

runtime.env contains required values. local.env is optional and lets an operator override a setting during a maintenance window without editing the main unit. After a change:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service
sudo journalctl -u my-app.service -n 50 --no-pager

The safest habit is simple: keep non-sensitive defaults in the unit or a normal config file, keep secrets out of package-owned units, lock down any file that contains credentials, and verify the loaded environment without leaking it into places where it does not belong.

Common Mistakes Worth Avoiding

The first mistake is putting secrets in ExecStart=:

ExecStart=/usr/local/bin/my-app --db-password=s3cret

That looks harmless when you are rushing, but command-line arguments are often easier to expose than environment files. They may show up in process listings, monitoring tools, shell history, crash reports, or copied service definitions. If the application supports reading a protected config file, that is usually better. If it expects an environment variable, use a protected EnvironmentFile= and keep the value out of the command line.

The second mistake is editing the vendor unit directly. A package upgrade can replace the file, and the next restart may silently drop your environment settings. Use a drop-in:

sudo systemctl edit my-app.service

Then add only the local override:

[Service]
EnvironmentFile=/etc/my-app/my-app.env

The third mistake is assuming the service sees the same shell environment you see in your terminal. It usually does not. Your interactive shell may have variables from .bashrc, .profile, an SSH session, or a deployment tool. A system service starts from systemd's managed environment. If the app needs PATH, JAVA_HOME, NODE_ENV, LD_LIBRARY_PATH, or a similar value, define it explicitly or use absolute paths.

For example, this is fragile:

ExecStart=npm start

This is easier to reason about:

WorkingDirectory=/opt/my-app
Environment="NODE_ENV=production"
ExecStart=/usr/bin/npm start

The fourth mistake is making the environment file writable by the service user when it does not need to be. A web app that can overwrite its own environment file can turn a normal application bug into a persistence problem. In many setups, the service user should read application data and write logs or uploads, but it should not be able to rewrite the credentials used to start the service.

When Environment Variables Are the Wrong Tool

Environment variables are popular because they are simple, but they are not always the best interface. If a value is large, structured, rotated often, or shared by several services, a real configuration file or secret store is usually easier to manage.

A database URL is a reasonable environment variable:

DATABASE_URL=postgresql://[email protected]:5432/app

A full JSON service account document is less pleasant. Quoting becomes awkward, accidental line breaks cause failures, and people are more likely to paste it into logs while debugging. In that case, store the JSON in a protected file and pass the file path:

GOOGLE_APPLICATION_CREDENTIALS=/etc/my-app/google-service-account.json

Then protect the JSON file separately:

sudo chown root:my-app /etc/my-app/google-service-account.json
sudo chmod 640 /etc/my-app/google-service-account.json

This does not make the secret magical. The application can still read it. Root can still read it. But it avoids cramming a complex secret into systemd's environment parser and makes file-level auditing clearer.

A Safer Review Checklist

Before restarting a service that uses environment variables, check four things:

systemctl cat my-app.service
sudo ls -l /etc/my-app/my-app.env
sudo systemd-analyze verify /etc/systemd/system/my-app.service
sudo systemctl daemon-reload

systemctl cat confirms which drop-ins are active. ls -l confirms the permissions are what you intended. systemd-analyze verify can catch some unit syntax problems before you restart. It will not validate every application-specific setting, but it is still a useful guardrail.

After restart, check the journal for startup errors:

sudo systemctl restart my-app.service
sudo journalctl -u my-app.service -n 100 --no-pager

If you need to confirm a variable loaded, query it carefully and redact the output before sharing it. For sensitive services, I prefer checking for a non-secret variable such as APP_ENV or LOG_LEVEL first. If that loaded from the same file, the file path and parser syntax are probably correct, and you may not need to print the secret-bearing values at all.

One final practical point: plan rotation before you need it. If a password or token is stored in an environment file, write down which service must restart after the value changes and whether that restart causes downtime. A credential that is easy to set but hard to rotate will eventually become an incident. For small services, the rotation runbook may be only four lines:

sudoedit /etc/my-app/my-app.env
sudo systemctl restart my-app.service
sudo systemctl status my-app.service
sudo journalctl -u my-app.service -n 50 --no-pager

That is enough if everyone knows the blast radius. For larger systems, prefer credentials that can overlap during rotation, so you can deploy the new value, verify it, and remove the old value without a rushed outage window.