Securely Managing Environment Variables in Systemd Service Units

Systemd, as the primary system and service manager for modern Linux distributions, relies on service unit files (.service) to define how applications are started, stopped, and maintained. A critical aspect of configuring any modern application is the injection of configuration settings, paths, and, most importantly, sensitive secrets like API keys or database credentials.

Improper management of these environment variables can lead to security vulnerabilities, difficulty in debugging, and non-portable configurations. This guide details the appropriate Systemd directives—Environment and EnvironmentFile—and demonstrates the secure use of drop-in configuration files to handle sensitive data, ensuring separation of concerns and robust security practices.

The Role of Environment Variables in Systemd

Environment variables provide a straightforward mechanism for configuring a service without modifying its binary or code. When Systemd starts a service, it constructs a complete environment (including the necessary PATH, user/group variables, etc.) and injects any variables defined in the unit file before executing the ExecStart command.

Systemd provides two primary directives within the [Service] section of a unit file for managing these variables.

1. Direct Definition: The `Environment` Directive

This method allows you to define variables directly within the Systemd unit file. This is suitable for non-sensitive configuration parameters that rarely change.

Usage and Syntax

The Environment directive accepts a space-separated list of variable assignments in the format "KEY=VALUE".

# /etc/systemd/system/my-app.service

[Unit]
Description=My Application Service

[Service]
User=myuser
WorkingDirectory=/opt/my-app

# Define variables directly in the unit file
Environment="APP_PORT=8080" "NODE_ENV=production"

ExecStart=/usr/local/bin/my-app --start

[Install]
WantedBy=multi-user.target

Limitations and Security

While convenient, the Environment directive should never be used for sensitive information (secrets, passwords, API keys). Unit files are often stored in configuration management systems or located in directories accessible to various users (even if read-only, they might be viewable by non-root users depending on configuration). Hardcoding secrets directly compromises security principles.

2. External Configuration: The `EnvironmentFile` Directive

For complex configurations, dynamic variables, or sensitive data, loading variables from an external file is the preferred method. This allows you to manage the variable file's permissions independently from the main unit file.

Usage and Syntax

The EnvironmentFile directive takes an absolute path to a configuration file. Systemd reads this file line-by-line, treating each line as a potential KEY=VALUE assignment.

[Service]
# Load variables from an external file
EnvironmentFile=/etc/config/my-app-settings.conf

ExecStart=/usr/local/bin/my-app --start

Environment File Format

The external file must adhere to a simple shell-like format:

Lines starting with # are treated as comments.
Lines starting with an empty variable assignment (VAR=) will clear the variable if it was previously set.
Variables are defined as KEY=VALUE.
Quoting the value (KEY="VALUE WITH SPACES") is supported.

# /etc/config/my-app-settings.conf

# Non-sensitive variables
MAX_WORKERS=4
LOG_LEVEL=INFO

# Sensitive variable (requires strict file permissions)
DB_PASSWORD=SecureRandomString12345

Handling Missing Files

By default, if the file specified by EnvironmentFile does not exist, Systemd will fail the service startup. If the environment file is optional, you can prefix the file path with a hyphen (-):

EnvironmentFile=-/etc/config/optional-settings.conf

If the file is prefixed with -, Systemd will ignore errors caused by the file not being present.

Best Practice: Using Drop-in Units for Sensitive Data

Modifying the core unit file (e.g., /usr/lib/systemd/system/my-app.service) is generally discouraged, especially if the file is managed by a package manager. Instead, use drop-in unit files to apply configuration overrides or additions.

This practice is crucial when dealing with sensitive environment variables, as it allows you to separate the standard service configuration from the local secrets file paths.

Step-by-Step Drop-in Configuration

1. Locate/Create the Drop-in Directory

For a service named my-app.service, the drop-in directory must be named my-app.service.d/ and reside in the /etc/systemd/system/ hierarchy.

sudo mkdir -p /etc/systemd/system/my-app.service.d/

2. Create the Configuration Override

Create a file inside the drop-in directory (e.g., secrets.conf). This file only needs the [Service] section and the specific directives you wish to override or add.

# /etc/systemd/system/my-app.service.d/secrets.conf

[Service]
# Load the secure credentials file
EnvironmentFile=/etc/secrets/my-app-credentials.env

3. Secure the External Environment File

This is the most critical security step. Ensure the external file containing the secrets has restrictive permissions. Ideally, it should be owned by root:root and only readable by the root user or the service user itself.

# Create the secrets file
sudo touch /etc/secrets/my-app-credentials.env

# Populate the file with secrets
sudo sh -c 'echo "DB_PASS=S3cr3tP@ssw0rd" >> /etc/secrets/my-app-credentials.env'

# Set restrictive permissions (root read-only)
sudo chmod 600 /etc/secrets/my-app-credentials.env

⚠️ Security Warning: File Permissions

If the file referenced by EnvironmentFile contains credentials, the permissions must be set to 0600 or stricter. If the file is readable by other users, the secrets will be exposed during service startup or manual inspection.

Troubleshooting and Verification

After making any changes to unit files or drop-ins, you must reload the Systemd manager configuration.

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

To verify which environment variables have been successfully loaded by Systemd for a running service, use the systemctl show command and specifically query the Environment property:

systemctl show my-app.service --property=Environment

Example Output (showing loaded variables):

Environment=APP_PORT=8080 NODE_ENV=production DB_PASS=S3cr3tP@ssw0rd

If the service fails to start, check the service logs using journalctl -xeu my-app.service. Common reasons for failure related to environment variables include:

Incorrect file path in EnvironmentFile.
Missing file (and the path was not prefixed with -).
Incorrect variable syntax in the external environment file (e.g., spaces around the = sign).

Summary of Best Practices

Scenario	Directive to Use	Location Best Practice	Security Considerations
Static, Non-Sensitive Config	`Environment`	Direct unit file or drop-in	Low security risk.
Sensitive Credentials (Secrets)	`EnvironmentFile`	External file, referenced via a drop-in (`*.service.d/`)	CRITICAL: Environment file must have `0600` permissions.
Modularity & Overrides	`EnvironmentFile`	Drop-in unit file	Separates configuration from vendor defaults.

By leveraging the EnvironmentFile directive within a dedicated drop-in unit and ensuring strict file permissions, administrators can securely and flexibly manage service configurations, adhering to principles of least privilege and separation of concerns.