Master DevOps Tools & Best Practices - DevOps Knowledge Hub Master DevOps Tools & Best Practices - DevOps Knowledge Hub

Mastering Multi-Stage Deployments Using Sequential Ansible Playbooks

Learn how to design and execute complex, multi-stage application deployments using Ansible. This guide covers creating sequential playbooks for distinct deployment phases, implementing effective error handling, and developing rollback strategies. Master the art of robust, automated application delivery with practical examples and best practices.

Mastering Multi-Stage Deployments Using Sequential Ansible Playbooks

Multi-stage Ansible deployments become necessary when "copy the files and restart the service" is no longer honest. A real deployment may need a database migration, a feature flag change, a package rollout, a service reload, a health check, and a rollback path if the new version fails. If all of that lives in one large playbook with unclear boundaries, every failed deploy turns into a reading exercise.

Sequential playbooks give each stage a clear job. You can run them from a CI/CD pipeline, AWX, Ansible Automation Platform, or a simple shell script. The important part is not the tool that presses the button. The important part is that the deployment has an order, each stage can be retried safely, and failure handling is explicit.

Why Sequential Playbooks for Multi-Stage Deployments?

Deploying an application often involves more than just copying files. You might need to:

  • Prepare the environment: Create directories, set permissions, install dependencies.
  • Update the database: Run schema migrations, seed initial data.
  • Deploy application code: Transfer new code versions, restart services.
  • Configure services: Update application configurations, reload daemons.
  • Perform post-deployment checks: Run smoke tests, verify service availability.

The sequence matters because some operations are easy to roll back and others are not. Reverting a symlink to the previous release is usually simple. Reverting a destructive database migration may not be. That difference should shape the deployment plan before anyone writes YAML.

Breaking these into distinct, sequential playbooks provides several advantages:

  • Modularity: Each playbook focuses on a single stage, making them easier to understand, maintain, and reuse.
  • Readability: Complex logic is divided into manageable chunks.
  • Control: You can execute specific stages independently or as part of a larger workflow.
  • Error Isolation: If a failure occurs in one stage, it's easier to pinpoint the cause and roll back specific changes without affecting other parts of the deployment.
  • Idempotency: Well-written playbooks are inherently idempotent, meaning running them multiple times has the same effect as running them once. This is crucial for safe retries.

There is a tradeoff. Separate playbooks add orchestration work. Variables, artifacts, and status may need to move from one stage to another. For a small internal service, one playbook with tagged blocks may be enough. For a customer-facing application with migrations and rollback requirements, the extra structure usually pays for itself.

Designing Your Multi-Stage Deployment Workflow

Before writing any Ansible code, plan your deployment stages. Identify the logical steps, their dependencies, and the order of execution. A common workflow might look like this:

  1. Pre-deployment Checks: Ensure the target environment is ready.
  2. Database Migration: Apply necessary database schema changes.
  3. Application Deployment: Deploy the new version of the application code.
  4. Service Restart/Reload: Bring the application services online with the new code.
  5. Post-deployment Verification: Run tests to confirm the deployment's success.

For each stage, consider what Ansible tasks are required and which playbook will contain them.

Also decide which stages are allowed to change production state. A smoke-test playbook should not quietly repair configuration. A preflight playbook should not install missing packages unless that is explicitly part of the deployment contract. Keeping read-only checks separate from mutating steps makes the workflow easier to trust.

Here is a practical directory layout:

deploy/
  inventories/
    staging.ini
    production.ini
  group_vars/
    all.yml
    production.yml
  playbooks/
    00-preflight.yml
    01-migrate-db.yml
    02-deploy-app.yml
    03-reload-services.yml
    04-smoke-test.yml
    rollback-app.yml

The numbers are not magic. They just make order visible in file listings and CI logs.

Executing Playbooks Sequentially

Ansible provides a straightforward way to run playbooks one after another using the --playbook-dir and ansible-playbook commands. The simplest method is to chain commands in your CI/CD pipeline or on the command line.

Let's assume you have the following playbook files:

  • 01-database-migration.yml
  • 02-deploy-application.yml
  • 03-restart-services.yml
  • 04-smoke-tests.yml

You can execute them sequentially like this:

ansible-playbook -i inventory.ini 01-database-migration.yml
ansible-playbook -i inventory.ini 02-deploy-application.yml
ansible-playbook -i inventory.ini 03-restart-services.yml
ansible-playbook -i inventory.ini 04-smoke-tests.yml

In practice, wrap that sequence so a failed stage stops the pipeline:

set -euo pipefail

ansible-playbook -i inventories/production.ini playbooks/00-preflight.yml
ansible-playbook -i inventories/production.ini playbooks/01-migrate-db.yml
ansible-playbook -i inventories/production.ini playbooks/02-deploy-app.yml
ansible-playbook -i inventories/production.ini playbooks/03-reload-services.yml
ansible-playbook -i inventories/production.ini playbooks/04-smoke-test.yml

set -e is not a deployment strategy by itself, but it prevents the worst mistake: continuing after a failed stage as if nothing happened. CI systems usually provide their own failure behavior, but the same idea applies.

Using ansible-playbook --skip-tags or --limit

In more advanced scenarios, you might combine multiple logical steps into a single playbook but use tags to control execution. However, for true multi-stage separation, distinct playbooks are generally preferred. If you want to run a subset of playbooks or skip certain ones, you can use command-line arguments.

Skipping a playbook: If 03-restart-services.yml fails because of a temporary service issue, you might rerun only that stage after fixing the cause. Do not skip stages blindly when earlier stages produce artifacts or state that later stages depend on.

Limiting to a specific stage: You can also limit the execution to a specific host or group using the --limit flag, which can be useful for testing.

For rolling deployments, --limit can also reduce blast radius:

ansible-playbook -i inventories/production.ini playbooks/02-deploy-app.yml --limit web_canary

Run the deployment against one host or one small group, verify it, then continue to the rest of the fleet. This is especially useful when your load balancer supports draining hosts before reload or restart.

Incorporating Error Handling and Rollback Strategies

Robust deployments require a plan for when things go wrong.

ignore_errors and failed_when

By default, Ansible stops execution if a task fails. You can control this behavior:

  • ignore_errors: true: Allows the playbook to continue even if a task fails. Use this cautiously, typically for non-critical tasks or when you have a subsequent task to clean up or compensate.
  • failed_when:: Define custom conditions under which a task should be considered failed. This is powerful for handling expected non-fatal errors or validating specific outcomes.
- name: Check service status (potentially non-fatal)
  command: systemctl status myapp
  register: service_status
  ignore_errors: true

- name: Fail if service is not active
  fail:
    msg: "Service myapp is not running!"
  when: "service_status.rc != 0"

Use ignore_errors sparingly. It is often better to register the result and make a clear decision. A deployment log full of ignored failures teaches people to stop reading failures.

For commands, prefer purpose-built modules when they exist. For example, use ansible.builtin.service, ansible.builtin.systemd, ansible.builtin.copy, ansible.builtin.template, and package modules instead of shelling out. Modules usually give better idempotency and clearer changed and failed states.

Rollback Playbooks

For critical deployments, have dedicated rollback playbooks. These playbooks should be designed to revert the changes made by their corresponding deployment playbooks.

  • 01-database-migration-rollback.yml: Reverts schema changes.
  • 02-deploy-application-rollback.yml: Deploys the previous application version or restores a backup.
  • 03-restart-services-rollback.yml: Restarts services in their previous state.

Database rollback deserves special care. Some migrations cannot be safely reversed after writes begin using the new schema. A safer pattern is often expand-and-contract: add backward-compatible schema changes, deploy application code that can work with both old and new shapes, backfill data if needed, then remove old columns or fields in a later deployment.

With that model, rollback usually means reverting application code and leaving the compatible schema in place, not trying to undo a risky database change under pressure.

Example Rollback Trigger: In your CI/CD pipeline, if the 04-smoke-tests.yml playbook fails, you would trigger the execution of rollback playbooks in reverse order.

# If 04-smoke-tests.yml fails:
ansible-playbook -i inventory.ini 03-restart-services-rollback.yml
ansible-playbook -i inventory.ini 02-deploy-application-rollback.yml
ansible-playbook -i inventory.ini 01-database-migration-rollback.yml

Using block, rescue, and always

Ansible's block, rescue, and always constructs provide a more structured way to handle errors within a single playbook. While not for sequencing across playbooks, they are excellent for encapsulating a series of tasks that might fail and defining what to do in case of failure.

- block:
    - name: Deploy new application code
      copy:
        src: /path/to/new/app/
        dest: /var/www/myapp/

    - name: Restart application service
      service:
        name: myapp
        state: restarted

  rescue:
    - name: Attempt to revert to previous version
      copy:
        src: /path/to/old/app/
        dest: /var/www/myapp/

    - name: Restart application service after rollback
      service:
        name: myapp
        state: restarted

  always:
    - name: Log deployment attempt
      debug:
        msg: "Deployment attempt finished."

This approach is useful for grouping related tasks within a single deployment stage playbook.

For cross-playbook rollback, let the orchestrator make the decision. A CI pipeline can run rollback playbooks only if a later stage fails. AWX job workflows can model the same success and failure branches visually. Keep the rollback command boring and rehearsed.

Passing Release State Between Stages

Sequential playbooks often need a shared release identifier. For example, the deploy stage needs to know which artifact to install, the smoke test needs to know which version to expect, and rollback needs to know the previous version.

Pass that state explicitly:

ansible-playbook -i inventories/production.ini playbooks/02-deploy-app.yml \
  -e release_version=2026.05.24.3 \
  -e artifact_url=https://artifacts.example.com/myapp/2026.05.24.3.tar.gz

Inside the playbook, record what changed:

- name: Write current release marker
  ansible.builtin.copy:
    dest: /opt/myapp/current-release.txt
    content: "{{ release_version }}\n"
    owner: root
    group: root
    mode: "0644"

That marker helps during incidents. When someone SSHes into a host, they can see what version the host believes it is running. You can also have the smoke-test playbook read the marker and compare it to the expected release.

Advanced Considerations

Managing State Between Playbooks

Sometimes, a task in one playbook needs to inform another playbook about its outcome. You can achieve this using:

  • Fact Caching: If fact caching is enabled, facts gathered by one playbook can be available to subsequent ones run within the same Ansible session.
  • Temporary Files/Databases: Write critical status information or outputs to a temporary file or a dedicated status table that subsequent playbooks can read.

Prefer explicit state over hidden state. Fact caching can be useful, but it can also confuse people when values are stale or when one runner has cache enabled and another does not. Release files, artifact metadata, CI variables, and deployment records are easier to inspect.

Version Control and Orchestration Tools

For complex orchestrations, consider integrating your sequential Ansible playbooks into a higher-level tool:

  • CI/CD Pipelines: Tools like Jenkins, GitLab CI, GitHub Actions, or CircleCI are excellent for defining and triggering multi-stage deployments. You define the sequence of ansible-playbook commands within the pipeline configuration.
  • Ansible Tower/AWX: For enterprise-grade orchestration, Ansible Tower (now Automation Platform) or its open-source counterpart AWX provides a robust UI for scheduling, monitoring, and managing complex job templates that can chain multiple playbooks.

If several people deploy the same system, central orchestration becomes less about convenience and more about control. It gives you consistent inventories, credentials, audit logs, approvals, and a visible history of which stage failed. Those details matter during a production incident.

Tagging for Granular Control

While we advocate for separate playbooks for distinct stages, you can also use tags within playbooks. If you have a very large playbook for a single stage (e.g., database migration), you can tag specific tasks and run only those using ansible-playbook --tags <tag_name>.

This is more about granular control within a stage rather than sequencing between stages.

Best Practices for Multi-Stage Deployments

  • Keep Playbooks Focused: Each playbook should do one thing well (e.g., database migration, application deployment).
  • Name Playbooks Clearly: Use a naming convention that reflects the stage and order (e.g., 01-, 02-).
  • Implement Idempotency: Ensure all tasks are idempotent to allow for safe retries.
  • Test Rollbacks: Regularly test your rollback procedures to ensure they work as expected.
  • Use Version Control: Store all your playbooks and inventory files in a version control system (like Git).
  • Automate the Orchestration: Use CI/CD pipelines or tools like Ansible Tower/AWX to automate the execution of your sequential playbooks.
  • Document Your Workflow: Clearly document the stages, their purpose, dependencies, and rollback procedures.
  • Make smoke tests real: Check the actual endpoint, login path, queue worker, or background job that matters. A plain process check is not enough.
  • Protect production inventories: Use separate inventories and credentials for staging and production. A typo in --limit should not deploy to the wrong place.
  • Use serial rollout when possible: serial lets you update a few hosts at a time and stop before the whole fleet is affected.
- name: Deploy application gradually
  hosts: web
  serial: 2
  tasks:
    - name: Install release
      ansible.builtin.unarchive:
        src: "{{ artifact_path }}"
        dest: /opt/myapp/releases/{{ release_version }}
        remote_src: true

With serial, Ansible processes hosts in batches. Combine it with load balancer draining if your application cannot be restarted without dropping active requests.

A Concrete Deployment Flow

A safe Ansible deployment for a web application might look like this:

00-preflight.yml checks disk space, confirms the target release exists, verifies database connectivity, and makes sure the hosts are in the expected environment. It does not change the system.

01-migrate-db.yml runs only backward-compatible migrations. It records the migration version and fails if the database is already ahead of the requested release.

02-deploy-app.yml downloads the artifact, unpacks it into a versioned release directory, templates configuration, and updates a current symlink. It does not restart services yet.

03-reload-services.yml drains each host from the load balancer, reloads or restarts the service, waits for the local health endpoint, and then returns the host to service.

04-smoke-test.yml calls the public endpoint through the same path users take. It checks the response body or version endpoint, not just a 200 from a load balancer default page.

This flow is slower than a one-command restart. It is also much easier to reason about when the deploy fails halfway through.

The Habit That Makes This Work

Sequential Ansible playbooks work best when each one has a narrow contract: what it expects, what it changes, how it proves success, and what to do if it fails. That contract matters more than the number of YAML files.

Start with the stages that reflect your real risk: preflight, migration, deploy, reload, smoke test, rollback. Keep the commands boring. Test the rollback before you need it. When a deployment breaks, you should be able to point to the exact stage that failed and decide the next step without rereading the entire automation tree.