Troubleshooting Failed Builds Quickly Using the Jenkins CLI

Jenkins is the backbone of continuous integration and continuous delivery (CI/CD) pipelines for countless organizations. While it automates the build, test, and deployment processes, build failures are an inevitable part of software development. When a build fails, rapid diagnosis and resolution are critical to minimize downtime and keep development moving forward.

While the Jenkins web UI provides extensive information, sometimes the fastest way to troubleshoot is directly through the Jenkins Command Line Interface (CLI). The CLI offers a powerful, scriptable, and often quicker alternative to navigating the UI, especially for repetitive tasks or when dealing with numerous build logs. This guide will walk you through leveraging the Jenkins CLI to quickly diagnose build failures, retrieve detailed logs, inspect environmental variables, and efficiently restart builds.

Why Use the Jenkins CLI for Troubleshooting?

The Jenkins CLI provides several advantages for troubleshooting:

• Speed: Quickly fetch logs and information without browser navigation.
• Automation: Integrate troubleshooting steps into scripts or automated reports.
• Remote Access: Perform diagnostics from any terminal with network access to your Jenkins instance.
• Efficiency: Filter logs and information using standard shell tools like grep, awk, and sed.

Prerequisites

Before you can start troubleshooting with the Jenkins CLI, ensure you have the following:

1. Jenkins Server Running: An active Jenkins instance that you have administrative access to.
2. Jenkins CLI JAR: Download the jenkins-cli.jar file from your Jenkins instance. You can usually find it at JENKINS_URL/jnlpJars/jenkins-cli.jar.
3. Authentication: The CLI requires authentication. The recommended method is to use an API token for a Jenkins user. Generate an API token via Your_User_Name -> Configure -> Add new Token.

Setting up the CLI

First, download the jenkins-cli.jar:

wget JENKINS_URL/jnlpJars/jenkins-cli.jar
# Or using curl
curl -O JENKINS_URL/jnlpJars/jenkins-cli.jar

To simplify commands, you can set environment variables for your Jenkins URL, username, and API token:

export JENKINS_URL="http://your-jenkins-instance.com"
export JENKINS_USER="your_username"
export JENKINS_API_TOKEN="your_api_token"

# Alias the CLI command for convenience
alias jcli='java -jar jenkins-cli.jar -s "$JENKINS_URL" -auth "$JENKINS_USER:$JENKINS_API_TOKEN"'

Now, you can simply use jcli followed by the command.

Identifying Failed Builds

The first step in troubleshooting is to identify which job and build have failed. While the CLI doesn't have a direct command to list only failed builds, you can list jobs and then inspect them, or use Groovy for more advanced filtering.

Listing Jobs

To see a list of all jobs on your Jenkins instance:

jcli list-jobs

This provides a basic list. To get more detailed information about a specific job, including its last build status, use get-job:

jcli get-job MyPipelineJob

The output (in XML format by default) will contain information like lastFailedBuild, lastSuccessfulBuild, etc., which you can parse.

Tip: Using Groovy for Advanced Filtering

For more advanced filtering, especially to find specific failed builds, you can execute Groovy scripts via the CLI. This is incredibly powerful.

jcli groovy =
    'Jenkins.instance.getAllItems(hudson.model.Job.class).each { job ->
        def lastBuild = job.getLastBuild()
        if (lastBuild != null && lastBuild.result == hudson.model.Result.FAILURE) {
            println "Failed Job: ${job.name}, Build: ${lastBuild.number}"
        }
    }'

Retrieving Detailed Build Logs

The most common and crucial step in troubleshooting is reviewing the build logs (console output). The Jenkins CLI makes this straightforward.

Getting Console Output

To retrieve the full console output of a specific build, use the console command:

jcli console MyPipelineJob 123

Replace MyPipelineJob with your job's name and 123 with the build number. This will dump the entire log to your terminal.

Filtering Logs for Errors

When logs are extensive, parsing them manually is inefficient. Leverage grep to quickly find relevant error messages, stack traces, or keywords.

jcli console MyPipelineJob 123 | grep -iE "error|fail|exception|stacktrace"

• -i: Ignore case.
• -E: Use extended regular expressions (allows | for OR).

This command significantly narrows down the output, helping you pinpoint the failure cause faster.

Monitoring Live Builds

For builds that are still running but appear to be stuck or are failing slowly, you can monitor their console output in real-time similar to tail -f:

jcli console MyPipelineJob LAST_BUILD_NUMBER --follow

This will continuously stream new log entries as they appear, until the build finishes or you stop the command.

Inspecting Build Environment Variables

Environment variables often play a critical role in build execution, influencing paths, secrets, and configuration. Incorrect or missing environment variables can lead to build failures. While there isn't a direct CLI command to list all environment variables of a past build, you can retrieve them using a Groovy script executed via the CLI.

First, ensure your pipeline explicitly prints relevant environment variables or that you have access to a dumpEnvVars step (if using Pipeline Utility Steps plugin). If not, you can use Groovy.

Using Groovy to Access Environment Variables

jcli groovy =
    'def job = Jenkins.instance.getItemByFullName("MyPipelineJob")
    def build = job.getBuildByNumber(123)
    if (build) {
        build.getEnvironment().each { key, value ->
            println "${key}=${value}"
        }
    } else {
        println "Build 123 not found for MyPipelineJob"
    }'

This script connects to the Jenkins API, retrieves the specified job and build, and then iterates through and prints all environment variables that were set during that build's execution.

• Security Warning: Be cautious when printing environment variables, as they may contain sensitive information (API keys, passwords, etc.). Only do this in secure environments and ensure proper access controls.

Analyzing Stage-Specific Failures in Pipelines

For Jenkins Pipelines, knowing which stage failed is crucial. While the raw console output will show [Pipeline] stage markers, which help delineate stages, the CLI itself doesn't provide a structured way to query stage status directly as the UI (e.g., Blue Ocean) does.

Locating Stage Failures in Logs

When reviewing the console output, look for the last [Pipeline] stage entry before the error message or stack trace. This usually indicates the stage where the failure occurred.

jcli console MyPipelineJob 123 | less

Inside less, you can search for [Pipeline] stage and then scroll to find the error.

Re-running or Restarting Failed Builds

Once you've identified the root cause of a failure and applied a fix (e.g., pushed new code, updated a configuration), you'll want to re-run the build. The CLI offers a straightforward way to do this.

Re-running an Entire Build

To trigger a new build for a job:

jcli build MyPipelineJob

If your job accepts parameters, you can pass them using the -p flag:

jcli build MyPipelineJob -p BRANCH=feature/fix-bug -p BUILD_VERSION=1.0.1

• --wait (-s): Wait for the build to complete.
• --verbose (-v): Display progress and build logs.

jcli build MyPipelineJob -p BRANCH=master --wait --verbose

Restarting from a Specific Stage (Advanced)

The Jenkins CLI does not have a direct restart-stage command. Restarting a pipeline from a specific stage is primarily a feature of the Jenkins UI (often enabled by the "Pipeline Steps" plugin) or requires specific pipeline logic.

However, you can achieve a similar effect by designing your pipeline to accept parameters that allow skipping initial stages. For example:

// In your Jenkinsfile

parameters {
    booleanParam(name: 'SKIP_SETUP_STAGE', defaultValue: false, description: 'Skip the initial setup stage')
}

pipeline {
    agent any
    stages {
        stage('Setup') {
            when {
                expression { !params.SKIP_SETUP_STAGE }
            }
            steps {
                echo 'Running setup...'
                // ... setup steps ...
            }
        }
        stage('Build') {
            steps {
                echo 'Building application...'
                // ... build steps ...
            }
        }
        // ... other stages ...
    }
}

Then, you can trigger this parameterized build via CLI to skip the "Setup" stage:

jcli build MyPipelineJob -p SKIP_SETUP_STAGE=true

This approach requires forethought in your Jenkinsfile design but offers powerful control over pipeline execution via the CLI.

Advanced Troubleshooting with Groovy (via CLI)

The groovy and groovy-script commands allow you to execute arbitrary Groovy code on the Jenkins controller. This provides unparalleled access to Jenkins' internal API for deep inspection and manipulation.

Example: Getting Build Details

jcli groovy =
    'def job = Jenkins.instance.getItemByFullName("MyPipelineJob")
    def build = job.getBuildByNumber(123)

    if (build) {
        println "Build #${build.number} for ${job.name}"
        println "Status: ${build.result}"
        println "Duration: ${build.durationString}"
        println "Description: ${build.description ?: "N/A"}"
        println "Causes:"
        build.getCauses().each { cause ->
            println "  - ${cause.shortDescription}"
        }
    } else {
        println "Build 123 not found for MyPipelineJob"
    }'

This script retrieves comprehensive information about a specific build, which can be invaluable for understanding why a build failed, especially if the console log is less clear.

Executing Local Groovy Scripts

For more complex Groovy scripts, write them in a .groovy file and execute it via groovy-script:

# my_troubleshooting_script.groovy

def jobName = System.getenv('JOB_NAME') ?: 'MyPipelineJob'
def buildNumber = System.getenv('BUILD_NUMBER') ? Integer.parseInt(System.getenv('BUILD_NUMBER')) : 123

def job = Jenkins.instance.getItemByFullName(jobName)
def build = job?.getBuildByNumber(buildNumber)

if (build) {
    println "Build details for ${job.name} #${build.number}"
    println "Status: ${build.result}"
    build.getAction(hudson.model.ParametersAction.class)?.getParameters()?.each { p ->
        println "Param: ${p.name} = ${p.value}"
    }
} else {
    println "Job ${jobName} or Build ${buildNumber} not found."
}

Then run it:

JOB_NAME=MyPipelineJob BUILD_NUMBER=123 jcli groovy-script my_troubleshooting_script.groovy

This allows for version-controlling your troubleshooting tools.

Tips for Efficient Troubleshooting

• Be Specific: When using grep, refine your patterns. Look for specific error codes, unique messages, or timestamps.
• Context is Key: Always consider the surrounding log lines. Errors often have precursors or follow-up messages that provide more context.
• Pipeline Structure: Design your Jenkinsfiles with clear stage names and verbose logging within critical steps. This makes it easier to pinpoint issues.
• Use tee: When running CLI commands, you can pipe output to tee to both display it and save it to a file for later analysis.
  bash jcli console MyPipelineJob 123 | tee build_123_log.txt | grep -i error
• Jenkins System Logs: Remember that Jenkins itself has system logs (JENKINS_HOME/logs). Sometimes a build failure is due to a Jenkins system issue, not the pipeline code. You can access these via the UI (Manage Jenkins -> System Log) or directly on the Jenkins server filesystem.

Conclusion

The Jenkins CLI is an indispensable tool for administrators and developers alike, offering a fast and powerful way to interact with your Jenkins instance. By mastering commands for retrieving logs, inspecting environment variables (via Groovy), and efficiently triggering builds, you can significantly reduce the time spent diagnosing and resolving build failures. Integrate these CLI techniques into your daily workflow to maintain highly performant and reliable CI/CD pipelines.

Continue exploring the extensive Jenkins CLI command list (jcli help) and the power of Groovy scripting to unlock even more advanced automation and troubleshooting capabilities.