Powerful Looping Strategies: Iterating Files and Lists in Bash Scripts

Bash loops are the engine of automation in shell scripting. Whether you need to process every file in a directory, perform a task a set number of times, or read configuration data line by line, loops provide the necessary structure to handle repetitive operations efficiently.

Mastering the two primary Bash loop constructs—for and while—is essential for writing robust, scalable, and intelligent scripts. This guide explores the foundational syntax, practical use cases, and best practices for iterating over lists, files, directories, and generating sequences to supercharge your automation workflows.

The `for` Loop: Iterating Over Fixed Sets

The for loop is ideal when you know the collection of items you need to process in advance. This collection can be an explicit list of values, the results of a command, or a set of files found via globbing.

1. Iterating Over Standard Lists

The most common use case involves iterating over a space-separated list of items.

Syntax

for VARIABLE in LIST_OF_ITEMS; do
    # Commands using $VARIABLE
done

Example: Processing a List of Users

# List of users to process
USERS="alice bob charlie"

for user in $USERS; do
  echo "Checking home directory for $user..."
  if [ -d "/home/$user" ]; then
    echo "$user is active."
  else
    echo "Warning: $user home directory missing."
  fi
done

2. C-Style Numeric Iteration

For tasks that require counting or specific numeric sequences, Bash supports a C-style for loop, often combined with brace expansion or the seq command.

Syntax (C-Style)

for (( INITIALIZATION; CONDITION; INCREMENT )); do
    # Commands
done

Example: Countdown Script

# Loop 5 times (i starts at 1, continues while i is less than or equal to 5)
for (( i=1; i<=5; i++ )); do
  echo "Iteration number: $i"
  sleep 1
done
echo "Done!"

Alternative: Using Brace Expansion for Simple Sequences

Brace expansion is simpler and faster than using seq for generating contiguous integers or sequences.

# Generates numbers from 10 to 1
for num in {10..1}; do
  echo "Counting down: $num"
done

3. Iterating Over Files and Directories (Globbing)

Using wildcards (*) within the for loop allows you to process files that match a specific pattern, such as all log files or all scripts in a directory.

Example: Archiving Log Files

It is crucial to quote the variable ("$file") when dealing with filenames, especially those containing spaces or special characters.

TARGET_DIR="/var/log/application"

# Loop over all files ending in .log in the target directory
for logfile in "$TARGET_DIR"/*.log; do

  # Check if a file actually exists (prevents running on literal "*.log" if no files match)
  if [ -f "$logfile" ]; then
    echo "Compressing $logfile..."
    gzip "$logfile"
  fi
done

The `while` Loop: Condition-Based Execution

The while loop continues executing a block of commands as long as a specified condition remains true. It is commonly used for reading input streams, monitoring conditions, or handling tasks where the number of iterations is unknown.

1. Basic `while` Loop

Syntax

while CONDITION; do
    # Commands
done

Example: Waiting for a Resource

This loop uses the test command ([ ]) to check if a directory exists before proceeding.

RESOURCE_PATH="/mnt/data/share"

while [ ! -d "$RESOURCE_PATH" ]; do
  echo "Waiting for resource $RESOURCE_PATH to be mounted..."
  sleep 5
done

echo "Resource is available. Starting backup."

2. The Robust `while read` Pattern

The most powerful application of the while loop is reading the contents of a file or output stream line by line. This pattern is far superior to using a for loop on the output of cat, as it reliably handles spaces and special characters.

Best Practice: Reading Line-by-Line

To ensure maximum robustness, we utilize three key components:
1. IFS=: Clears the Internal Field Separator, ensuring the entire line, including leading/trailing spaces, is read into the variable.
2. read -r: The -r option prevents backslash interpretation (raw reading), which is critical for paths and complex strings.
3. Input Redirection (<): Redirects the file content into the loop, ensuring the loop runs in the current shell context (preventing subshell issues).

# File containing data, one item per line
CONFIG_FILE="/etc/app/servers.txt"

while IFS= read -r server_name; do

  # Skip empty lines or commented lines
  if [[ -z "$server_name" || "$server_name" =~ ^# ]]; then
    continue
  fi

  echo "Pinging server: $server_name"
  ping -c 1 "$server_name"

done < "$CONFIG_FILE"

Tip: Avoiding cat in Loops

Never use cat file | while read line; do... when reading files. Piping creates a subshell, meaning variables set within the loop are lost when the loop finishes. Use the input redirection pattern (while ... done < file) instead.

Advanced Looping Control and Techniques

Effective scripts require the ability to control loop execution based on runtime conditions.

1. Controlling Flow: `break` and `continue`

break: Immediately exits the entire loop, regardless of remaining iterations or conditions.
continue: Skips the current iteration and immediately jumps to the next iteration (or re-evaluates the while condition).

Example: Search and Stop

SEARCH_TARGET="target.conf"

for file in /etc/*; do
  if [ -f "$file" ] && [[ "$file" == *"$SEARCH_TARGET"* ]]; then
    echo "Found target configuration at: $file"
    break  # Stop processing once found
  elif [ -d "$file" ]; then
    continue # Skip directories, only check files
  fi
  echo "Checking file: $file"
done

2. Handling Complex Delimiters using `IFS`

While reading files line-by-line requires clearing IFS, iterating over a list separated by a different character (like a comma) requires temporarily setting IFS.

CSV_DATA="data1,data2,data3,data4"
OLD_IFS=$IFS # Save the original IFS
IFS=','       # Set IFS to the comma character

for item in $CSV_DATA; do
  echo "Found item: $item"
done

IFS=$OLD_IFS # Restore the original IFS immediately after the loop

Warning: Global IFS Changes

Always save the original $IFS before modifying it within a script (e.g., OLD_IFS=$IFS). Failure to restore the original value can cause unpredictable behavior in subsequent commands.

Best Practices for Robust Bash Loops

Practice	Rationale
Always Quote Variables	Use `"$variable"` to prevent word splitting and glob expansion, especially in file iteration.
Use `while IFS= read -r`	The most reliable method for processing files line-by-line, handling spaces and special characters correctly.
Check for Existence	When using globbing (`*.txt`), always include a check (`if [ -f "$file" ];`) to ensure the loop doesn't process the literal pattern name if no files match.
Localize Variables	Use `local` keyword inside functions to prevent loop variables from accidentally overwriting global variables.
Use Built-ins Over External Commands	Use brace expansion (`{1..10}`) or C-style loops over spawning external commands like `seq` for performance.

Conclusion

for and while loops are fundamental to Bash scripting, enabling complex automation tasks with minimal code repetition. By consistently applying robust patterns—such as the while IFS= read -r approach for file processing and diligent quoting in for loops—you can build scripts that are reliable, performant, and resistant to unexpected data formats, bringing true power to your shell automation efforts.

Powerful Looping Strategies: Iterating Files and Lists in Bash Scripts