Powerful Looping Strategies: Iterating Files and Lists in Bash Scripts

Bash loops are where small shell commands become useful automation. Whether you need to process every file in a directory, perform a task a set number of times, or read configuration data line by line, loops give you the structure to repeat work without copy-pasting commands.

The two loops you will use most are for and while. Use for when you already have a known set of items, such as an array or a file glob. Use while when the loop is driven by a condition or by reading input. That simple split keeps many scripts easier to reason about.

---

The for Loop: Iterating Over Fixed Sets

The for loop is ideal when you know the collection of items you need to process in advance. This collection can be an explicit list of values, the results of a command, or a set of files found via globbing.

1. Iterating Over Standard Lists

The simplest use case is iterating over a short list of words written directly in the script.

Syntax

for VARIABLE in LIST_OF_ITEMS; do
    # Commands using $VARIABLE
done

Example: Processing a List of Users

# List of users to process
USERS="alice bob charlie"

for user in $USERS; do
  echo "Checking home directory for $user..."
  if [ -d "/home/$user" ]; then
    echo "$user is active."
  else
    echo "Warning: $user home directory missing."
  fi
done

That pattern is fine for simple names. If an item can contain spaces, use an array instead of a space-separated string:

USERS=("alice" "bob" "mary jane")

for user in "${USERS[@]}"; do
  echo "Checking $user"
done

2. C-Style Numeric Iteration

For tasks that require counting or specific numeric sequences, Bash supports a C-style for loop, often combined with brace expansion or the seq command.

Syntax (C-Style)

for (( INITIALIZATION; CONDITION; INCREMENT )); do
    # Commands
done

Example: Countdown Script

# Loop 5 times (i starts at 1, continues while i is less than or equal to 5)
for (( i=1; i<=5; i++ )); do
  echo "Iteration number: $i"
  sleep 1
done
echo "Done!"

Alternative: Using Brace Expansion for Simple Sequences

Brace expansion is simpler and faster than using seq for generating contiguous integers or sequences.

# Generates numbers from 10 to 1
for num in {10..1}; do
  echo "Counting down: $num"
done

3. Iterating Over Files and Directories (Globbing)

Using wildcards (*) within the for loop allows you to process files that match a specific pattern, such as all log files or all scripts in a directory.

Example: Archiving Log Files

Quote the variable ("$file") when dealing with filenames, especially those containing spaces or special characters.

TARGET_DIR="/var/log/application"

# Loop over all files ending in .log in the target directory
for logfile in "$TARGET_DIR"/*.log; do

  # Check if a file actually exists (prevents running on literal "*.log" if no files match)
  if [ -f "$logfile" ]; then
    echo "Compressing $logfile..."
    gzip "$logfile"
  fi
done

The while Loop: Condition-Based Execution

The while loop continues executing a block of commands as long as a specified condition remains true. It is commonly used for reading input streams, monitoring conditions, or handling tasks where the number of iterations is unknown.

1. Basic while Loop

Syntax

while CONDITION; do
    # Commands
done

Example: Waiting for a Resource

This loop uses the test command ([ ]) to check if a directory exists before proceeding.

RESOURCE_PATH="/mnt/data/share"

while [ ! -d "$RESOURCE_PATH" ]; do
  echo "Waiting for resource $RESOURCE_PATH to be mounted..."
  sleep 5
done

echo "Resource is available. Starting backup."

2. The Robust while read Pattern

The most powerful application of the while loop is reading the contents of a file or output stream line by line. This pattern is far superior to using a for loop on the output of cat, as it reliably handles spaces and special characters.

Best Practice: Reading Line-by-Line

To ensure maximum robustness, we utilize three key components:

1. IFS=: Clears the Internal Field Separator, ensuring the entire line, including leading/trailing spaces, is read into the variable.
2. read -r: The -r option prevents backslash interpretation (raw reading), which is critical for paths and complex strings.
3. Input Redirection (<): Redirects the file content into the loop, ensuring the loop runs in the current shell context (preventing subshell issues).

# File containing data, one item per line
CONFIG_FILE="/etc/app/servers.txt"

while IFS= read -r server_name; do
  
  # Skip empty lines or commented lines
  if [[ -z "$server_name" || "$server_name" =~ ^# ]]; then
    continue
  fi

  echo "Pinging server: $server_name"
  ping -c 1 "$server_name"

done < "$CONFIG_FILE"

Tip: Avoiding cat in Loops

Prefer while ... done < file over cat file | while ... when reading a file. In most Bash configurations, a pipeline runs the loop in a subshell, so variables changed inside the loop are lost when the loop finishes.

3. Handling Filenames from find

For recursive file processing, avoid parsing plain find output line by line. Filenames can contain spaces and, rarely, newlines. Use null-delimited output:

find /var/log/application -type f -name '*.log' -print0 |
while IFS= read -r -d '' logfile; do
  echo "Found log: $logfile"
  gzip -- "$logfile"
done

The -print0 and read -d '' pair treats the null byte as the separator. The -- before "$logfile" tells gzip that following values are operands, not options, which protects you from filenames beginning with -.

Advanced Looping Control and Techniques

Effective scripts require the ability to control loop execution based on runtime conditions.

1. Controlling Flow: break and continue

• break: Immediately exits the entire loop, regardless of remaining iterations or conditions.
• continue: Skips the current iteration and immediately jumps to the next iteration (or re-evaluates the while condition).

Example: Search and Stop

SEARCH_TARGET="target.conf"

for file in /etc/*; do
  if [ -f "$file" ] && [[ "$file" == *"$SEARCH_TARGET"* ]]; then
    echo "Found target configuration at: $file"
    break  # Stop processing once found
  elif [ -d "$file" ]; then
    continue # Skip directories, only check files
  fi
  echo "Checking file: $file"
done

2. Handling Complex Delimiters using IFS

While reading files line-by-line requires clearing IFS, iterating over a list separated by a different character (like a comma) requires temporarily setting IFS.

CSV_DATA="data1,data2,data3,data4"
OLD_IFS=$IFS # Save the original IFS
IFS=','       # Set IFS to the comma character

for item in $CSV_DATA; do
  echo "Found item: $item"
done

IFS=$OLD_IFS # Restore the original IFS immediately after the loop

Warning: Global IFS Changes

Always save the original $IFS before modifying it within a script (e.g., OLD_IFS=$IFS). Failure to restore the original value can cause unpredictable behavior in subsequent commands.

Best Practices for Robust Bash Loops

Practice	Rationale
Always Quote Variables	Use "$variable" to prevent word splitting and glob expansion, especially in file iteration.
Use while IFS= read -r	The most reliable method for processing files line-by-line, handling spaces and special characters correctly.
Check for Existence	When using globbing (*.txt), always include a check (if [ -f "$file" ];) to ensure the loop doesn't process the literal pattern name if no files match.
Localize Variables	Use local keyword inside functions to prevent loop variables from accidentally overwriting global variables.
Use Built-ins Over External Commands	Use brace expansion ({1..10}) or C-style loops over spawning external commands like seq for performance.

A Practical Rule of Thumb

Use arrays for in-memory lists, globs for simple file sets, while IFS= read -r for line-oriented input, and null-delimited find output for recursive filename handling. Quote expansions by default. Add existence checks around globs. Keep break and continue for cases where they make the loop easier to read, not as a way to hide complicated control flow.

Most Bash loop bugs come from word splitting, unexpected filenames, or assuming input is cleaner than it is. If your loop handles spaces, empty lines, comments, and missing matches deliberately, it will survive real automation work.