Nginx Log Monitoring: Key Commands for Analyzing Web Traffic and Errors

Nginx log monitoring is often the fastest way to answer the question people ask during an incident: "What is actually happening right now?" Metrics can tell you that 5xx errors rose. Logs can show the path, upstream, status code, client IP, and timing details for the requests that failed.

The commands here use ordinary Linux tools: tail, grep, awk, sort, uniq, less, and their compressed-log variants. They are not a replacement for a real log platform, but they are exactly what you need when you have SSH access to a server and need a quick, defensible answer.

Understanding Nginx Log Types

Nginx typically generates two primary types of logs, which are configured within the nginx.conf or associated configuration files:

1. Access Logs (access.log): Records every request processed by the server. This log is vital for understanding user behavior, traffic volume, geographic distribution, and response times. By default, fields often include IP address, request method, URI, HTTP status code, request size, and user agent.
2. Error Logs (error.log): Records diagnostic information, warnings, and critical errors encountered by Nginx itself (e.g., configuration issues, upstream timeouts, resource exhaustion). This log is the first stop for troubleshooting server-side failures.

Standard Log Locations

While locations can be customized, Nginx logs are typically found in the following directories on most distributions:

Distribution Type	Default Log Path
Debian/Ubuntu	/var/log/nginx/
RHEL/CentOS	/var/log/nginx/
Custom Install (Source)	Varies, check nginx.conf

We will use /var/log/nginx/access.log and /var/log/nginx/error.log as the primary examples.

---

1. Real-Time Monitoring with tail

The tail command is essential for watching current server activity as it happens. The -f (follow) flag keeps the output scrolling in real-time.

Monitoring Live Access Traffic

To view new requests coming into the server, use tail -f on the access log:

tail -f /var/log/nginx/access.log

Monitoring Errors Simultaneously

It is often helpful to monitor errors while testing configuration changes or deployments. You can do this by running two separate terminal sessions, or by using a tool like multitail (if installed):

tail -f /var/log/nginx/error.log

Tip: If you need to see the last 100 lines before following new entries, use tail -n 100 -f /var/log/nginx/access.log.

---

2. Searching and Filtering with grep

grep (Global Regular Expression Print) is the workhorse for finding specific lines within log files. It allows you to rapidly filter logs based on status codes, IP addresses, methods, and more.

Finding Specific HTTP Status Codes

When troubleshooting, quickly identifying all requests that resulted in a specific error is critical. We use spaces around the status code to prevent false positives from similar numbers (e.g., avoiding matching 200 in 2000).

Find all 404 (Not Found) errors:

grep " 404 " /var/log/nginx/access.log

Find all 5xx Server Errors:

grep -E " 50[0-9] " /var/log/nginx/access.log

Filtering by Request Path or IP Address

To see all requests made by a specific client IP address or all attempts to access a specific path (e.g., /admin):

# Filter by client IP address
grep "192.168.1.10" /var/log/nginx/access.log

# Filter for attempts to access a specific URL
grep "/wp-login.php" /var/log/nginx/access.log

Real-Time Filtering

You can pipe the output of tail -f into grep to monitor only specific events as they occur:

# Live feed of only 5xx errors
tail -f /var/log/nginx/access.log | grep -E " 50[0-9] "

---

3. Handling Large and Rotated Logs

Log files can become massive quickly. Nginx typically uses log rotation utilities (logrotate) to compress old logs using gzip.

Reviewing Large Files with less

Instead of loading the entire file into memory (which can crash a terminal session), use less to page through it. less also allows backward navigation and efficient searching.

less /var/log/nginx/access.log
# Inside less, press 'G' to go to the end, 'g' to go to the beginning, and '/' to search.

Searching Compressed Logs with zgrep

To search through rotated logs (.gz files) without manually uncompressing them, use the z variants of common commands (zcat, zgrep).

# Search for a 403 error in a compressed log file
zgrep " 403 " /var/log/nginx/access.log.1.gz

---

4. Structured Analysis with awk, cut, and sort

Nginx logs, especially those using the standard combined format, are structured by spaces. This structure allows tools like awk and cut to extract specific data fields for statistical analysis.

In the default combined format, key fields are typically:

• $1: Remote IP Address
• $7: Requested URI
• $9: HTTP Status Code
• $10: Bytes sent
• $12: HTTP Referer
• $14: User Agent

Finding the Most Requested Pages

This pipeline uses awk to extract the URI ($7), sort to group identical entries, uniq -c to count them, and sort -nr to list them numerically in reverse order (highest count first).

awk '{print $7}' /var/log/nginx/access.log | \
sort | uniq -c | sort -nr | head -10

Counting Status Codes

To quickly get a breakdown of all status codes recorded in the log:

awk '{print $9}' /var/log/nginx/access.log | \
sort | uniq -c | sort -nr

Example Output:

  1543 200
   321 301
   15 404
    2 500

Identifying High Latency Requests (If Logged)

If your Nginx configuration logs the upstream response time ($upstream_response_time), you can use awk to find slow requests (e.g., slower than 1 second).

Note: This assumes the response time is the 12th field ($12). Check your log_format configuration before trusting the field number.

awk '($12 > 1.0) {print $12, $7}' /var/log/nginx/access.log | sort -nr

---

Best Practices for Log Analysis

Use grep -v for Exclusion

Sometimes you need to filter out common noise, such as health checks or known benign bots. The -v flag in grep inverts the match, showing lines that do not match the pattern.

# View access logs, excluding all successful 200 responses
grep -v " 200 " /var/log/nginx/access.log

# View logs, excluding known Googlebot user agents
grep -v "Googlebot" /var/log/nginx/access.log

Compare Access Logs with Error Logs

When you see a spike in 502 or 504 responses, check the error log for the same time window. Access logs show the client-facing result. Error logs often explain the proxy-side reason:

grep "upstream" /var/log/nginx/error.log | tail -50
grep -E "connect\\(\\) failed|upstream timed out|no live upstreams" /var/log/nginx/error.log

For example, a 502 in the access log plus connect() failed (111: Connection refused) in the error log points toward an upstream service that is not accepting connections. A 504 plus upstream timed out points toward a slow upstream or timeout setting that is too low for that request path.

Be Careful with Field Numbers

Many examples assume Nginx's combined log format. Real production configs often add $request_time, $upstream_response_time, $host, $request_id, or forwarded IP fields. Before using an awk '{print $9}' command in a serious investigation, check the active format:

nginx -T 2>/dev/null | grep -A3 "log_format"

If your log format wraps the request in quotes, space-separated awk fields still work for simple checks, but they are easy to misread for user agents and referrers because those values contain spaces. For deeper analysis, ship logs to a parser or use a format such as JSON escaping in a dedicated access log.

Secure Handling

Nginx access logs contain sensitive data like IP addresses and potentially request parameters. Ensure that when transferring logs for analysis, you use secure protocols (SCP/SFTP) and restrict access to the log directory to authorized personnel (typically the root or syslog user).

# Check permissions
ls -l /var/log/nginx/

A Practical Workflow

Start with the symptom. If users report errors, count status codes and isolate the failing paths. If the server feels slow, look for request timing fields or add them to the log format before the next incident. If one IP is noisy, count top client addresses:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

Then move from broad to narrow: status code, path, upstream error, time window, client pattern. Keep the exact commands you used in the incident notes. They make the next review much easier, and they help another engineer reproduce your findings instead of relying on a vague memory of what the logs "looked like."

Nginx logs are plain text, which is a strength when the right tools are close at hand. Know your log format, avoid over-trusting copied field numbers, and pair access-log patterns with error-log messages before deciding what broke.