Diagnosing and Resolving Common MongoDB Replication Lag Issues

MongoDB replication lag is not just a number on a dashboard. It changes how your application behaves. A user updates a profile, another request reads from a secondary, and the old value comes back. A node fails, but the best secondary is still behind, so failover takes longer than expected. A reporting query lands on the wrong member and suddenly the replica set looks healthy except for one secondary that keeps drifting away from the primary.

The useful way to think about replication lag is simple: the primary is producing oplog entries faster than one or more secondaries can fetch and apply them. The fix depends on which side of that sentence is true in your environment. Sometimes the primary is writing too much in bursts. Sometimes the secondary is underpowered. Sometimes the network is slow. Sometimes the lag is intentional because the member is configured with secondaryDelaySecs. Your first job is to separate those cases before making changes.

Start with the Actual Shape of the Lag

Do not begin by resizing the oplog or restarting mongod. First find out whether lag is steady, spiky, limited to one member, or affecting every secondary.

In mongosh, start with:

rs.status()

Look at each member's stateStr, optimeDate, lastHeartbeatMessage, and health fields. If one secondary is behind and the others are current, you probably have a member-specific issue: disk, CPU, local reads, local maintenance, or a bad network path. If every secondary is behind, look harder at primary write volume, network throughput out of the primary, or an unusually large operation.

For a quick oplog window check, run:

rs.printReplicationInfo()

The oplog window tells you how much time is covered by the current oplog. It does not say that replication is healthy. It says how far back a secondary can be before it risks needing an initial sync. If your oplog window is 6 hours and your maintenance windows routinely take 8 hours, you have a real operational risk even when current lag is zero.

For secondaries, this is also useful:

rs.printSecondaryReplicationInfo()

In older examples you may see rs.printSlaveReplicationInfo(). Newer wording uses "secondary", but older shell helpers and older blog posts may still use "slave". The fields matter more than the name.

If you want a small script for a live shell, compare the primary optime with each secondary:

const status = rs.status();
const primary = status.members.find(m => m.stateStr === "PRIMARY");

status.members
  .filter(m => m.stateStr === "SECONDARY")
  .forEach(m => {
    const lagSeconds = (primary.optimeDate - m.optimeDate) / 1000;
    print(`${m.name}: ${lagSeconds}s behind primary`);
  });

Treat that as a snapshot, not a diagnosis. A secondary that is 20 seconds behind during a batch import may be fine if it catches up quickly. A secondary that is always 20 seconds behind during normal traffic deserves attention.

Check Whether the Lag Is Intentional

Before chasing a false incident, inspect the replica set configuration:

rs.conf()

A delayed member is configured to trail the primary by design. In modern MongoDB configuration, look for secondaryDelaySecs on a member. That member is useful for some recovery scenarios because it can preserve an older view of data for a short period. It should not be used for fresh reads, and its expected delay should be excluded from normal lag alerts.

The mistake I see in real operations is alerting on every delayed member as if it were broken. Alert on delay beyond the configured delay. If a member is delayed by 1 hour and shows 1 hour and 5 minutes of lag, the real lag is about 5 minutes.

When the Oplog Window Is Too Small

The oplog is a capped collection in the local database. Secondaries read it and apply the operations in order. If a secondary falls behind far enough that the primary no longer has the oplog entries it needs, ordinary catch-up is no longer possible. The member usually needs an initial sync or a restore from a suitable backup.

This is why the oplog window matters. You want it to cover more than your expected downtime, maintenance, network interruption, and peak write bursts. There is no universal "correct" oplog size. A quiet cluster may keep days of history in a small oplog. A busy cluster with heavy updates may burn through the same size in a short period.

If the oplog window is shrinking during peak traffic, increase it before the next maintenance window. On supported MongoDB versions, use replSetResizeOplog rather than dropping and recreating local.oplog.rs. Dropping the oplog on a replica set member is a high-risk recovery maneuver, not a normal tuning step.

Run the resize command on the member whose oplog you want to resize:

use admin
db.adminCommand({ replSetResizeOplog: 1, size: 10240 })

The size value is in megabytes. A value of 10240 means roughly 10 GB. Resize each member as needed. In managed environments such as MongoDB Atlas, use the platform's supported configuration path instead of assuming direct filesystem or process control.

After resizing, verify the new window under real write load. A bigger oplog reduces the chance of falling off the oplog, but it does not make a slow secondary apply operations faster.

When One Secondary Is Slow

If only one secondary lags, log in to that host and look at the ordinary system symptoms. MongoDB is often blamed for what is really disk saturation.

Use tools such as:

iostat -xz 1
vmstat 1
top
mongostat --host secondary.example.com:27017
mongotop --host secondary.example.com:27017

High disk utilization, high await times, or a long I/O queue usually means the secondary cannot write fast enough. This can happen when a cheaper instance type is used for secondaries, when EBS or network storage has lower provisioned throughput, or when backups and filesystem snapshots run at the same time as peak application writes.

CPU can matter too, especially with compression, encryption, document moves, index maintenance, or a workload with many small updates. Memory pressure shows up as page faults, cache churn, and a secondary that keeps reading from disk while trying to apply oplog entries.

The practical fix is usually boring: give the secondary storage and CPU comparable to the primary, reduce competing work on that host, or move heavy reads somewhere else. A replica set member is not free reporting capacity. It still has to keep up with replication.

When Reads on Secondaries Cause the Problem

Read scaling with secondaries is useful, but it is easy to overdo. A dashboard query that scans a large collection can compete with oplog application. The secondary may still accept reads, but replication falls behind because the same CPU, cache, and disk are being used for user queries.

Check the profiler and current operations on the lagging member:

db.currentOp({ active: true })

If you see long reads, aggregation jobs, or maintenance scripts, decide whether that secondary should really serve that workload. For reporting, a hidden or dedicated secondary may be a better fit. For application reads, set maxStalenessSeconds so the driver avoids secondaries that are too far behind.

For consistency-critical paths, use primary reads. Examples include login state, checkout confirmation, password changes, account settings, and anything where a user expects to read their own write immediately. Secondary reads are best for data where brief staleness is acceptable.

When the Primary Produces Bursts

Large writes can make healthy secondaries look broken. Bulk imports, wide multi-document updates, TTL cleanup, large deletes, and index changes can produce a burst of oplog activity that takes time to apply.

Look for recent operations on the primary:

db.currentOp({ active: true })

Also check application deploys, data repair jobs, backfills, and scheduled tasks. Replication lag that starts at exactly 02:00 is often not mysterious. It is a batch job.

When you control the job, split it into smaller chunks. For example, update documents by _id ranges, pause between batches, and watch lag while the job runs. With bulkWrite, unordered writes can improve throughput, but error handling needs to be explicit because failures may be partial. The goal is not always to make the primary finish as fast as possible. The goal is to let the replica set absorb the work without losing its recovery margin.

Indexes and Oplog Application

In a normal replica set, indexes are replicated. If indexes differ between members because of manual work, failed maintenance, or a node that was restored incorrectly, a secondary can become painfully slow at applying updates and deletes. The oplog operation may need to find a document, and without the expected index the secondary can do much more work than the primary did.

Compare index definitions on the affected collections:

db.orders.getIndexes()

Run the same command on the primary and lagging secondary. If they differ, find out why before making more changes. Rebuilding a large index can itself create load, so plan it during a quiet period or rebuild the member from a known-good source if the divergence is broad.

Do not use old advice that says background index builds solve all replication concerns. MongoDB index build behavior has changed across versions, and the right operational choice depends on your version and topology. Use the current server documentation for the exact version you run.

Network Problems Are Usually Visible Somewhere Else

Network lag tends to show up as unstable heartbeats, intermittent errors, or poor throughput between specific hosts or regions. Basic checks still help:

ping primary.example.com
traceroute primary.example.com

But low ping latency does not prove enough bandwidth. Replication can be limited by throughput, packet loss, firewall inspection, cross-region links, or noisy shared networking. If lag appears only for a remote secondary, compare it with a secondary in the same region as the primary. If same-region members are fine and the remote member is behind, the topology may be asking too much of the link.

For cross-region replica sets, be honest about the tradeoff. They can help with disaster recovery, but they are more exposed to latency and bandwidth limits. If the remote member is meant for reads, use staleness controls and test failover behavior instead of assuming it will behave like a local secondary.

Be Careful with Restart and Resync Advice

Restarting mongod can clear a transient issue, but it can also make an incident worse if the node was close to falling off the oplog. Before a restart, check the oplog window and current lag. If the node needs two hours to catch up and the oplog window is only three hours during peak traffic, a long restart may leave you with an initial sync instead of a catch-up.

Initial sync is a valid repair option when a secondary is stale, corrupted, or missing required oplog history. It is also expensive. It copies data, builds indexes, and consumes network and disk resources from sync sources. In production, prefer adding or rebuilding one member at a time so the replica set keeps enough voting and data-bearing members to tolerate failures.

If a member is so far behind that it cannot catch up, take a fresh backup or snapshot-based path that matches your operational standards. Do not delete a data directory because a checklist says so. Confirm the member is disposable, confirm the replica set can tolerate the rebuild, and confirm you have enough oplog window or a reliable initial sync source.

Alert on What Users and Operators Care About

A good alert is not "replication lag is greater than 1 second" for every system. Some applications can tolerate 30 seconds on analytics reads. Others cannot tolerate stale reads on account state. Alert thresholds should reflect the use case.

Useful alerts include:

• Replication lag above the application tolerance for a sustained period.
• Oplog window below the longest expected maintenance or recovery interval.
• A secondary in RECOVERING, STARTUP2, or unhealthy state longer than expected.
• Disk I/O saturation on any data-bearing member.
• Heartbeat failures or network errors between members.

Dashboards should show lag next to write volume, disk latency, CPU, memory pressure, and network throughput. Lag by itself tells you there is a problem. The neighboring graphs usually tell you which problem.

A Practical Triage Order

When you are on call, use this order:

1. Confirm which members are lagging with rs.status().
2. Check whether any lag is intentional because of secondaryDelaySecs.
3. Check the oplog window with rs.printReplicationInfo().
4. Compare lag with write spikes, batch jobs, and recent deploys.
5. Inspect the lagging secondary's disk, CPU, memory, and local query load.
6. Check network errors and latency between the affected members.
7. Decide whether the member can catch up, needs load removed, needs more resources, or must be rebuilt.

The best outcome is usually not a dramatic command. It is finding the bottleneck and removing it without creating data divergence. MongoDB replication lag is manageable when you treat it as a capacity and topology signal, not as a generic MongoDB failure.