Problem Statement

One of the secondary MongoDB servers stopped working. When checked, the disk usage of the server was 100% and it was due to the sudden growth of WiredTigerLAS.wt.

Sample error

2020-04-27T05:54:06.382+0000 E STORAGE  [repl writer worker 5] WiredTiger error (28) [1587966846:382630][3908:0x7f50fba07700], 
file:WiredTigerLAS.wt, WT_SESSION.commit_transaction: __posix_file_write, 579: /mnt/mongo-replica/mongo/WiredTigerLAS.wt: 
handle-write: pwrite: failed to write 24576 bytes at offset 78434086912: No space left on device

Reason of the Mongo process crash

Mongod will crash if the size exceeds ‘wiredTigerMaxCacheOverflowFileSizeGB‘. https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-wiredtigermaxcacheoverflowfilesizegb

What is WiredTigerLAS.wt

WiredTigerLAS table is a temporary table and will be written to hold any uncommitted transaction that happens in the cluster due to many reasons like network blips, heavy IO wait in the shards servers, etc. And yes, data once written to this table won’t get cleared until the next Mongo service restart.

Solutions

  1. Set enableMajorityReadConcern: false in mongod.conf and restart the service. Monitor for few days.
  2. If you have multiple Arbiters, shut them down keep just one arbiter. This is best practice recommendation for PSA(Primary-Secondary-Arbiter) architecture. Read this for more info.
  3. Try upgrading Mongo to the latest version

One thought on “[Solved] MongoDB crashed due to disk full – WiredTigerLAS.wt”

Leave a Reply

Your email address will not be published. Required fields are marked *