MongoDB WiredTiger slow queries

Recently we hit production MongoDB (version 3.2.6) issue. MongoDB was reporting lots slow queries. Our application was starting to show performance issues. Some of the slow responses were for covered queries.

mongostat was reporting very high %used for WiredTiger cache and it was not coming down. As a result we were seeing significantly high value for  db.serverStatus().wiredTiger.cache[“pages evicted by application threads”]. This was causing slowdown of many queries. Ideally this value should be zero. This will happen if the cache % used hits 96%. Ideally it should be around 80%

Currently experimenting with WiredTiger eviction parameters to see if it makes any difference:

  • eviction_trigger
  • eviction_target
  • eviction_dirty_target
  • eviction=(threads_min=X,threads_max=Y)

It looks like the eviction server is not able to keep up with evicting pages and it gets into a state where application threads are evicting pages causing slowdown  😦

Solution:

We had thousands of collections in this database and 10’s of thousands of indices. Most of the collections were collection shards to work-around MMAPv1 collection lock contention. Before we sharded the collections one of them grew very big. 10’s of millions of entries. In this scenarios, depending on applications CRUD pattern, you can hit cache related issues. There are two solutions that worked for me with WiredTiger. Either to evenly balance the sharded collection or to consolidate the sharded collections.

MongoDB WiredTiger slow queries