Troubleshooting data being read from cold storage after warmup

There are few ways to warmup data to get faster queries:

The Problem

Sometimes, we will see that although the data is already warmed, some data are being read still from Storage (S3).

Possible solution

Sometimes when data are read from cold storage, it is an indication that cache eviction has kicked in. Cache (disk) usage can be checked by running SHOW CACHE.

If the disk usage (or one of the nodes) is getting close to the 80% limit, cache eviction starts to do its job and delete the least recently used data.

If you notice that the disk usage is often getting to 80% and reading cold queries is a problem, you should upgrade to an engine with more disk space should solve these cold read queries.