Does the redis key still exist after it has expired?

Recently we discovered an interesting problem in the Redis cluster. After spending a lot of time debugging and testing, we were able to reduce Redis memory usage by 25% in some clusters by changing key expiration.

#Twitter runs multiple caching services internally. One of them is implemented by Redis. Our Redis cluster stores some important Twitter use case data, such as impression and engagement data, ad spend counts, and direct messages.

Question background (Recommended Learning: Redis Video Tutorial )

As early as 2016, Twitter's Cache team carried out the architecture of the Redis cluster Lots of updates. There have been some changes to Redis, including an update from Redis version 2.4 to version 3.2.

After this update, several issues emerged, such as users starting to see memory usage inconsistent with what they expected or prepared to use, increased latency, and key clearing issues. Key clearance is a big problem, which may cause the data that should be persisted to be deleted, or the request may be sent to the original storage of the data.

Preliminary Investigation

The affected teams and the caching team have begun a preliminary investigation. We are seeing increased latency related to the key clearing that is happening now. When Redis receives a write request but does not have the memory to hold the write, it stops what it is doing, clears the key and then saves the new key.

However, we still need to find out what is causing these newly cleared memory usage increases.

We suspect that the memory is full of expired but not yet deleted keys. Some people suggest using scanning. The scanning method will read all keys and delete expired keys.

In Redis, key has two expiration methods, active expiration and passive expiration. The scan will trigger passive expiration of the key. When the key is read, the TTL will be checked. If the TTL has expired, the TTL will be deleted and nothing will be returned.

The Redis documentation describes the active expiration of keys in version 3.2. The active expiration of a key begins with a function called activeExpireCycle. It runs several times per second on an internal timer called cron.

What the activeExpireCycle function does is iterate through each key space, check for a random kry with a TTL set, and if the percentage threshold of expired kry is met, repeat this process until the time limit is met.

This method of scanning all krys is effective, and when the scan is completed, the memory usage also goes down. It seems that Redis no longer effectively expires keys.

However, the solution at that time was to increase the size of the cluster and more hardware, so that the keys would be distributed more and there would be more available memory. This is disappointing because the previously mentioned project to upgrade Redis reduces the size and cost of running these clusters by making them more efficient.

For more Redis-related technical articles, please visit the Introduction to Using Redis Database Tutorial column to learn!

The above is the detailed content of Does the redis key still exist after it has expired?. For more information, please follow other related articles on the PHP Chinese website!