What is the method used in Redis key-value design?

Irregular phenomena in the use of Redis

• The key names stored in Redis are not standardized and are more arbitrary;

• Redis is used as a repository, there is a risk of data loss, and there is no reloading plan;

• Redis cache key, no expiration time is set, and the cache takes up low-frequency data A large amount of memory, which in turn causes the service to crash;

• Redis caches a large number of big keys, which will occupy a large amount of network bandwidth when the application is obtained, and deletion can also easily cause congestion;

• Improper use of the Redis client causes other client connection timeouts. The reason may be that the client password is incorrect and the connection pool is not used. A large number of connection retries lead to the exhaustion of system port resources;

• Improper use of Redis client commands leads to a large number of slow queries and affects other application services, such as using keys* or flushall commands during peak business periods;

Redis usage business Scenario recommendations and suggestions

• High concurrency scenarios: hotspot data caching can improve the overall system response speed and reduce database IO pressure;

• Limited time scenarios : Use the Redis expire command to set session expiration and renewal, mobile phone verification code, etc.;

• Using Redis's list and ordered set data structures can implement a variety of complex ranking applications

• Data set operations: Use Redis list, set, sorted set to facilitate data calculations, such as intersection, union, difference, etc.;

• Continuous sign-in: You can use the bitmap data structure of redis to implement sign-in related services;

• Counter: Use Redis incr and incrby commands to implement api call count statistics, api current limiting and other scenarios;

• Distributed lock: Use the setnx function of Redis to write distributed locks, typical open source components such as redisson;

How to design an elegant key

It can be said that when it comes to redis performance optimization online, unreasonable key design is often the root cause of the problem. In essence, from what I have seen personally, most When students use redis, they have almost no concept of key design, because the scenario most students use is key/val, and the corresponding data structure is string key/string val;

Students who have a deeper understanding of redis may know that the key design should be as short as possible when storing, and it is best to have a sense of hierarchy in the middle. It is best to divide it with:...

So how can we design a more elegant key? Let’s talk about it in detail based on the editor’s actual experience in use and the pitfalls that have been encountered;

1. Follow the following best practice conventions

1. Follow the basics Format: [Business Name]:[Data Name]:[id];

2. The length of the key shall not exceed 44 bytes;

3. No Contains special characters;

Regarding the above suggestions, this has the following advantages:

• It is highly readable, such as when we Design such a key structure, order:user:10. At a glance, you can tell that this is a key related to user orders;

• is convenient for maintenance and management, different applications, or different businesses Using different prefixes makes it easy to find and locate keys in visual client tools or command lines;

• avoid key conflicts and avoid multiple people using userId during use. Cache key conflict caused by value as key;

• Use string type as key, and the underlying encoding includes int, embstr and raw, which can effectively reduce memory usage. Using embstr can process strings smaller than 44 bytes with smaller memory usage because it uses continuous memory space

Recommended value:

• The value of a single key is less than 10KB;

• For collection type keys, it is recommended that the number of elements be less than 1000;

2. Try to avoid bigkey

1. What is bigkey

BigKey is usually judged comprehensively based on the size of the Key and the number of members in the Key, for example :

• The data volume of Key itself is too large: a Key of String type, its value is 5 MB;

• Members in Key Too many numbers: A ZSET type Key, its number of members is 10,000;

• The number of members in the Key is too large: A Hash type Key, its number of members is although There are only 1,000 but the total value size of these members is 100 MB;

2. The dangers of BigKey

Network Blocking

• When executing a read request for BigKey, a small amount of QPS may cause the bandwidth usage to be full, causing the Redis instance and even the physical machine where it is located to slow down;

Data skew

• The memory usage of the Redis instance where BigKey is located is much higher than that of other instances, and the memory resources of the data sharding cannot be reached. Balance;

Redis blocking

• Computing hash, list, zset, etc. with many elements will take a long time and cause the main thread to be blocked;

CPU Pressure

• Data serialization and deserialization of BigKey will cause CPU usage to soar, affecting Redis instances and other local applications;

3. How to discover BigKey

Execute the redis-cli --bigkeys command on the installed machine

• Using the --bigkeys parameter provided by redis-cli, you can traverse and analyze all keys, and return the overall statistical information of the Key and the top 1 big key of each data;

Scan by scan

• Write a program, use scan to scan all keys in Redis, and use strlen, hlen and other commands to determine the length of the key (MEMORY USAGE is not recommended here) );

Use third-party tools

• Use third-party tools, such as Redis-Rdb-Tools to analyze RDB snapshots file to comprehensively analyze memory usage;

Use network monitoring

• Custom tools to monitor the network in and out of Redis Data will proactively alarm when it exceeds the warning value;

3. Use appropriate data types

As mentioned above, many students who are using redis for the first time have many business scenarios. , all are done with a simple structure of key/val, without thinking deeply about whether it is reasonable to do so, or whether it will cause related performance problems in the future;

Regarding this issue, from Fundamentally speaking, we need to deeply understand and master the commonly used data types of redis. On this basis, we can design efficient storage structure data for different business scenarios;

Let us think about how to cache What about data like a list of user objects?

• Option 1: key is usrId, value is the serialized string of the object, and the data structure is similar to the following;

Advantages: Convenient access, simple and rough, you only need to convert json and objects when accessing;

Disadvantages: Data coupling, not flexible enough, once the object adds new fields Or the fields are deleted, and the cost of cache reconstruction is very high;

• Option 2: Use a list structure to cache the user ID list, the data structure is as follows;

Advantages: small memory usage and efficient operation;

Disadvantage: after obtaining val, further database search is required to obtain the complete object;

Option 3: Use hash structure, cache objects, and the data is as follows;

Advantages: The bottom layer uses ziplist, which takes up little space and can flexibly access any field of the object. ;

Disadvantages: Relatively complex coding;

Recommendations for using Redis cache in practical applications

• [Recommended] Warm up the cache. Before accessing data, the cache should be preheated to avoid a large number of requests directly entering the data storage layer; appropriate hot and cold data should be divided according to business conditions, and hot data should be preheated. Such as authorization information, apikey, etc.;

• [Recommended] Use local cache together. In a distributed architecture, although local caching can improve the stability and speed of data access, it needs to be used with caution to avoid introducing stateful server nodes. Avoid local caches from excessively occupying application server resources and causing application node crashes

• [Recommended] Cache change strategy, the database should be updated first, and then the cache;

• [Recommendation] A business call needs to access multiple redis servers, and pipeline or other batch operation methods can be used;

• [Recommendation] Large List, Set, Hash, storage The quantity is huge. When fetching a large number of elements, a large delay will occur, blocking the execution of other commands. It is recommended to split it into multiple small lists, sets or hash tables

Use business specifications

Whether it is redis or other intermediates used in development When developing and using software, it is best to formulate a set of reasonable specifications in advance. This specification should be recognized by most developers and tested in practice, and can effectively avoid some problems. Once designated as a specification, It should become a daily rule to guide internal developers. Here are the following points:

• Redis should be positioned as cache data and cannot be used to store large-scale data (cannot replace the database);

• Redis is suitable for scenarios where there is more reading and less writing. If there are high-frequency writes and low-frequency query scenarios, it is not recommended;

• When the key is not sure When it comes to the survival time, it is best to set the expiration time to control the life cycle of the key;

• Separation of hot and cold data should be considered. For queries, use Redis for high-frequency business queries, and consider using the database for low-frequency queries;

• When the program processes data, It should be considered that Redis has the risk of data loss, so it is necessary to automatically load and cache lost data from the database to Redis

• Use O(N) commands with caution, such as list, set, hash data When performing structure operations, hgetall, lrange, smembers, zrange, etc. are not unusable. Priority is given to using hscan, sscan, and zscan instead.

The above is the detailed content of What is the method used in Redis key-value design?. For more information, please follow other related articles on the PHP Chinese website!