Redis 缓存过期淘汰策略

Redis 缓存过期淘汰策略

面试题

  • 生产生你们的 Redis 内存设置多少?
  • 如何配置、修改 Redis 的内存大小?
  • 如果内存满了你怎么办
  • Redis 清理内存的方式?定期删除和惰性删除了解过吗?
  • Redis 缓存淘汰策略有哪些?分别是多少?你用哪个?
  • Redis 的 LRU 了解过吗?请手写 LRU
  • LRU 和 LFU 算法的区别是什么?
    LRU means Least Recently Used
    LFU means Least Frequently Used

Redis 内存满了怎么办

  • Redis 默认内存多少?在哪里查看?如何设置修改?
    • 查看 Redis 最大占用内存,打开 Redis 配置文件,设置 maxmemory 参数,maxmemory 是 bytes 字节类型,注意转换。
      ############################## MEMORY MANAGEMENT ################################
      
      # Set a memory usage limit to the specified amount of bytes.
      # When the memory limit is reached Redis will try to remove keys
      # according to the eviction policy selected (see maxmemory-policy).
      #
      # If Redis can't remove keys according to the policy, or if the policy is
      # set to 'noeviction', Redis will start to reply with errors to commands
      # that would use more memory, like SET, LPUSH, and so on, and will continue
      # to reply to read-only commands like GET.
      #
      # This option is usually useful when using Redis as an LRU or LFU cache, or to
      # set a hard memory limit for an instance (using the 'noeviction' policy).
      #
      # WARNING: If you have replicas attached to an instance with maxmemory on,
      # the size of the output buffers needed to feed the replicas are subtracted
      # from the used memory count, so that network problems / resyncs will
      # not trigger a loop where keys are evicted, and in turn the output
      # buffer of replicas is full with DELs of keys evicted triggering the deletion
      # of more keys, and so forth until the database is completely emptied.
      #
      # In short... if you have replicas attached it is suggested that you set a lower
      # limit for maxmemory so that there is some free RAM on the system for replica
      # output buffers (but this is not needed if the policy is 'noeviction').
      #
      # maxmemory <bytes>
    • Redis 默认内存多少可以用?
      • 如果不设置最大内存大小或者设置最大内存大小为 0,在 64 位操作系统下不限制内存大小,在 32 位操作系统下最多使用 3GB 内存。
      • 注意,在 64bit 系统下,maxmemory 设置为 0 表示不限制 Redis 内存使用。
    • 一般生产上你如何配置? 一般推荐 Redis 设置内存为最大物理内存的 3/4
    • 如何修改 Redis 内存设置
      • 通过修改文件配置
        maxmemory 104857600
      • 通过命令修改
        127.0.0.1:6379> config set maxmemory 104857600
        OK
        127.0.0.1:6379> config get maxmemory
        1) "maxmemory"
        2) "104857600"
    • 什么命令查看 Redis 内存使用情况?
      • info memory
      • config get maxmemory
  • 真要打满了会怎样?如果 Redis 内存使用超过了设置的最大值会怎样? 将 maxmemory 设置为 1 byte。
    (error) OOM command not allowed when used memory > 'maxmemory'.
  • 结论
    • 设置了 maxmemory 的选项,假如 Redis 内存使用达到上线;
    • 没有加上过期时间就会导致数据写满 maxmemory,为避免出现类似情况,需要使用内存淘汰策略。

往 Redis 里写的数据是怎么没了?它如何删除的?

Redis 过期键的删除策略

如果一个键是过期的,那它到了过期时间之后是不是马上就从内存中被被删除呢?

如果回答 Yes,立即删除,你自己走还是面试官送你走?

如果不是,那过期后到底什么时候被删除呢?是个什么操作?

三种不同的删除策略

立即删除

Redis 不可能时时刻刻遍历所有被设置了生存时间的 Key,来检测数据是否已经到达过期时间,然后对它进行删除。

立即删除能保证内存中数据的最大新鲜度,因为它保证过期键值会在过期后马上被删除,其所占用的内存也会随之释放。但是立即删除对 CPU 是最不友好的。因为删除操作会占用 CPU 的时间,如果刚好碰上了 CPU 很忙的时候,比如正在做交集或排序等计算的时候,就会给 CPU 造成额外的压力,这会产生大量的性能消耗,同时也会影响数据的读取操作。

总结: 对 CPU 不友好,用处理器性能换取存储空间(拿时间换空间)

惰性删除

数据到达过期时间,不做处理,等下次访问该数据时,如果未过期,返回数据;发现已过期,删除,返回不存在。

惰性删除策略的缺点是,它对内存是最不友好的

如果一个键已经过期,而这个键又仍然保留在 Redis 中,那么只要这个过期键不被删除,它所占用的内存就不会释放。

在使用惰性删除策略时,如果数据库中有非常多的过期键,而这些过期键又恰好没有被访问到的话,那么它们也许永远也不会被删除(除非用户手动执行 FLUSHDB),我们甚至可以将这种情况看作是一种内存泄漏 —— 无用的垃圾数据占用了大量的内存,而服务器却不会自己去释放它们,这对于运行状态非常依赖于内存的 Redis 服务器来说,肯定不是一个好消息。

总结: 对 memory 不友好,用存储空间换取处理器性能(拿空间换时间)

开启惰性淘汰:lazyfree-lazy-eviction=yes

定期删除

定期删除策略是前两种策略的折中:定期删除策略每隔一段时间执行一次删除过期键操作并通过限制删除操作执行时长和频率来减少删除操作对 CPU 时间的影响。

周期性轮询 Redis 库中的时效性数据,采用随机抽取的策略,利用过期数据占比的方式控制删除频度

  • 特点一:CPU 性能占用设置有峰值,检测频度可自定义设置;
  • 特点二:内存压力不是很大,长期占用内存的冷数据会被持续清理。
  • 总结:周期性抽查存储空间(随机抽查,重点抽查)

举例:Redis 默认每隔 100ms 检查是否过期的 Key,有过期 Key 则删除。注意:Redis 不是每隔 100ms 将所有的key检查一次而是随机抽取进行检查(如果每隔 100ms,全部 Key 进行检查,Redis 直接进去 ICU)。因此,如果只采用定期删除策略,会导致看很多 Key 到时间没有删除。

定期删除策略的难点是确定删除操作执行的时长和频率:如果删除操作执行的太频繁或者执行的时间太长,定期删除策略就会退化成立即删除策略,以至于将 CPU 时间过多地消耗在删除过期键上面。如果删除操作执行得太少,或者执行的时间太短,定期删除策略又会和惰性删除策略一样,出现浪费内存的情况。因此采用定期删除策略的话,服务器必须根据情况,合理地设置删除操作的执行时长和执行频率

总结: 定期抽样 Key,判断是否过期;漏网之鱼。

潜在问题

  • 定期删除时,从来没有被抽查到
  • 惰性删除时,也从来没有被点中使用过

上述两个潜在问题,会导致大量过期的 Key 堆积在内存中,导致 Redis 内存空间紧张或者很快耗尽

⭐️ Redis 缓存淘汰策略

Redis 配置文件

############################## MEMORY MANAGEMENT ################################

# Set a memory usage limit to the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys
# according to the eviction policy selected (see maxmemory-policy).
#
# If Redis can't remove keys according to the policy, or if the policy is
# set to 'noeviction', Redis will start to reply with errors to commands
# that would use more memory, like SET, LPUSH, and so on, and will continue
# to reply to read-only commands like GET.
#
# This option is usually useful when using Redis as an LRU or LFU cache, or to
# set a hard memory limit for an instance (using the 'noeviction' policy).
#
# WARNING: If you have replicas attached to an instance with maxmemory on,
# the size of the output buffers needed to feed the replicas are subtracted
# from the used memory count, so that network problems / resyncs will
# not trigger a loop where keys are evicted, and in turn the output
# buffer of replicas is full with DELs of keys evicted triggering the deletion
# of more keys, and so forth until the database is completely emptied.
#
# In short... if you have replicas attached it is suggested that you set a lower
# limit for maxmemory so that there is some free RAM on the system for replica
# output buffers (but this is not needed if the policy is 'noeviction').
#
# maxmemory <bytes>

# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select one from the following behaviors:
#
# volatile-lru -> Evict using approximated LRU, only keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU, only keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key having an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
#
# LRU means Least Recently Used
# LFU means Least Frequently Used
#
# Both LRU, LFU and volatile-ttl are implemented using approximated
# randomized algorithms.
#
# Note: with any of the above policies, when there are no suitable keys for
# eviction, Redis will return an error on write operations that require
# more memory. These are usually commands that create new keys, add data or
# modify existing keys. A few examples are: SET, INCR, HSET, LPUSH, SUNIONSTORE,
# SORT (due to the STORE argument), and EXEC (if the transaction includes any
# command that requires memory).
#
# The default is:
#
# maxmemory-policy noeviction

# LRU, LFU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. By default Redis will check five keys and pick the one that was
# used least recently, you can change the sample size using the following
# configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs more CPU. 3 is faster but not very accurate.
#
# maxmemory-samples 5

# Eviction processing is designed to function well with the default setting.
# If there is an unusually large amount of write traffic, this value may need to
# be increased.  Decreasing this value may reduce latency at the risk of
# eviction processing effectiveness
#   0 = minimum latency, 10 = default, 100 = process without regard to latency
#
# maxmemory-eviction-tenacity 10

# Starting from Redis 5, by default a replica will ignore its maxmemory setting
# (unless it is promoted to master after a failover or manually). It means
# that the eviction of keys will be just handled by the master, sending the
# DEL commands to the replica as keys evict in the master side.
#
# This behavior ensures that masters and replicas stay consistent, and is usually
# what you want, however if your replica is writable, or you want the replica
# to have a different memory setting, and you are sure all the writes performed
# to the replica are idempotent, then you may change this default (but be sure
# to understand what you are doing).
#
# Note that since the replica by default does not evict, it may end using more
# memory than the one set via maxmemory (there are certain buffers that may
# be larger on the replica, or data structures may sometimes take more memory
# and so forth). So make sure you monitor your replicas and make sure they
# have enough memory to never hit a real out-of-memory condition before the
# master hits the configured maxmemory setting.
#
# replica-ignore-maxmemory yes

# Redis reclaims expired keys in two ways: upon access when those keys are
# found to be expired, and also in background, in what is called the
# "active expire key". The key space is slowly and interactively scanned
# looking for expired keys to reclaim, so that it is possible to free memory
# of keys that are expired and will never be accessed again in a short time.
#
# The default effort of the expire cycle will try to avoid having more than
# ten percent of expired keys still in memory, and will try to avoid consuming
# more than 25% of total memory and to add latency to the system. However
# it is possible to increase the expire "effort" that is normally set to
# "1", to a greater value, up to the value "10". At its maximum value the
# system will use more CPU, longer cycles (and technically may introduce
# more latency), and will tolerate less already expired keys still present
# in the system. It's a tradeoff between memory, CPU and latency.
#
# active-expire-effort 1

LRU 和 LFU 算法的区别是什么

区别

LRU:最近最少使用页面置换算法,淘汰最长时间未被使用的页面,看页面最后一次被使用到发生调度的时间长短,首先淘汰最长时间未被使用的页面。

LFU:最近最不常用页面置换算法,淘汰一定时期内被访问次数最少的页,看一定时间段内页面被使用的频率,淘汰一定时期内被访问次数最少的页

举例

  • 某次时期 Time 为 10 分钟,如果每分钟进行一次调页,主存块为 3,若所需页面走向为 2 1 2 1 2 3 4
  • 假设到页面 4 时会发生缺页中断
  • 若按 LRU 算法,应换页面 1(1 页面最久未被使用),但按 LFU 算法应换页面 3(十分钟内,页面 3 只使用了一次)

可见 LRU 关键是看页面最后一次被使用到发生调度的时间长短,而 LFU 关键是看一定时间段内页面被使用的频率!

淘汰策略

  1. noeviction:不会驱逐任何 Key,表示即使内存达到上限也不进行置换,所有能引起内存增加的命令都会返回 error;
  2. allkeys-lru:对所有 Key 使用 LRU 算法进行删除,有限删除最近最不经常使用的 Key,用以保存新数据;
  3. volatile-lru:对所有设置了过期时间的 Key 使用 LRU 算法进行删除;
  4. allkeys-random:对所有 Key 随机删除;
  5. volatile-random:对所有设置了过期时间的 Key 随机删除;
  6. volatile-ttl:删除马上要过期的 Key;
  7. allkeys-lfu:对所有 Key 使用 LFU 算法进行删除;
  8. volatile-lfu:对所有设置了过期时间的 Key 使用 LFU 算法进行删除。

总结

2 * 4 = 8 个选项

  • 2 个 维度
    • 过期键中筛选
    • 所有键中筛选
  • 4 个方面
    • LRU
    • LFU
    • random
    • ttl

日常推荐使用

  • 在所有的 Key 都是最近最经常使用,那么就需要选择 allkeys-lru 进行置换最近最不经常使用的 Key,如果你不确定使用那种策略,那么推荐使用 allkeys-lru
  • 如果所有的 Key 的访问概率都差不多,那么可以选用 allkeys-random 策略去置换数据;
  • 如果对数据有足够的了解,能够为 Key 指定 hint(通过 expire/ttl 指定),那么可以选择 volatile-ttl 进行置换。

如何配置、修改

  • 直接使用 config 命令
  • 直接修改 redis.conf 配置文件

⭐️ Redis 缓存淘汰策略配置性能建议

  • 避免存储 BigKey
  • 开启惰性淘汰,lazyfree-lazy-eviction=yes