sentinel是一个管理redis实例的工具，它可以实现对redis的监控、通知、自动故障转移。sentinel不断的检测redis实例是否可以正常工作，通过API向其他程序报告redis的状态，如果redis master不能工作，则会自动启动故障转移进程，将其中的一个slave提升为master，其他的slave重新设置新的master服务器。

Sentinel介绍

Redis Sentinel是一个分布式架构，包含若干个Sentinel节点和Redis数据节点，每个Sentinel节点会对数据节点和其余Sentinel节点进行监控，当发现节点不可达时，会对节点做下线标识.
如果被标识的是主节点，他还会选择和其他Sentinel节点进行“协商”，当大多数的Sentinel节点都认为主节点不可达时，他们会选举出一个Sentinel节点来完成自动故障转移工作，同时将这个变化通知给Redis应用方.
整个过程完全自动，不需要人工介入，所以可以很好解决Redis的高可用问题.

redis主从复制
Redis主从复制可将主节点数据同步给从节点，从节点此时有两个作用:

  • 一旦主节点宕机，从节点作为主节点的备份可以随时顶上来
  • 扩展主节点的读能力，分担主节点读压力

Redis Sentinel有以下几个功能:

  • 监控：Sentinel节点会定期检测Redis数据节点和其余Sentinel节点是否可达
  • 通知：Sentinel节点会将故障转移通知给应用方
  • 主节点故障转移：实现从节点晋升为主节点并维护后续正确的主从关系
  • 配置提供者：在Redis Sentinel结构中，客户端在初始化的时候连接的是Sentinel节点集合，从中获取主节点信息

环境：
　　　Centos 7.9.2009
　　　Redis version：5.0.13

分别有3个Sentinel节点，1个主节点，2个从节点组成一个Redis Sentinel
| hostname | ip | redis port | sentinel port |
| — | — | — | — |
| node1 | 10.21.248.7 | 16379 | 26379 |
| node2 | 10.21.248.179 | 16379 | 26379 |
| node3 | 10.21.108.47 | 16379 | 26379 |

安装Redis

1
2
3
4
5
6
$ mkdir -p /data/release
$ cd /data/release
$ wget -c 'https://download.redis.io/releases/redis-5.0.13.tar.gz'
$ tar -xf redis-5.0.13.tar.gz
$ cd /data/release/redis-5.0.13/
$ make && make install

部署Redis Sentinel

Node1 Redis配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
$ mkdir -p /data/redis/{conf,data/16379,logs,sentinel}
$ cd /data/redis/conf
$ cat redis-16379.conf 
daemonize yes
bind 10.21.248.7
masterauth "YWY0MGIx@201"
requirepass "YWY0MGIx@201"
replica-read-only yes
maxclients 10000
maxmemory 1gb
pidfile "/data/redis/redis_16379.pid"
port 16379
tcp-backlog 511
timeout 600
tcp-keepalive 0
loglevel verbose
logfile "/data/redis/logs/redis-16379.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/data/redis/data/16379"
replica-serve-stale-data yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

Node1 Sentinel配置
注意：sentinel myid每个节点都要唯一

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cd /data/redis/conf
$ cat sentinel-26379.conf 
port 26379
bind 10.21.248.7
daemonize yes
dir "/data/redis/sentinel"
logfile "/data/redis/logs/sentinel-26379.log"
sentinel myid 525e0e47f39d28d46b707401054b0680057c800a
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.21.248.7 16379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster YWY0MGIx@201
sentinel config-epoch mymaster 59
sentinel leader-epoch mymaster 59

Node1 启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ cd /data/redis
$ cat redis_sentinel.sh
#!/bin/bash
num="$1"
function start() {
  #redis
  /usr/local/bin/redis-server /data/redis/conf/redis-16379.conf 
  
  #sentinel
  /usr/local/bin/redis-sentinel /data/redis/conf/sentinel-26379.conf
}
function stop() {
  #redis
  /usr/local/bin/redis-cli -h 10.21.248.7 -p 16379 -a YWY0MGIx@201 shutdown
  #sentinel
  /usr/local/bin/redis-cli -h 10.21.248.7 -p 26379 -a YWY0MGIx@201 shutdown
}
function status() {
  ps -ef |grep -i 'redis-' | grep -iv 'grep'
}
case $num in
  start)
     start;;
  stop)
     stop;;
  restart)
     stop
     start;;
  status)
     status;;
  *)
     echo "Usage: start|stop|restart|status";;
esac

Node2 Redis配置

从节点需新增replicaof masterip 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
$ mkdir -p /data/redis/{conf,data/16379,logs,sentinel}
$ cd /data/redis/conf
$ cat redis-16379.conf
daemonize yes
bind 10.21.248.179
masterauth "YWY0MGIx@201"
requirepass "YWY0MGIx@201"
replica-read-only yes
maxclients 10000
maxmemory 1gb
pidfile "/data/redis/redis_16379.pid"
port 16379
tcp-backlog 511
timeout 600
tcp-keepalive 0
loglevel verbose
logfile "/data/redis/logs/redis-16379.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/data/redis/data/16379"
replica-serve-stale-data yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
# Generated by CONFIG REWRITE
replicaof 10.21.248.7 16379

Node2 Sentinel配置
注意：sentinel myid每个节点都要唯一

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cd /data/redis/conf
$ cat sentinel-26379.conf  
port 26379
bind 10.21.248.179
daemonize yes
dir "/data/redis/sentinel"
logfile "/data/redis/logs/sentinel-26379.log"
sentinel myid 525e0e47f39d28d46b707401054b0680057c800b
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.21.248.7 16379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster YWY0MGIx@201
sentinel config-epoch mymaster 59
sentinel leader-epoch mymaster 59

Node2 启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ cd /data/redis
$ cat redis_sentinel.sh 
#!/bin/bash
num="$1"
function start() {
  #redis
  /usr/local/bin/redis-server /data/redis/conf/redis-16379.conf 
  
  #sentinel
  /usr/local/bin/redis-sentinel /data/redis/conf/sentinel-26379.conf
}
function stop() {
  #redis
  /usr/local/bin/redis-cli -h 10.21.248.179 -p 16379 -a YWY0MGIx@201 shutdown
  #sentinel
  /usr/local/bin/redis-cli -h 10.21.248.179 -p 26379 -a YWY0MGIx@201 shutdown
}
function status() {
  ps -ef |grep -i 'redis-' | grep -iv 'grep'
}
case $num in
  start)
     start;;
  stop)
     stop;;
  restart)
     stop
     start;;
  status)
     status;;
  *)
     echo "Usage: start|stop|restart|status";;
esac

Node3 Redis配置

从节点需新增replicaof masterip 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
$ mkdir -p /data/redis/{conf,data/16379,logs,sentinel}
$ cd /data/redis/conf
$ cat redis-16379.conf
daemonize yes
bind 10.21.108.47
masterauth "YWY0MGIx@201"
requirepass "YWY0MGIx@201"
replica-read-only yes
maxclients 10000
maxmemory 1gb
pidfile "/data/redis/redis_16379.pid"
port 16379
tcp-backlog 511
timeout 600
tcp-keepalive 0
loglevel verbose
logfile "/data/redis/logs/redis-16379.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/data/redis/data/16379"
replica-serve-stale-data yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
replicaof 10.21.248.7 16379

Node3 Sentinel配置
注意：sentinel myid每个节点都要唯一

1
2
3
4
5
6
7
8
9
10
11
12
13
$ cat sentinel-26379.conf 
port 26379
bind 10.21.108.47
daemonize yes
dir "/data/redis/sentinel"
logfile "/data/redis/logs/sentinel-26379.log"
sentinel myid 525e0e47f39d28d46b707401054b0680057c800c
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.21.248.7 16379 2
sentinel down-after-milliseconds mymaster 5000
sentinel auth-pass mymaster YWY0MGIx@201
sentinel config-epoch mymaster 59
sentinel leader-epoch mymaster 59

Node3 启动脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ cd /data/redis
$ cat redis_sentinel.sh
#!/bin/bash
num="$1"
function start() {
  #redis
  /usr/local/bin/redis-server /data/redis/conf/redis-16379.conf 
  
  #sentinel
  /usr/local/bin/redis-sentinel /data/redis/conf/sentinel-26379.conf
}
function stop() {
  #redis
  /usr/local/bin/redis-cli -h 10.21.108.47 -p 16379 -a YWY0MGIx@201 shutdown
  #sentinel
  /usr/local/bin/redis-cli -h 10.21.108.47 -p 26379 -a YWY0MGIx@201 shutdown
}
function status() {
  ps -ef |grep -i 'redis-' | grep -iv 'grep'
}
case $num in
  start)
     start;;
  stop)
     stop;;
  restart)
     stop
     start;;
  status)
     status;;
  *)
     echo "Usage: start|stop|restart|status";;
esac

启动各个节点的Redis Sentinel

1
2
$ cd /data/redis
$ sh redis_sentinel.sh start

查看Redis Sentinel日志

1
2
3
4
5
6
7
8
9
10
11
12
$ cd /data/redis/logs
$ cat sentinel-26379.log
9897:X 14 Aug 2022 21:41:04.503 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9897:X 14 Aug 2022 21:41:04.504 # Redis version=5.0.13, bits=64, commit=00000000, modified=0, pid=9897, just started
9897:X 14 Aug 2022 21:41:04.504 # Configuration loaded
9901:X 14 Aug 2022 21:41:04.505 * Running mode=sentinel, port=26379.
9901:X 14 Aug 2022 21:41:04.506 # Sentinel ID is 525e0e47f39d28d46b707401054b0680057c800a
9901:X 14 Aug 2022 21:41:04.506 # +monitor master mymaster 10.21.248.7 16379 quorum 2
9901:X 14 Aug 2022 21:41:09.032 * +sentinel sentinel 525e0e47f39d28d46b707401054b0680057c800b 10.21.248.179 26379 @ mymaster 10.21.248.7 16379
9901:X 14 Aug 2022 21:42:34.765 * +slave slave 10.21.248.179:16379 10.21.248.179 16379 @ mymaster 10.21.248.7 16379
9901:X 14 Aug 2022 21:42:34.766 * +slave slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.7 16379
9901:X 14 Aug 2022 21:42:35.889 * +sentinel sentinel 525e0e47f39d28d46b707401054b0680057c800c 10.21.108.47 26379 @ mymaster 10.21.248.7 16379

任意登陆一个节点的Redis Sentinel，可查看Redis Sentinel状态

1
2
3
4
$ redis-cli -h 10.21.248.7 -p 26379
10.21.248.7:26379> info sentinel  #查看redis sentinel状态
10.21.248.7:26379> sentinel masters  #查看sentinel masters信息
10.21.248.7:26379> sentinel slaves mymaster #查看sentinel slave信息

模拟stop master Sentinel

1
2
$ cd /data/redis
$ sh redis_sentinel.sh stop

node2的sentinel日志

1
2
3
4
5
6
7
8
9
10
11
$ cd /data/redis/logs
$ cat sentinel-26379.log
15383:X 14 Aug 2022 21:45:57.635 # +sdown master mymaster 10.21.248.7 16379
15383:X 14 Aug 2022 21:45:57.635 # +sdown sentinel 525e0e47f39d28d46b707401054b0680057c800a 10.21.248.7 26379 @ mymaster 10.21.248.7 16379
15383:X 14 Aug 2022 21:45:57.745 # +new-epoch 58
15383:X 14 Aug 2022 21:45:57.756 # +vote-for-leader 525e0e47f39d28d46b707401054b0680057c800c 58
15383:X 14 Aug 2022 21:45:58.142 # +config-update-from sentinel 525e0e47f39d28d46b707401054b0680057c800c 10.21.108.47 26379 @ mymaster 10.21.248.7 16379
15383:X 14 Aug 2022 21:45:58.142 # +switch-master mymaster 10.21.248.7 16379 10.21.248.179 16379
15383:X 14 Aug 2022 21:45:58.142 * +slave slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.179 16379
15383:X 14 Aug 2022 21:45:58.142 * +slave slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
15383:X 14 Aug 2022 21:46:03.189 # +sdown slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379

启动旧redis master，则自动以从节点的身份加入到redis sentinel

1
2
3
4
5
$ cd /data/redis/logs
$ cat sentinel-26379.log
323096:X 14 Aug 2022 21:47:09.917 # -sdown slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:47:10.684 # -sdown sentinel 525e0e47f39d28d46b707401054b0680057c800a 10.21.248.7 26379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:47:19.833 * +convert-to-slave slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379

模拟master redis sentinel服务器宕机

关机/重启命令略
node3 sentinel日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ cd /data/redis/logs
$ cat sentinel-26379.log
323096:X 14 Aug 2022 21:48:45.033 # +sdown sentinel 525e0e47f39d28d46b707401054b0680057c800b 10.21.248.179 26379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.641 # +sdown master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.693 # +odown master mymaster 10.21.248.179 16379 #quorum 2/2
323096:X 14 Aug 2022 21:48:45.693 # +new-epoch 59
323096:X 14 Aug 2022 21:48:45.693 # +try-failover master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.694 # +vote-for-leader 525e0e47f39d28d46b707401054b0680057c800c 59
323096:X 14 Aug 2022 21:48:45.696 # 525e0e47f39d28d46b707401054b0680057c800a voted for 525e0e47f39d28d46b707401054b0680057c800c 59
323096:X 14 Aug 2022 21:48:45.795 # +elected-leader master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.795 # +failover-state-select-slave master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.848 # +selected-slave slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.848 * +failover-state-send-slaveof-noone slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:45.907 * +failover-state-wait-promotion slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:46.708 # +promoted-slave slave 10.21.248.7:16379 10.21.248.7 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:46.709 # +failover-state-reconf-slaves master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:46.807 * +slave-reconf-sent slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:47.737 * +slave-reconf-inprog slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:47.737 * +slave-reconf-done slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:47.837 # -odown master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:47.837 # +failover-end master mymaster 10.21.248.179 16379
323096:X 14 Aug 2022 21:48:47.838 # +switch-master mymaster 10.21.248.179 16379 10.21.248.7 16379
323096:X 14 Aug 2022 21:48:47.838 * +slave slave 10.21.108.47:16379 10.21.108.47 16379 @ mymaster 10.21.248.7 16379
323096:X 14 Aug 2022 21:48:47.838 * +slave slave 10.21.248.179:16379 10.21.248.179 16379 @ mymaster 10.21.248.7 16379
323096:X 14 Aug 2022 21:48:52.858 # +sdown slave 10.21.248.179:16379 10.21.248.179 16379 @ mymaster 10.21.248.7 16379

启动旧redis master，则自动以从节点的身份加入到redis sentinel

1
2
3
4
$ cd /data/redis/logs
$ cat sentinel-26379.log
323096:X 14 Aug 2022 21:50:40.487 # -sdown sentinel 525e0e47f39d28d46b707401054b0680057c800b 10.21.248.179 26379 @ mymaster 10.21.248.7 16379
323096:X 14 Aug 2022 21:50:40.965 # -sdown slave 10.21.248.179:16379 10.21.248.179 16379 @ mymaster 10.21.248.7 16379

本文出自”Jack Wang Blog”：http://www.yfshare.vip/2022/08/27/Redis-Sentinel/