Redis集群详解
Redis集群详解
Redis有三种集群模式,分别是:
*主从模式
*Sentinel模式
*Cluster模式
三种集群模式各有特点,关于Redis介绍可以参考这里:NoSQL(二)——Redis
Redis官网:https://redis.io/,最新版本6.0.5
主从模式
主从模式介绍
主从模式是三种模式中最简单的,在主从复制中,数据库分为两类:主数据库(master)和从数据库(slave)。
其中主从复制有如下特点:
*主数据库可以进行读写操作,当读写操作导致数据变化时会自动将数据同步给从数据库
*从数据库一般都是只读的,并且接收主数据库同步过来的数据
*一个master可以拥有多个slave,但是一个slave只能对应一个master
*slave挂了不影响其他slave的读和master的读和写,重新启动后会将数据从master同步过来
*master挂了以后,不影响slave的读,但redis不再提供写服务,master重启后redis将重新对外提供写服务
*master挂了以后,不会在slave节点中重新选一个master
工作机制:
当slave启动后,主动向master发送SYNC命令。master接收到SYNC命令后在后台保存快照(RDB持久化)和缓存保存快照这段时间的命令,然后将保存的快照文件和缓存的命令发送给slave。slave接收到快照文件和命令后加载快照文件和缓存的执行命令。
复制初始化后,master每次接收到的写命令都会同步发送给slave,保证主从数据一致性。
安全设置:
当master节点设置密码后,
客户端访问master需要密码
启动slave需要密码,在配置文件中配置即可
客户端访问slave不需要密码
缺点:
从上面可以看出,master节点在主从模式中唯一,若master挂掉,则redis无法对外提供写服务。
主从模式搭建
环境准备:
master节点192.168.30.128
slave节点192.168.30.129
slave节点192.168.30.130
全部下载安装:
#cd/software #wgethttp://download.redis.io/releases/redis-6.0.5.tar.gz #tarzxfredis-5.0.4.tar.gz&&mvredis-6.0.5//usr/local/redis #cd/usr/local/redis&&make&&makeinstall #echo$? 0
全部配置成服务:
服务文件
#vim/usr/lib/systemd/system/redis.service
[Unit]
Description=Redispersistentkey-valuedatabase
After=network.target
After=network-online.target
Wants=network-online.target[Service]
ExecStart=/usr/local/bin/redis-server/usr/local/redis/redis.conf--supervisedsystemd
ExecStop=/usr/libexec/redis-shutdown
Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755[Install]
WantedBy=multi-user.target
shutdown脚本
#vim/usr/libexec/redis-shutdown #!/bin/bash # #Wrappertocloseproperlyredisandsentinel testx"$REDIS_DEBUG"!=x&&set-x REDIS_CLI=/usr/local/bin/redis-cli #Retrieveservicename SERVICE_NAME="$1" if[-z"$SERVICE_NAME"];then SERVICE_NAME=redis fi #Gettheproperconfigfilebasedonservicename CONFIG_FILE="/usr/local/redis/$SERVICE_NAME.conf" #Useawktoretrievehost,portfromconfigfile HOST=`awk'/^[[:blank:]]*bind/{print$2}'$CONFIG_FILE|tail-n1` PORT=`awk'/^[[:blank:]]*port/{print$2}'$CONFIG_FILE|tail-n1` PASS=`awk'/^[[:blank:]]*requirepass/{print$2}'$CONFIG_FILE|tail-n1` SOCK=`awk'/^[[:blank:]]*unixsocket\s/{print$2}'$CONFIG_FILE|tail-n1` #Justincase,usedefaulthost,port HOST=${HOST:-127.0.0.1} if["$SERVICE_NAME"=redis];then PORT=${PORT:-6379} else PORT=${PORT:-26739} fi #Setupadditionalparameters #e.gpassword-protectedredisinstances [-z"$PASS"]||ADDITIONAL_PARAMS="-a$PASS" #shutdowntheserviceproperly if[-e"$SOCK"];then $REDIS_CLI-s$SOCK$ADDITIONAL_PARAMSshutdown else $REDIS_CLI-h$HOST-p$PORT$ADDITIONAL_PARAMSshutdown fi
#chmod+x/usr/libexec/redis-shutdown #useradd-s/sbin/nologinredis #chown-Rredis:redis/usr/local/redis #chown-Rreids:redis/data/redis #yuminstall-ybash-completion&&source/etc/profile#命令补全 #systemctldaemon-reload #systemctlenableredis
修改配置:
192.168.30.128
#mkdir-p/data/redis #vim/usr/local/redis/redis.conf bind192.168.30.128#监听ip,多个ip用空格分隔 daemonizeyes#允许后台启动 logfile"/usr/local/redis/redis.log"#日志路径 dir/data/redis#数据库备份文件存放目录 masterauth123456#slave连接master密码,master可省略 requirepass123456#设置master连接密码,slave可省略 appendonlyyes#在/data/redis/目录生成appendonly.aof文件,将每一次写操作请求都追加到appendonly.aof文件中 #echo'vm.overcommit_memory=1'>>/etc/sysctl.conf #sysctl-p
192.168.30.129
#mkdir-p/data/redis #vim/usr/local/redis/redis.conf bind192.168.30.129 daemonizeyes logfile"/usr/local/redis/redis.log" dir/data/redis replicaof192.168.30.1286379 masterauth123456 requirepass123456 appendonlyyes #echo'vm.overcommit_memory=1'>>/etc/sysctl.conf #sysctl-p
192.168.30.130
#mkdir-p/data/redis #vim/usr/local/redis/redis.conf bind192.168.30.130 daemonizeyes logfile"/usr/local/redis/redis.log" dir/data/redis replicaof192.168.30.1286379 masterauth123456 requirepass123456 appendonlyyes #echo'vm.overcommit_memory=1'>>/etc/sysctl.conf #sysctl-p
全部启动redis:
#systemctlstartredis
查看集群状态:
#redis-cli-h192.168.30.128-a123456 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. 192.168.30.128:6379>inforeplication #Replication role:master connected_slaves:2 slave0:ip=192.168.30.129,port=6379,state=online,offset=168,lag=1 slave1:ip=192.168.30.130,port=6379,state=online,offset=168,lag=1 master_replid:fb4941e02d5032ad74c6e2383211fc58963dbe90 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:168 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:168
#redis-cli-h192.168.30.129-a123456inforeplication Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. #Replication role:slave master_host:192.168.30.128 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 slave_repl_offset:196 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:fb4941e02d5032ad74c6e2383211fc58963dbe90 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:196 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:196
数据演示:
192.168.30.128:6379>keys* (emptylistorset) 192.168.30.128:6379>setkey1100 OK 192.168.30.128:6379>setkey2lzx OK 192.168.30.128:6379>keys* 1)"key1" 2)"key2"
#redis-cli-h192.168.30.129-a123456 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. 192.168.30.129:6379>keys* 1)"key2" 2)"key1" 192.168.30.129:6379>CONFIGGETdir 1)"dir" 2)"/data/redis" 192.168.30.129:6379>CONFIGGETdbfilename 1)"dbfilename" 2)"dump.rdb" 192.168.30.129:6379>getkey1 "100" 192.168.30.129:6379>getkey2 "lzx" 192.168.30.129:6379>setkey3aaa (error)READONLYYoucan'twriteagainstareadonlyreplica.
#redis-cli-h192.168.30.130-a123456 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. 192.168.30.130:6379>keys* 1)"key2" 2)"key1" 192.168.30.130:6379>CONFIGGETdir 1)"dir" 2)"/data/redis" 192.168.30.130:6379>CONFIGGETdbfilename 1)"dbfilename" 2)"dump.rdb" 192.168.30.130:6379>getkey1 "100" 192.168.30.130:6379>getkey2 "lzx" 192.168.30.130:6379>setkey3aaa (error)READONLYYoucan'twriteagainstareadonlyreplica.
可以看到,在master节点写入的数据,很快就同步到slave节点上,而且在slave节点上无法写入数据。
Sentinel模式
Sentinel模式介绍
主从模式的弊端就是不具备高可用性,当master挂掉以后,Redis将不能再对外提供写入操作,因此sentinel应运而生。
sentinel中文含义为哨兵,顾名思义,它的作用就是监控redis集群的运行状况,特点如下:
*sentinel模式是建立在主从模式的基础上,如果只有一个Redis节点,sentinel就没有任何意义
*当master挂了以后,sentinel会在slave中选择一个做为master,并修改它们的配置文件,其他slave的配置文件也会被修改,比如slaveof属性会指向新的master
*当master重新启动后,它将不再是master而是做为slave接收新的master的同步数据
*sentinel因为也是一个进程有挂掉的可能,所以sentinel也会启动多个形成一个sentinel集群
*多sentinel配置的时候,sentinel之间也会自动监控
*当主从模式配置密码时,sentinel也会同步将配置信息修改到配置文件中,不需要担心
*一个sentinel或sentinel集群可以管理多个主从Redis,多个sentinel也可以监控同一个redis
*sentinel最好不要和Redis部署在同一台机器,不然Redis的服务器挂了以后,sentinel也挂了
工作机制:
*每个sentinel以每秒钟一次的频率向它所知的master,slave以及其他sentinel实例发送一个PING命令
*如果一个实例距离最后一次有效回复PING命令的时间超过down-after-milliseconds选项所指定的值,则这个实例会被sentinel标记为主观下线。
*如果一个master被标记为主观下线,则正在监视这个master的所有sentinel要以每秒一次的频率确认master的确进入了主观下线状态
*当有足够数量的sentinel(大于等于配置文件指定的值)在指定的时间范围内确认master的确进入了主观下线状态,则master会被标记为客观下线
*在一般情况下,每个sentinel会以每10秒一次的频率向它已知的所有master,slave发送INFO命令
*当master被sentinel标记为客观下线时,sentinel向下线的master的所有slave发送INFO命令的频率会从10秒一次改为1秒一次
*若没有足够数量的sentinel同意master已经下线,master的客观下线状态就会被移除;
若master重新向sentinel的PING命令返回有效回复,master的主观下线状态就会被移除
当使用sentinel模式的时候,客户端就不要直接连接Redis,而是连接sentinel的ip和port,由sentinel来提供具体的可提供服务的Redis实现,这样当master节点挂掉以后,sentinel就会感知并将新的master节点提供给使用者。
Sentinel模式搭建
环境准备:
master节点192.168.30.128sentinel端口:26379
slave节点192.168.30.129sentinel端口:26379
slave节点192.168.30.130sentinel端口:26379
修改配置:
前面已经下载安装了redis,这里省略,直接修改sentinel配置文件。
192.168.30.128
#vim/usr/local/redis/sentinel.conf
daemonizeyes
logfile"/usr/local/redis/sentinel.log"
dir"/usr/local/redis/sentinel" #sentinel工作目录
sentinelmonitormymaster192.168.30.12863792 #判断master失效至少需要2个sentinel同意,建议设置为n/2+1,n为sentinel个数
sentinelauth-passmymaster123456
sentineldown-after-millisecondsmymaster30000 #判断master主观下线时间,默认30s
这里需要注意,sentinelauth-passmymaster123456需要配置在sentinelmonitormymaster192.168.30.12863792下面,否则启动报错:
#/usr/local/bin/redis-sentinel/usr/local/redis/sentinel.conf ***FATALCONFIGFILEERROR*** Readingtheconfigurationfile,atline104 >>>'sentinelauth-passmymaster123456' Nosuchmasterwithspecifiedname.
全部启动sentinel:
#mkdir/usr/local/redis/sentinel&&chown-Rredis:redis/usr/local/redis
#/usr/local/bin/redis-sentinel/usr/local/redis/sentinel.conf
任一主机查看日志:
#tail-f/usr/local/redis/sentinel.log
21574:X09May201915:32:04.298#SentinelIDis30c417116a8edbab09708037366c4a7471beb770
21574:X09May201915:32:04.298#+monitormastermymaster192.168.30.1286379quorum2
21574:X09May201915:32:04.299*+slaveslave192.168.30.129:6379192.168.30.1296379@mymaster192.168.30.1286379
21574:X09May201915:32:04.300*+slaveslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1286379
21574:X09May201915:32:16.347*+sentinelsentinel79b8d61626afd4d059fb5a6a63393e9a1374e78f192.168.30.12926379@mymaster192.168.30.1286379
21574:X09May201915:32:31.584*+sentinelsentineld7b429dcba792103ef0d80827dd0910bd9284d21192.168.30.13026379@mymaster192.168.30.1286379
Sentinel模式下的几个事件:
·+reset-master:主服务器已被重置。
·+slave:一个新的从服务器已经被Sentinel识别并关联。
·+failover-state-reconf-slaves:故障转移状态切换到了reconf-slaves状态。
·+failover-detected:另一个Sentinel开始了一次故障转移操作,或者一个从服务器转换成了主服务器。
·+slave-reconf-sent:领头(leader)的Sentinel向实例发送了[SLAVEOF](/commands/slaveof.html)命令,为实例设置新的主服务器。
·+slave-reconf-inprog:实例正在将自己设置为指定主服务器的从服务器,但相应的同步过程仍未完成。
·+slave-reconf-done:从服务器已经成功完成对新主服务器的同步。
·-dup-sentinel:对给定主服务器进行监视的一个或多个Sentinel已经因为重复出现而被移除——当Sentinel实例重启的时候,就会出现这种情况。
·+sentinel:一个监视给定主服务器的新Sentinel已经被识别并添加。
·+sdown:给定的实例现在处于主观下线状态。
·-sdown:给定的实例已经不再处于主观下线状态。
·+odown:给定的实例现在处于客观下线状态。
·-odown:给定的实例已经不再处于客观下线状态。
·+new-epoch:当前的纪元(epoch)已经被更新。
·+try-failover:一个新的故障迁移操作正在执行中,等待被大多数Sentinel选中(waitingtobeelectedbythemajority)。
·+elected-leader:赢得指定纪元的选举,可以进行故障迁移操作了。
·+failover-state-select-slave:故障转移操作现在处于select-slave状态——Sentinel正在寻找可以升级为主服务器的从服务器。
·no-good-slave:Sentinel操作未能找到适合进行升级的从服务器。Sentinel会在一段时间之后再次尝试寻找合适的从服务器来进行升级,又或者直接放弃执行故障转移操作。
·selected-slave:Sentinel顺利找到适合进行升级的从服务器。
·failover-state-send-slaveof-noone:Sentinel正在将指定的从服务器升级为主服务器,等待升级功能完成。
·failover-end-for-timeout:故障转移因为超时而中止,不过最终所有从服务器都会开始复制新的主服务器(slaveswilleventuallybeconfiguredtoreplicatewiththenewmasteranyway)。
·failover-end:故障转移操作顺利完成。所有从服务器都开始复制新的主服务器了。
·+switch-master:配置变更,主服务器的IP和地址已经改变。这是绝大多数外部用户都关心的信息。
·+tilt:进入tilt模式。
·-tilt:退出tilt模式。
master宕机演示:
192.168.30.128
#systemctlstopredis
#tail-f/usr/local/redis/sentinel.log
22428:X09May201915:51:29.287#+sdownmastermymaster192.168.30.1286379
22428:X09May201915:51:29.371#+odownmastermymaster192.168.30.1286379#quorum2/2
22428:X09May201915:51:29.371#+new-epoch1
22428:X09May201915:51:29.371#+try-failovermastermymaster192.168.30.1286379
22428:X09May201915:51:29.385#+vote-for-leader30c417116a8edbab09708037366c4a7471beb7701
22428:X09May201915:51:29.403#d7b429dcba792103ef0d80827dd0910bd9284d21votedfor30c417116a8edbab09708037366c4a7471beb7701
22428:X09May201915:51:29.408#79b8d61626afd4d059fb5a6a63393e9a1374e78fvotedfor30c417116a8edbab09708037366c4a7471beb7701
22428:X09May201915:51:29.451#+elected-leadermastermymaster192.168.30.1286379
22428:X09May201915:51:29.451#+failover-state-select-slavemastermymaster192.168.30.1286379
22428:X09May201915:51:29.528#+selected-slaveslave192.168.30.129:6379192.168.30.1296379@mymaster192.168.30.1286379
22428:X09May201915:51:29.528*+failover-state-send-slaveof-nooneslave192.168.30.129:6379192.168.30.1296379@mymaster192.168.30.1286379
22428:X09May201915:51:29.594*+failover-state-wait-promotionslave192.168.30.129:6379192.168.30.1296379@mymaster192.168.30.1286379
22428:X09May201915:51:30.190#+promoted-slaveslave192.168.30.129:6379192.168.30.1296379@mymaster192.168.30.1286379
22428:X09May201915:51:30.190#+failover-state-reconf-slavesmastermymaster192.168.30.1286379
22428:X09May201915:51:30.258*+slave-reconf-sentslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1286379
22428:X09May201915:51:30.511#-odownmastermymaster192.168.30.1286379
22428:X09May201915:51:31.233*+slave-reconf-inprogslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1286379
22428:X09May201915:51:31.233*+slave-reconf-doneslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1286379
22428:X09May201915:51:31.297#+failover-endmastermymaster192.168.30.1286379
22428:X09May201915:51:31.297#+switch-mastermymaster192.168.30.1286379192.168.30.1296379
22428:X09May201915:51:31.298*+slaveslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1296379
22428:X09May201915:51:31.298*+slaveslave192.168.30.128:6379192.168.30.1286379@mymaster192.168.30.1296379
22428:X09May201915:52:31.307#+sdownslave192.168.30.128:6379192.168.30.1286379@mymaster192.168.30.1296379
从日志中可以看到,master已经从192.168.30.128转移到192.168.30.129上
192.168.30.129上查看集群信息
#/usr/local/bin/redis-cli-h192.168.30.129-p6379-a123456 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. 192.168.30.129:6379>inforeplication #Replication role:master connected_slaves:1 slave0:ip=192.168.30.130,port=6379,state=online,offset=291039,lag=1 master_replid:757aff269236ed2707ba584a86a40716c1c76d74 master_replid2:47a862fc0ff20362be29096ecdcca6d432070ee9 master_repl_offset:291182 second_repl_offset:248123 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:291182 192.168.30.129:6379>setkey4linux OK
当前集群中只有一个slave——192.168.30.130,master是192.168.30.129,且192.168.30.129具有写权限。
192.168.30.130上查看redis的配置文件也可以看到replicaof192.168.30.1296379,这是sentinel在选举master是做的修改。
重新把192.168.30.128上进程启动
#systemctlstartredis
#tail-f/usr/local/redis/sentinel.log
22428:X09May201915:51:31.297#+switch-mastermymaster192.168.30.1286379192.168.30.1296379
22428:X09May201915:51:31.298*+slaveslave192.168.30.130:6379192.168.30.1306379@mymaster192.168.30.1296379
22428:X09May201915:51:31.298*+slaveslave192.168.30.128:6379192.168.30.1286379@mymaster192.168.30.1296379
22428:X09May201915:52:31.307#+sdownslave192.168.30.128:6379192.168.30.1286379@mymaster192.168.30.1296379
22428:X09May201916:01:24.872#-sdownslave192.168.30.128:6379192.168.30.1286379@mymaster192.168.30.1296379
查看集群信息
#/usr/local/bin/redis-cli-h192.168.30.128-p6379-a123456 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. 192.168.30.128:6379>inforeplication #Replication role:slave master_host:192.168.30.129 master_port:6379 master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 slave_repl_offset:514774 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:757aff269236ed2707ba584a86a40716c1c76d74 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:514774 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:376528 repl_backlog_histlen:138247 192.168.30.128:6379>getkey4 "linux" 192.168.30.128:6379>setkey5 (error)ERRwrongnumberofargumentsfor'set'command
即使192.168.30.128重新启动redis服务,也是作为slave加入redis集群,192.168.30.129仍然是master。
Cluster模式
Cluster模式介绍
sentinel模式基本可以满足一般生产的需求,具备高可用性。但是当数据量过大到一台服务器存放不下的情况时,主从模式或sentinel模式就不能满足需求了,这个时候需要对存储的数据进行分片,将数据存储到多个Redis实例中。cluster模式的出现就是为了解决单机Redis容量有限的问题,将Redis的数据根据一定的规则分配到多台机器。
cluster可以说是sentinel和主从模式的结合体,通过cluster可以实现主从和master重选功能,所以如果配置两个副本三个分片的话,就需要六个Redis实例。因为Redis的数据是根据一定规则分配到cluster的不同机器的,当数据量过大时,可以新增机器进行扩容。
使用集群,只需要将redis配置文件中的cluster-enable配置打开即可。每个集群中至少需要三个主数据库才能正常运行,新增节点非常方便。
cluster集群特点:
*多个redis节点网络互联,数据共享
*所有的节点都是一主一从(也可以是一主多从),其中从不提供服务,仅作为备用
*不支持同时处理多个key(如MSET/MGET),因为redis需要把key均匀分布在各个节点上,
并发量很高的情况下同时创建key-value会降低性能并导致不可预测的行为
*支持在线增加、删除节点*客户端可以连接任何一个主节点进行读写
Cluster模式搭建
环境准备:
三台机器,分别开启两个redis服务(端口)
192.168.30.128端口:7001,7002
192.168.30.129端口:7003,7004
192.168.30.130端口:7005,7006
修改配置文件:
192.168.30.128
#mkdir/usr/local/redis/cluster
#cp/usr/local/redis/redis.conf/usr/local/redis/cluster/redis_7001.conf
#cp/usr/local/redis/redis.conf/usr/local/redis/cluster/redis_7002.conf
#chown-Rredis:redis/usr/local/redis
#mkdir-p/data/redis/cluster/{redis_7001,redis_7002}&&chown-Rredis:redis/data/redis
#vim/usr/local/redis/cluster/redis_7001.conf bind192.168.30.128 port7001 daemonizeyes pidfile"/var/run/redis_7001.pid" logfile"/usr/local/redis/cluster/redis_7001.log" dir"/data/redis/cluster/redis_7001" #replicaof192.168.30.1296379 masterauth123456 requirepass123456 appendonlyyes cluster-enabledyes cluster-config-filenodes_7001.conf cluster-node-timeout15000
#vim/usr/local/redis/cluster/redis_7002.conf bind192.168.30.128 port7002 daemonizeyes pidfile"/var/run/redis_7002.pid" logfile"/usr/local/redis/cluster/redis_7002.log" dir"/data/redis/cluster/redis_7002" #replicaof192.168.30.1296379 masterauth"123456" requirepass"123456" appendonlyyes cluster-enabledyes cluster-config-filenodes_7002.conf cluster-node-timeout15000
其它两台机器配置与192.168.30.128一致,此处省略
启动redis服务:
#redis-server/usr/local/redis/cluster/redis_7001.conf
#tail-f/usr/local/redis/cluster/redis_7001.log
#redis-server/usr/local/redis/cluster/redis_7002.conf
#tail-f/usr/local/redis/cluster/redis_7002.log
其它两台机器启动与192.168.30.128一致,此处省略
安装ruby并创建集群(低版本):
如果redis版本比较低,则需要安装ruby。任选一台机器安装ruby即可
#yum-ygroupinstall"DevelopmentTools" #yuminstall-ygdbm-devellibdb4-devellibffi-devellibyamllibyaml-develncurses-developenssl-develreadline-develtcl-devel #mkdir-p~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS} #wgethttp://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz-P~/rpmbuild/SOURCES #wgethttp://raw.githubusercontent.com/tjinjin/automate-ruby-rpm/master/ruby22x.spec-P~/rpmbuild/SPECS #rpmbuild-bb~/rpmbuild/SPECS/ruby22x.spec #rpm-ivh~/rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.x86_64.rpm #geminstallredis#目的是安装这个,用于配置集群
#cp/usr/local/redis/src/redis-trib.rb/usr/bin/
#redis-trib.rbcreate--replicas1192.168.30.128:7001192.168.30.128:7002192.168.30.129:7003192.168.30.129:7004192.168.30.130:7005192.168.30.130:7006
创建集群:
我这里是redis6.0.5,所以不需要安装ruby,直接创建集群即可
#redis-cli-a123456--clustercreate192.168.30.128:7001192.168.30.128:7002192.168.30.129:7003192.168.30.129:7004192.168.30.130:7005192.168.30.130:7006--cluster-replicas1 Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe. >>>Performinghashslotsallocationon6nodes... Master[0]->Slots0-5460 Master[1]->Slots5461-10922 Master[2]->Slots10923-16383 Addingreplica192.168.30.129:7004to192.168.30.128:7001 Addingreplica192.168.30.130:7006to192.168.30.129:7003 Addingreplica192.168.30.128:7002to192.168.30.130:7005 M:80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001 slots:[0-5460](5461slots)master S:b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002 replicates6788453ee9a8d7f72b1d45a9093838efd0e501f1 M:4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003 slots:[5461-10922](5462slots)master S:b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004 replicates80c80a3f3e33872c047a8328ad579b9bea001ad8 M:6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005 slots:[10923-16383](5461slots)master S:277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006 replicates4d74ec66e898bf09006dac86d4928f9fad81f373 CanIsettheaboveconfiguration?(type'yes'toaccept):yes#输入yes,接受上面配置 >>>Nodesconfigurationupdated >>>Assignadifferentconfigepochtoeachnode >>>SendingCLUSTERMEETmessagestojointhecluster
可以看到,
192.168.30.128:7001是master,它的slave是192.168.30.129:7004;
192.168.30.129:7003是master,它的slave是192.168.30.130:7006;
192.168.30.130:7005是master,它的slave是192.168.30.128:7002
自动生成nodes.conf文件:
#ls/data/redis/cluster/redis_7001/ appendonly.aofdump.rdbnodes-7001.conf #vim/data/redis/cluster/redis_7001/nodes-7001.conf 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574544063125connected10923-16383 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574544070006connected b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574544083715connected 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001myself,master-015574544060001connected0-5460 b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574544073664connected 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574544070003connected5461-10922 varscurrentEpoch6lastVoteEpoch0
集群操作
登录集群:
#redis-cli-c-h192.168.30.128-p7001-a123456#-c,使用集群方式登录
查看集群信息:
192.168.30.128:7001>CLUSTERINFO#集群状态
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:580
cluster_stats_messages_pong_sent:551
cluster_stats_messages_sent:1131
cluster_stats_messages_ping_received:546
cluster_stats_messages_pong_received:580
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1131
列出节点信息:
192.168.30.128:7001>CLUSTERNODES#列出节点信息 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574551760005connected10923-16383 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574551740006connected b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574551750005connected 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001myself,master-015574551750001connected0-5460 b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574551749894connected 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574551759953connected5461-10922
这里与nodes.conf文件内容相同
写入数据:
192.168.30.128:7001>setkey111aaa ->Redirectedtoslot[13680]locatedat192.168.30.130:7005#说明数据到了192.168.30.130:7005上 OK 192.168.30.130:7005>setkey222bbb ->Redirectedtoslot[2320]locatedat192.168.30.128:7001#说明数据到了192.168.30.128:7001上 OK 192.168.30.128:7001>setkey333ccc ->Redirectedtoslot[7472]locatedat192.168.30.129:7003#说明数据到了192.168.30.129:7003上 OK 192.168.30.129:7003>getkey111 ->Redirectedtoslot[13680]locatedat192.168.30.130:7005 "aaa" 192.168.30.130:7005>getkey333 ->Redirectedtoslot[7472]locatedat192.168.30.129:7003 "ccc" 192.168.30.129:7003>
可以看出rediscluster集群是去中心化的,每个节点都是平等的,连接哪个节点都可以获取和设置数据。
当然,平等指的是master节点,因为slave节点根本不提供服务,只是作为对应master节点的一个备份。
增加节点:
192.168.30.129上增加一节点:
#cp/usr/local/redis/cluster/redis_7003.conf/usr/local/redis/cluster/redis_7007.conf #vim/usr/local/redis/cluster/redis_7007.conf bind192.168.30.129 port7007 daemonizeyes pidfile"/var/run/redis_7007.pid" logfile"/usr/local/redis/cluster/redis_7007.log" dir"/data/redis/cluster/redis_7007" #replicaof192.168.30.1296379 masterauth"123456" requirepass"123456" appendonlyyes cluster-enabledyes cluster-config-filenodes_7007.conf cluster-node-timeout15000 #mkdir/data/redis/cluster/redis_7007 #chown-Rredis:redis/usr/local/redis&&chown-Rredis:redis/data/redis #redis-server/usr/local/redis/cluster/redis_7007.conf
192.168.30.130上增加一节点:
#cp/usr/local/redis/cluster/redis_7005.conf/usr/local/redis/cluster/redis_7008.conf #vim/usr/local/redis/cluster/redis_7007.conf bind192.168.30.130 port7008 daemonizeyes pidfile"/var/run/redis_7008.pid" logfile"/usr/local/redis/cluster/redis_7008.log" dir"/data/redis/cluster/redis_7008" #replicaof192.168.30.1306379 masterauth"123456" requirepass"123456" appendonlyyes cluster-enabledyes cluster-config-filenodes_7008.conf cluster-node-timeout15000 #mkdir/data/redis/cluster/redis_7008 #chown-Rredis:redis/usr/local/redis&&chown-Rredis:redis/data/redis #redis-server/usr/local/redis/cluster/redis_7008.conf
集群中增加节点:
192.168.30.129:7003>CLUSTERMEET192.168.30.1297007 OK 192.168.30.129:7003>CLUSTERNODES 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003myself,master-015574573610003connected5461-10922 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574573647461connected0-5460 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574573620006connected b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574573630004connected b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574573620005connected e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574573627290connected 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574573637395connected10923-16383
192.168.30.129:7003>CLUSTERMEET192.168.30.1307008 OK 192.168.30.129:7003>CLUSTERNODES 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003myself,master-015574574890003connected5461-10922 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574574890001connected0-5460 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574574890006connected b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574574880004connected b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574574894725connected 1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008master-015574574892590connected e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574574890000connected 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574574904755connected10923-16383
可以看到,新增的节点都是以master身份加入集群的
更换节点身份:
将新增的192.168.30.130:7008节点身份改为192.168.30.129:7007的slave
#redis-cli-c-h192.168.30.130-p7008-a123456clusterreplicatee51ab166bc0f33026887bcf8eba0dff3d5b0bf14
clusterreplicate后面跟node_id,更改对应节点身份。也可以登入集群更改
#redis-cli-c-h192.168.30.130-p7008-a123456 192.168.30.130:7008>CLUSTERREPLICATEe51ab166bc0f33026887bcf8eba0dff3d5b0bf14 OK 192.168.30.130:7008>CLUSTERNODES 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574583168813connected 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574583148641connected0-5460 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574583160003connected5461-10922 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574583158725connected10923-16383 b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574583178905connected 1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574583150007connected b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574583150001connected e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574583140000connected
查看相应的nodes.conf文件,可以发现有更改,它记录当前集群的节点信息
#cat/data/redis/cluster/redis_7001/nodes-7001.conf
1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574582361697connected
6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574582350005connected10923-16383
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574582341036connected
b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574582351295connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001myself,master-015574582340001connected0-5460
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574582360004connected
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574582360000connected
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574582330893connected5461-10922
varscurrentEpoch7lastVoteEpoch0
删除节点:
192.168.30.130:7008>CLUSTERFORGET1a1c7f02fce87530bd5abdfc98df1cffce4f1767
(error)ERRItriedhardbutIcan'tforgetmyself...#无法删除登录节点192.168.30.130:7008>CLUSTERFORGETe51ab166bc0f33026887bcf8eba0dff3d5b0bf14
(error)ERRCan'tforgetmymaster!#不能删除自己的master节点192.168.30.130:7008>CLUSTERFORGET6788453ee9a8d7f72b1d45a9093838efd0e501f1
OK#可以删除其它的master节点192.168.30.130:7008>CLUSTERNODES
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574588873283connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574588870001connected0-5460
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574588860003connected5461-10922
b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave-015574588883515connected
1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574588850007connected
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574588832891connected
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574588853100connected192.168.30.130:7008>CLUSTERFORGETb4d3eb411a7355d4767c6c23b4df69fa183ef8bc
OK#可以删除其它的slave节点192.168.30.130:7008>CLUSTERNODES
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574590313973connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574590324071connected0-5460
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574590354343connected5461-10922
6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574590340005connected10923-16383
1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574590320007connected
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574590340001connected
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574590344270connected
保存配置:
192.168.30.130:7008>CLUSTERSAVECONFIG#将节点配置信息保存到硬盘
OK
#cat/data/redis/cluster/redis_7001/nodes-7001.conf1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574582361697connected
6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574582350005connected10923-16383
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574582341036connected
b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574582351295connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001myself,master-015574582340001connected0-5460
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574582360004connected
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574582360000connected
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574582330893connected5461-10922
varscurrentEpoch7lastVoteEpoch0#redis-cli-c-h192.168.30.130-p7008-a123456
Warning:Usingapasswordwith'-a'or'-u'optiononthecommandlineinterfacemaynotbesafe.192.168.30.130:7008>CLUSTERNODES
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574595007413connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master-015574595000001connected0-5460
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574595010003connected5461-10922
6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574595000005connected10923-16383
b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574594997375connected
1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574594990007connected
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004slave80c80a3f3e33872c047a8328ad579b9bea001ad8015574595017501connected
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574594980000connected
可以看到,之前删除的节点又恢复了,这是因为对应的配置文件没有删除,执行CLUSTERSAVECONFIG恢复。
模拟master节点挂掉:
192.168.30.128
#netstat-lntp|grep7001 tcp00192.168.30.128:170010.0.0.0:*LISTEN6701/redis-server1 tcp00192.168.30.128:70010.0.0.0:*LISTEN6701/redis-server1 #kill6701
192.168.30.130:7008>CLUSTERNODES
277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574611780003connected
80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001master,fail-155746095048315574609471451disconnected
4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574611749223connected5461-10922
6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574611810035connected10923-16383
b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574611799935connected
1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574611760007connected
b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004master-015574611789818connected0-5460
e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574611790000connected
对应7001的一行可以看到,masterfail,状态为disconnected;而对应7004的一行,slave已经变成master。
重新启动7001节点:
#redis-server/usr/local/redis/cluster/redis_7001.conf 192.168.30.130:7008>CLUSTERNODES 277daeb8660d5273b7c3e05c263f861ed5f17b92192.168.30.130:7006@17006slave4d74ec66e898bf09006dac86d4928f9fad81f373015574613070003connected 80c80a3f3e33872c047a8328ad579b9bea001ad8192.168.30.128:7001@17001slaveb6331cbc986794237c83ed2d5c30777c1551546e015574613054418connected 4d74ec66e898bf09006dac86d4928f9fad81f373192.168.30.129:7003@17003master-015574613079623connected5461-10922 6788453ee9a8d7f72b1d45a9093838efd0e501f1192.168.30.130:7005@17005master-015574613049355connected10923-16383 b4d3eb411a7355d4767c6c23b4df69fa183ef8bc192.168.30.128:7002@17002slave6788453ee9a8d7f72b1d45a9093838efd0e501f1015574613060005connected 1a1c7f02fce87530bd5abdfc98df1cffce4f1767192.168.30.130:7008@17008myself,slavee51ab166bc0f33026887bcf8eba0dff3d5b0bf14015574613050007connected b6331cbc986794237c83ed2d5c30777c1551546e192.168.30.129:7004@17004master-015574613089728connected0-5460 e51ab166bc0f33026887bcf8eba0dff3d5b0bf14192.168.30.129:7007@17007master-015574613070000connected
可以看到,7001节点启动后为slave节点,并且是7004的slave节点。即master节点如果挂掉,它的slave节点变为新master节点继续对外提供服务,而原来的master节点如果重启,则变为新master节点的slave节点。
另外,如果这里是拿7007节点做测试的话,会发现7008节点并不会切换,这是因为7007节点上根本没数据。集群数据被分为三份,采用哈希槽(hashslot)的方式来分配16384个slot的话,它们三个节点分别承担的slot区间是:
节点7004覆盖0-5460
节点7003覆盖5461-10922
节点7005覆盖10923-16383
更多参考:
Redis集群常用命令