在RedisTemplate中使用scan代替keys指令操作
keys*这个命令千万别在生产环境乱用。特别是数据庞大的情况下。因为Keys会引发Redis锁,并且增加Redis的CPU占用。很多公司的运维都是禁止了这个命令的
当需要扫描key,匹配出自己需要的key时,可以使用scan命令
scan操作的Helper实现
importjava.io.IOException; importjava.nio.charset.StandardCharsets; importjava.util.ArrayList; importjava.util.List; importjava.util.function.Consumer; importorg.springframework.beans.factory.annotation.Autowired; importorg.springframework.data.redis.connection.RedisConnection; importorg.springframework.data.redis.core.Cursor; importorg.springframework.data.redis.core.ScanOptions; importorg.springframework.data.redis.core.StringRedisTemplate; importorg.springframework.stereotype.Component; @Component publicclassRedisHelper{ @Autowired privateStringRedisTemplatestringRedisTemplate; /** *scan实现 *@parampattern表达式 *@paramconsumer对迭代到的key进行操作 */ publicvoidscan(Stringpattern,Consumerconsumer){ this.stringRedisTemplate.execute((RedisConnectionconnection)->{ try(Cursor cursor=connection.scan(ScanOptions.scanOptions().count(Long.MAX_VALUE).match(pattern).build())){ cursor.forEachRemaining(consumer); returnnull; }catch(IOExceptione){ e.printStackTrace(); thrownewRuntimeException(e); } }); } /** *获取符合条件的key *@parampattern表达式 *@return */ publicList keys(Stringpattern){ List keys=newArrayList<>(); this.scan(pattern,item->{ //符合条件的key Stringkey=newString(item,StandardCharsets.UTF_8); keys.add(key); }); returnkeys; } }
但是会有一个问题:没法移动cursor,也只能scan一次,并且容易导致redis链接报错
先了解下scan、hscan、sscan、zscan
http://doc.redisfans.com/key/scan.html
keys为啥不安全?
keys的操作会导致数据库暂时被锁住,其他的请求都会被堵塞;业务量大的时候会出问题
SpringRedisTemplate实现scan
1.hscansscanzscan
例子中的"field"是值redis的key,即从key为"field"中的hash中查找
redisTemplate的opsForHash,opsForSet,opsForZSet可以分别对应sscan、hscan、zscan
当然这个网上的例子其实也不对,因为没有拿着cursor遍历,只scan查了一次
可以偷懒使用.count(Integer.MAX_VALUE),一下子全查回来;但是这样子和keys有啥区别呢?搞笑脸&疑问脸
可以使用(JedisCommands)connection.getNativeConnection()的hscan、sscan、zscan方法实现cursor遍历,参照下文2.2章节
try{ Cursor>cursor=redisTemplate.opsForHash().scan("field", ScanOptions.scanOptions().match("*").count(1000).build()); while(cursor.hasNext()){ Objectkey=cursor.next().getKey(); ObjectvalueSet=cursor.next().getValue(); } //关闭cursor cursor.close(); }catch(IOExceptione){ e.printStackTrace(); }
cursor.close();游标一定要关闭,不然连接会一直增长;可以使用clientlists``infoclients``infostats命令查看客户端连接状态,会发现scan操作一直存在
我们平时使用的redisTemplate.execute是会主动释放连接的,可以查看源码确认
clientlist ...... id=1531156addr=xxx:55845fd=8name=age=80idle=11flags=Ndb=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=rcmd=scan ...... org.springframework.data.redis.core.RedisTemplate#execute(org.springframework.data.redis.core.RedisCallback,boolean,boolean) finally{ RedisConnectionUtils.releaseConnection(conn,factory); }
2.scan
2.1网上给的例子多半是这个
这个connection.scan没法移动cursor,也只能scan一次
publicSetscan(StringmatchKey){ Set keys=redisTemplate.execute((RedisCallback >)connection->{ Set keysTmp=newHashSet<>(); Cursor cursor=connection.scan(newScanOptions.ScanOptionsBuilder().match("*"+matchKey+"*").count(1000).build()); while(cursor.hasNext()){ keysTmp.add(newString(cursor.next())); } returnkeysTmp; }); returnkeys; }
2.2使用MultiKeyCommands
获取connection.getNativeConnection;connection.getNativeConnection()实际对象是Jedis(debug可以看出),Jedis实现了很多接口
publicclassJedisextendsBinaryJedisimplementsJedisCommands,MultiKeyCommands,AdvancedJedisCommands,ScriptingCommands,BasicCommands,ClusterCommands,SentinelCommands
当scan.getStringCursor()存在且不是0的时候,一直移动游标获取
publicSetscan(Stringkey){ returnredisTemplate.execute((RedisCallback >)connection->{ Set keys=Sets.newHashSet(); JedisCommandscommands=(JedisCommands)connection.getNativeConnection(); MultiKeyCommandsmultiKeyCommands=(MultiKeyCommands)commands; ScanParamsscanParams=newScanParams(); scanParams.match("*"+key+"*"); scanParams.count(1000); ScanResult scan=multiKeyCommands.scan("0",scanParams); while(null!=scan.getStringCursor()){ keys.addAll(scan.getResult()); if(!StringUtils.equals("0",scan.getStringCursor())){ scan=multiKeyCommands.scan(scan.getStringCursor(),scanParams); continue; }else{ break; } } returnkeys; }); }
发散思考
cursor没有close,到底谁阻塞了,是Redis么
测试过程中,我基本只要发起十来个scan操作,没有关闭cursor,接下来的请求都卡住了
redis侧分析
clientlists``infoclients``infostats查看
发现连接数只有十几个,也没有阻塞和被拒绝的连接
configgetmaxclients查询redis允许的最大连接数是10000
1)"maxclients"
2)"10000"`
redis-cli在其他机器上也可以直接登录操作
综上,redis本身没有卡死
应用侧分析
netstat查看和redis的连接,6333是redis端口;连接一直存在
➜~netstat-an|grep6333 netstat-an|grep6333 tcp400xx.xx.xx.aa.52981xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52979xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52976xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52971xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52969xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52967xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52964xx.xx.xx.bb.6333ESTABLISHED tcp400xx.xx.xx.aa.52961xx.xx.xx.bb.6333ESTABLISHED
jstack查看应用的堆栈信息
发现很多WAITING的线程,全都是在获取redis连接
所以基本可以断定是应用的redis线程池满了
"http-nio-7007-exec-2"#139daemonprio=5os_prio=31tid=0x00007fda36c1c000nid=0xdd03waitingoncondition[0x00007000171ff000] java.lang.Thread.State:WAITING(parking) atsun.misc.Unsafe.park(NativeMethod) -parkingtowaitfor<0x00000006c26ef560>(ajava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) atjava.util.concurrent.locks.LockSupport.park(LockSupport.java:175) atjava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) atorg.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:590) atorg.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:441) atorg.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362) atredis.clients.util.Pool.getResource(Pool.java:49) atredis.clients.jedis.JedisPool.getResource(JedisPool.java:226) atredis.clients.jedis.JedisPool.getResource(JedisPool.java:16) atorg.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:276) atorg.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:469) atorg.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132) atorg.springframework.data.redis.core.RedisTemplate.executeWithStickyConnection(RedisTemplate.java:371) atorg.springframework.data.redis.core.DefaultHashOperations.scan(DefaultHashOperations.java:244)
综上,是应用侧卡死
后续
过了一个中午,redisclientlists显示scan连接还在,没有释放;应用线程也还是处于卡死状态
检查configgettimeout,redis未设置超时时间,可以用configsettimeoutxxx设置,单位秒;但是设置了redis的超时,redis释放了连接,应用还是一样卡住
1)"timeout"
2)"0"
netstat查看和redis的连接,6333是redis端口;连接从ESTABLISHED变成了CLOSE_WAIT;
jstack和原来表现一样,卡在JedisConnectionFactory.getConnection
➜~netstat-an|grep6333 netstat-an|grep6333 tcp400xx.xx.xx.aa.52981xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52979xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52976xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52971xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52969xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52967xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52964xx.xx.xx.bb.6333CLOSE_WAIT tcp400xx.xx.xx.aa.52961xx.xx.xx.bb.6333CLOSE_WAIT
回顾一下TCP四次挥手
ESTABLISHED表示连接已被建立
CLOSE_WAIT表示远程计算器关闭连接,正在等待socket连接的关闭
和现象符合
redis连接池配置
根据上面netstat-an基本可以确定redis连接池的大小是8;结合代码配置,没有指定的话,默认也确实是8
redis.clients.jedis.JedisPoolConfig privateintmaxTotal=8; privateintmaxIdle=8; privateintminIdle=0;
如何配置更大的连接池呢?
A.原配置
@Bean publicRedisConnectionFactoryredisConnectionFactory(){ RedisStandaloneConfigurationredisStandaloneConfiguration=newRedisStandaloneConfiguration(); redisStandaloneConfiguration.setHostName(redisHost); redisStandaloneConfiguration.setPort(redisPort); redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd)); JedisConnectionFactorycf=newJedisConnectionFactory(redisStandaloneConfiguration); cf.afterPropertiesSet(); returncf; } readTimeout,connectTimeout不指定,有默认值2000ms org.springframework.data.redis.connection.jedis.JedisConnectionFactory.MutableJedisClientConfiguration privateDurationreadTimeout=Duration.ofMillis(Protocol.DEFAULT_TIMEOUT); privateDurationconnectTimeout=Duration.ofMillis(Protocol.DEFAULT_TIMEOUT);
B.修改后配置
配置方式一:部分接口已经Deprecated了
@Bean publicRedisConnectionFactoryredisConnectionFactory(){ JedisPoolConfigjedisPoolConfig=newJedisPoolConfig(); jedisPoolConfig.setMaxTotal(16);//--最多可以建立16个连接了 jedisPoolConfig.setMaxWaitMillis(10000);//--10s获取不到连接池的连接, //--直接报错Couldnotgetaresourcefromthepool jedisPoolConfig.setMaxIdle(16); jedisPoolConfig.setMinIdle(0); JedisConnectionFactorycf=newJedisConnectionFactory(jedisPoolConfig); cf.setHostName(redisHost);//--@Deprecated cf.setPort(redisPort);//--@Deprecated cf.setPassword(redisPasswd);//--@Deprecated cf.setTimeout(30000);//--@Deprecated貌似没生效,30s超时,没有关闭连接池的连接; //--redis没有设置超时,会一直ESTABLISHED;redis设置了超时,且超时之后,会一直CLOSE_WAIT cf.afterPropertiesSet(); returncf; }
配置方式二:这是群里好友给找的新的配置方式,效果一样
RedisStandaloneConfigurationredisStandaloneConfiguration=newRedisStandaloneConfiguration(); redisStandaloneConfiguration.setHostName(redisHost); redisStandaloneConfiguration.setPort(redisPort); redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd)); JedisPoolConfigjedisPoolConfig=newJedisPoolConfig(); jedisPoolConfig.setMaxTotal(16); jedisPoolConfig.setMaxWaitMillis(10000); jedisPoolConfig.setMaxIdle(16); jedisPoolConfig.setMinIdle(0); cf=newJedisConnectionFactory(redisStandaloneConfiguration,JedisClientConfiguration.builder() .readTimeout(Duration.ofSeconds(30)) .connectTimeout(Duration.ofSeconds(30)) .usePooling().poolConfig(jedisPoolConfig).build());
以上这篇在RedisTemplate中使用scan代替keys指令操作就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持毛票票。