about Cassandra “NoHostAvailableException” when all host available

再次遇到NoHostAvailableException,诡异的是:检查了下Cassandra的节点,又都是Up状态,和之前遇到的情况如出一辙。
所以有必要关于这个异常做个记录,汇总下两次遇到这种情况的原因:

(1) localDc配置错误

之前一直work良好,后来Review设计时,发现之前开启远程dc failover可能有点问题,具体参考之前文章的总结:

开启dc level failover后,是否能满足性能需求,如果不能,和一个挂掉的系统又有何区别。

考虑到这种情况,不如整个应用层做failover,而不是只远程访问远程Cassandra DC.

应用层代码是这么控制的:

DCAwareRoundRobinPolicy childPolicy = isDCLevelFailoverSupport?new DCAwareRoundRobinPolicy(localDC,1,true):new DCAwareRoundRobinPolicy(localDC);

使用feature taggle关闭后(isDCLevelFailoverSupport = false):

Server直接启动不了:

Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
        ... 34 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
        at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:205)
        at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:44)
        at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:271)
        at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:112)
        at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:92)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:134)
        at com.datastax.driver.core.AbstractSession.executeAsync(AbstractSession.java:63)
        ... 36 more

查看driver层代码:

com.datastax.driver.core.policies.DCAwareRoundRobinPolicy

        CopyOnWriteArrayList<Host> localLiveHosts = perDcLiveHosts.get(localDc);  //获取配置的localdc的所有节点
        final List<Host> hosts = localLiveHosts == null ? Collections.<Host>emptyList() : cloneList(localLiveHosts);
        final int startIdx = index.getAndIncrement();

        return new AbstractIterator<Host>() {

            private int idx = startIdx;
            private int remainingLocal = hosts.size(); //匹配配置的dcid的节点的数量

            // For remote Dcs
            private Iterator<String> remoteDcs;
            private List<Host> currentDcHosts;
            private int currentDcRemaining;

            @Override
            protected Host computeNext() {
                while (true) {
                    if (remainingLocal > 0) {
                        remainingLocal--;
                        int c = idx++ % hosts.size();
                        if (c < 0) {
                            c += hosts.size();
                        }
                        return hosts.get(c);
                    }

                    if (currentDcHosts != null && currentDcRemaining > 0) {
                        currentDcRemaining--;
                        int c = idx++ % currentDcHosts.size();
                        if (c < 0) {
                            c += currentDcHosts.size();
                        }
                        return currentDcHosts.get(c);
                    }

                    ConsistencyLevel cl = statement.getConsistencyLevel() == null
                            ? configuration.getQueryOptions().getConsistencyLevel()
                            : statement.getConsistencyLevel(); //获取一致性级别

                    if (dontHopForLocalCL && cl.isDCLocal()) //这里是关键,使用之前构造器关闭远程访问时,这个dontHopForLocalCL是false(即不要打破一致性约束),且当前我们的应用是localquorom级别。
                        return endOfData(); //所以返回null

                    if (remoteDcs == null) {
                        Set<String> copy = new HashSet<String>(perDcLiveHosts.keySet());
                        copy.remove(localDc);
                        remoteDcs = copy.iterator();
                    }

                    if (!remoteDcs.hasNext())
                        break;

                    String nextRemoteDc = remoteDcs.next();
                    CopyOnWriteArrayList<Host> nextDcHosts = perDcLiveHosts.get(nextRemoteDc);
                    if (nextDcHosts != null) {
                        // Clone for thread safety
                        List<Host> dcHosts = cloneList(nextDcHosts);
                        currentDcHosts = dcHosts.subList(0, Math.min(dcHosts.size(), usedHostsPerRemoteDc));
                        currentDcRemaining = currentDcHosts.size();
                    }
                }
                return endOfData();
            }
        };
    }

实际上,默认不会直接用上面的,而是将上面做一个子策略包装到com.datastax.driver.core.policies.TokenAwarePolicy里面去。

                .withLoadBalancingPolicy(new TokenAwarePolicy(childPolicy))

而这个策略的query node plan(选择发送请求到哪个节点上的选择器)是:

com.datastax.driver.core.policies.TokenAwarePolicy:

    @Override
            protected Host computeNext() {
                while (iter.hasNext()) {
                    Host host = iter.next();
                    if (host.isUp() && childPolicy.distance(host) == HostDistance.LOCAL) //必须是local dc
                        return host;
                }

                if (childIterator == null)
                    childIterator = childPolicy.newQueryPlan(loggedKeyspace, statement);

                while (childIterator.hasNext()) {  //这种情况下,子策略一个节点都没有,理由如上
                    Host host = childIterator.next();
                    // Skip it if it was already a local replica
                    if (!replicas.contains(host) || childPolicy.distance(host) != HostDistance.LOCAL)
                        return host;
                }
                return endOfData();
            }
        };

所以可见,假设配置的dcid不属于任何cassandra nodes之时,且关闭了可以远程访问的情况下,这个时候driver会认为没有任何节点Up可用。
其实从log中也可以看到具体的情况,当然它本身没有告诉你后面的异常是因为这个引起的。

[10/30/2017 08:51:07.771][][main]INFO  DCAwareRoundRobinPolicy-Using provided data-center name 'MyDC' for DCAwareRoundRobinPolicy
[10/30/2017 08:51:07.772][][main]WARN  DCAwareRoundRobinPolicy-Some contact points don't match local data center. Local DC = MyDC. Non-conforming contact points:
......

解决方案: 配置对local dc.

(2)system_auth存在单点故障,且Driver版本不够高

现象:当Cassandra节点down了后,重启后,忽然发现不工作了,查询日志,没有可用节点,但是明明可用。

解决:

从日志看,是因为授权失败,导致Control Connection不再重试,所以所有节点再重启后,感知不到节点已经启动起来了,所以提示NoHostAvailableException:

[07/10/2017 02:28:56.817][][Reconnection-1]ERROR AbstractReconnectionHandler-Authentication error on host sjwdcaat101.webex.com/10.252.67.130:9042: Username and/or password are incorrect
[07/10/2017 02:28:56.817][][Reconnection-1]ERROR AbstractReconnectionHandler-Retry against testcasandra.test.com/10.224.2.110 have been suspended. It won't be retried unless the node is restarted.

但是明明密码可以登录。却说用户名和密码不对,后来查了下,是因为system_auth表配置的复制因子被设置成1,所以当这个节点挂掉后,无法授权,而刚好这种错误在低版本的driver中是不做重试的,所以后面即使启动起来也不行了。

解决:
(1)修改复制策略,不要搞成单点:

ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };

(2)升级Driver: 2.1.10+
新版本:

com.datastax.driver.core.AbstractReconnectionHandler

    // Retrying on authentication errors makes sense for applications that can update the credentials at runtime, we don't want to force them
    // to restart.
    protected boolean onAuthenticationException(AuthenticationException e, long nextDelayMs) {
        return true;
    }

旧版本:

    protected boolean onAuthenticationException(AuthenticationException e, long nextDelayMs) {
        return false;
    }

总结:

Cassandra所有节点都是up状态,但是却报NoHostAvailableException这种错误,最后查出的原因往往比较搞笑,但是这种错误确实让人很抓狂,因为是眼瞅没有问题但是客户端彻底不工作了。遇到这种错误,还是应该第一步找到最早出问题的那个时间点,然后还原场景做分析,方能有的放矢。

发布者

傅, 健

程序员,Java、C++、开源爱好者. About me

发表评论