Kafka Streams代理连接超时设置

Kafka Streams代理连接超时设置

问题描述:

我们正在使用kafka-streams 2.3.1,我刚刚注意到,如果代理关闭,streams应用程序似乎很满足尝试永久连接的需求.

We are using kafka-streams 2.3.1 and I've just noticed that if broker is down, the streams app seems to be content to try to keep trying connecting forever.

new KafkaStreams(createTopology(), properties()).start()

 o.apache.kafka.clients.NetworkClient - [AdminClient clientId=test] Connection to node -1 (broker/127.0.0.1:9092) could not be established. Broker may not be available.

在此过程中,流状态为REBALANCING,因此无法确定连接是否刚刚断开.

The streams state is REBALANCING while this is going on so there's no good way to determine if the connection is just broken.

是否可以为代理连接尝试设置超时或重试次数?

Is there a way to set either a timeout or a number of retries for broker(s) connection attempts?

很遗憾,此问题没有好的解决方法.这个问题实际上是一个消费者问题,因为消费者只是尝试重新连接,但并未将其内部状态显示给Kafka Streams.另外,不可能将使用者配置为在某个时候放弃.

There is unfortunately no good workaround for this problem. The issue is actually a consumer issue, as the consumer just tries to reconnect but does not surface it's internal state to Kafka Streams. Also, it's not possible to configure the consumer to give up at some point.

有一个KIP可以向Kafka Streams添加一个"DISCONNECTED"状态,但是最近进展不大...这很复杂...

There is a KIP to add a "DISCONNECTED" state to Kafka Streams, but there was not much progress lately... It's complicated... https://cwiki.apache.org/confluence/display/KAFKA/KIP-457%3A+Add+DISCONNECTED+status+to+Kafka+Streams