Re: Problem producing to Kinesis

Posted by Tzu-Li (Gordon) Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Problem-producing-to-Kinesis-tp20730p20732.html

Hi,

This could be related: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-4-and-below-STOPS-writing-to-Kinesis-after-June-12th-td22687.html#a22701.

Shortly put, the KPL library version used by default in the 1.4.x Kinesis connector, is no longer supported by AWS.

Users would need to use a upgraded version >= 0.12.6 and build the Kinesis connector for the producer to work.

We should probably add a warning about this in the Kinesis connector docs.

Cheers,

Gordon

On 14 June 2018 at 9:16:04 PM, Lasse Nedergaard ([hidden email]) wrote:

Hi. 

We see the same error and to my understanding it’s a known error from Amazon. See. https://github.com/awslabs/amazon-kinesis-producer/issues/39#issuecomment-396219522

We don’t have a workaround and haven’t found the reason for the exception. It is one off the reason why we move to Kafka in the near future. 

Med venlig hilsen / Best regards

Lasse Nedergaard

Den 14. jun. 2018 kl. 20.24 skrev Alexey Tsitkin <[hidden email]>:

  Hi,

I'm trying to run a simple program which consumes from one kinesis stream, does a simple transformation, and produces to another stream.

Running on Flink 1.4.0.

Code can be seen here (if needed I can also paste it directly on this thread):

https://stackoverflow.com/questions/50847164/flink-producing-to-kinesis-not-working

Consuming the source stream works great, but trying to use the producer causes the exception:

org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages.

    at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.add(Daemon.java:176)

    at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.KinesisProducer.addUserRecord(KinesisProducer.java:477)

    at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer.invoke(FlinkKinesisProducer.java:248)

    ...

Did anyone have something similar?

Or is there any way to debug the daemon itself, to understand the source of the error?

As you can see, this is a trivial example, which I mostly copy-pasted from the documentation.

Thanks,

Alexey