Re: Problem producing to Kinesis

Posted by Alexey Tsitkin on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Problem-producing-to-Kinesis-tp20730p20734.html

Thanks Gordon!

This indeed seems like the cause of the issue.
I've ran the program using 1.5.0, after building the appropriate connector, and it's working as expected.

Wondering how difficult is it to upgrade the 1.4 connector to a newer KPL version, as this kind of blocks running on EMR and producing to Kinesis... :-)

Alexey

On Thu, Jun 14, 2018 at 12:20 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

This could be related: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-1-4-and-below-STOPS-writing-to-Kinesis-after-June-12th-td22687.html#a22701.

Shortly put, the KPL library version used by default in the 1.4.x Kinesis connector, is no longer supported by AWS.

Users would need to use a upgraded version >= 0.12.6 and build the Kinesis connector for the producer to work.

We should probably add a warning about this in the Kinesis connector docs.

Cheers,

Gordon

On 14 June 2018 at 9:16:04 PM, Lasse Nedergaard ([hidden email]) wrote:

Hi. 

We see the same error and to my understanding it’s a known error from Amazon. See. https://github.com/awslabs/amazon-kinesis-producer/issues/39#issuecomment-396219522

We don’t have a workaround and haven’t found the reason for the exception. It is one off the reason why we move to Kafka in the near future. 

Med venlig hilsen / Best regards

Lasse Nedergaard

Den 14. jun. 2018 kl. 20.24 skrev Alexey Tsitkin <[hidden email]>:

  Hi,

I'm trying to run a simple program which consumes from one kinesis stream, does a simple transformation, and produces to another stream.

Running on Flink 1.4.0.

Code can be seen here (if needed I can also paste it directly on this thread):

https://stackoverflow.com/questions/50847164/flink-producing-to-kinesis-not-working

Consuming the source stream works great, but trying to use the producer causes the exception:

org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages.

    at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.add(Daemon.java:176)

    at org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.KinesisProducer.addUserRecord(KinesisProducer.java:477)

    at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer.invoke(FlinkKinesisProducer.java:248)

    ...

Did anyone have something similar?

Or is there any way to debug the daemon itself, to understand the source of the error?

As you can see, this is a trivial example, which I mostly copy-pasted from the documentation.

Thanks,

Alexey