Kinesis Producer in 1.4.2: testing locally with Kinesalite not working

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Kinesis Producer in 1.4.2: testing locally with Kinesalite not working

Philipp Bussche
Hi there,

when trying to use a KinesisProducer which has both aws.region and
aws.endpoint set I am receiving the following error message:

07/22/2018 19:52:42 Source: EventStream -> (Sink: Unnamed, EventAndVenueMap
-> (Filter, Sink: EventToInventorySink), Sink: ElasticSearchEventsSink)(4/8)
switched to FAILED
java.lang.IllegalArgumentException: Either AWS region ('aws.region') or AWS
endpoint ('aws.region') must be set in the config.
        at
org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil.validateAwsConfiguration(KinesisConfigUtil.java:248)
        at
org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil.getValidatedProducerConfiguration(KinesisConfigUtil.java:191)
        at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer.open(FlinkKinesisProducer.java:210)

However when I am only using aws.endpoint (since I want to test against a
local Kinesalite instance) I am getting this:

07/22/2018 19:59:42 Source: EventStream -> (Sink: Unnamed, EventAndVenueMap
-> (Filter, Sink: EventToInventorySink), Sink: ElasticSearchEventsSink)(2/8)
switched to FAILED
java.lang.NullPointerException
        at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
        at java.util.regex.Matcher.reset(Matcher.java:309)
        at java.util.regex.Matcher.<init>(Matcher.java:229)
        at java.util.regex.Pattern.matcher(Pattern.java:1093)
        at java.util.regex.Pattern.matches(Pattern.java:1134)
        at
org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration.setRegion(KinesisProducerConfiguration.java:1279)

It seems that the KinesisProducer needs the aws.region set but can't have
both set which would mean I can't test against my local instance of
Kinesalite anymore with 1.4.2 ? This was working fine in 1.3.2 as having
both set then wasn't picked up as a problem.

Thanks
Philipp



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Kinesis Producer in 1.4.2: testing locally with Kinesalite not working

vino yang
Hi Philipp,

It sounds this is a bug, can you create a JIRA issue for this problem?

Thanks, vino.

2018-07-23 2:01 GMT+08:00 Philipp Bussche <[hidden email]>:
Hi there,

when trying to use a KinesisProducer which has both aws.region and
aws.endpoint set I am receiving the following error message:

07/22/2018 19:52:42     Source: EventStream -> (Sink: Unnamed, EventAndVenueMap
-> (Filter, Sink: EventToInventorySink), Sink: ElasticSearchEventsSink)(4/8)
switched to FAILED
java.lang.IllegalArgumentException: Either AWS region ('aws.region') or AWS
endpoint ('aws.region') must be set in the config.
        at
org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil.validateAwsConfiguration(KinesisConfigUtil.java:248)
        at
org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil.getValidatedProducerConfiguration(KinesisConfigUtil.java:191)
        at
org.apache.flink.streaming.connectors.kinesis.FlinkKinesisProducer.open(FlinkKinesisProducer.java:210)

However when I am only using aws.endpoint (since I want to test against a
local Kinesalite instance) I am getting this:

07/22/2018 19:59:42     Source: EventStream -> (Sink: Unnamed, EventAndVenueMap
-> (Filter, Sink: EventToInventorySink), Sink: ElasticSearchEventsSink)(2/8)
switched to FAILED
java.lang.NullPointerException
        at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
        at java.util.regex.Matcher.reset(Matcher.java:309)
        at java.util.regex.Matcher.<init>(Matcher.java:229)
        at java.util.regex.Pattern.matcher(Pattern.java:1093)
        at java.util.regex.Pattern.matches(Pattern.java:1134)
        at
org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration.setRegion(KinesisProducerConfiguration.java:1279)

It seems that the KinesisProducer needs the aws.region set but can't have
both set which would mean I can't test against my local instance of
Kinesalite anymore with 1.4.2 ? This was working fine in 1.3.2 as having
both set then wasn't picked up as a problem.

Thanks
Philipp



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/