(DEPRECATED) Apache Flink User Mailing List archive.

Using FlinkKinesisConsumer through a proxy

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
Vijay Balakrishnan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Using FlinkKinesisConsumer through a proxy

Vijay Balakrishnan
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Vijay Balakrishnan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Vijay Balakrishnan
I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Aljoscha Krettek
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Aljoscha Krettek
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Tzu-Li (Gordon) Tai
Hi,

Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).

The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:

```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```

Could you try that out and see if it works for you?

I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].

Cheers,
Gordon

On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <[hidden email]> wrote:
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Vijay Balakrishnan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Vijay Balakrishnan
Hi Gordon,
This still didn't work :(

Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyDomain", "...");
            inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", "http://.....com");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPort", "911");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyUsername", "...");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPassword", "..");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "nonProxyHosts", "


How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay

On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi,

Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).

The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:

```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```

Could you try that out and see if it works for you?

I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].

Cheers,
Gordon

On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <[hidden email]> wrote:
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Tzu-Li (Gordon) Tai
Hi Vijay,

I’m pretty sure that this should work with the properties that you provided, unless the AWS Kinesis SDK isn’t working as expected.

What I’ve tested is that with those properties, the ClientConfiguration used to build the Kinesis client has the proxy domain / host / ports etc. properly set.
And according to [1], this should be enough to configure the constructed Kinesis client to connect via the proxy.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html


On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan ([hidden email]) wrote:

Hi Gordon,
This still didn't work :(

Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyDomain", "...");
            inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", "http://.....com");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPort", "911");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyUsername", "...");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPassword", "..");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "nonProxyHosts", "


How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay

On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi,

Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).

The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:

```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```

Could you try that out and see if it works for you?

I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].

Cheers,
Gordon

On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <[hidden email]> wrote:
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Vijay Balakrishnan
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Vijay Balakrishnan
Hi Gordon,
Finally figured out my issue.Do not need to add http:// in proxyHost name.
String proxyHost= "proxy-chain....com";//not http://proxy-chain...com
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", proxyHost);//<== mo http:// in proxyHost name

TIA,
Vijay


On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Vijay,

I’m pretty sure that this should work with the properties that you provided, unless the AWS Kinesis SDK isn’t working as expected.

What I’ve tested is that with those properties, the ClientConfiguration used to build the Kinesis client has the proxy domain / host / ports etc. properly set.
And according to [1], this should be enough to configure the constructed Kinesis client to connect via the proxy.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html


On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan ([hidden email]) wrote:

Hi Gordon,
This still didn't work :(

Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyDomain", "...");
            inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", "http://.....com");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPort", "911");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyUsername", "...");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPassword", "..");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "nonProxyHosts", "


How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay

On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi,

Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).

The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:

```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```

Could you try that out and see if it works for you?

I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].

Cheers,
Gordon

On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <[hidden email]> wrote:
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Tzu-Li (Gordon) Tai
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Using FlinkKinesisConsumer through a proxy

Tzu-Li (Gordon) Tai
Good to hear that it's working, thanks for the update!

On Sat, Dec 1, 2018, 4:29 AM Vijay Balakrishnan <[hidden email] wrote:
Hi Gordon,
Finally figured out my issue.Do not need to add http:// in proxyHost name.
String proxyHost= "proxy-chain....com";//not http://proxy-chain...com
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", proxyHost);//<== mo http:// in proxyHost name

TIA,
Vijay


On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi Vijay,

I’m pretty sure that this should work with the properties that you provided, unless the AWS Kinesis SDK isn’t working as expected.

What I’ve tested is that with those properties, the ClientConfiguration used to build the Kinesis client has the proxy domain / host / ports etc. properly set.
And according to [1], this should be enough to configure the constructed Kinesis client to connect via the proxy.

Cheers,
Gordon

[1] https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html


On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan ([hidden email]) wrote:

Hi Gordon,
This still didn't work :(

Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyDomain", "...");
            inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyHost", "http://.....com");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPort", "911");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyUsername", "...");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "proxyPassword", "..");
            kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX  + "nonProxyHosts", "


How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay

On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:
Hi,

Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).

The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:

```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```

Could you try that out and see if it works for you?

I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].

Cheers,
Gordon

On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <[hidden email]> wrote:
Hi,

I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.

Best,
Aljoscha

On 3. Oct 2018, at 17:29, Vijay Balakrishnan <[hidden email]> wrote:

I have been trying with all variations  to no avail of java -Dhttp.nonProxyHosts=..  -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration

On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <[hidden email]> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. 
Works outside the proxy.

Properties kinesisConsumerConfig = new Properties();
        kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);

        if (local) {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
        } else {
            kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
        }

        //only for Consumer
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
        kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
        FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
                "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Free forum by Nabble Edit this page