avro-confluent supports authentication enabled schema registry

[tao xiao]

Hi team,

Confluent schema registry supports HTTP basic authentication[1] but I don't find the corresponding configs in Flink documentation[2]. Is this achievable in Flink avro-confluent?

[1]https://docs.confluent.io/platform/current/confluent-security-plugins/schema-registry/install.html#authentication-mechanisms

[2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/avro-confluent/#format-options

--

[Fabian Paul]

Hi Tao,

Thanks for reaching out. Have you tried the following

 'value.avro-confluent.schema-registry.url' = 'https://{SR_API_KEY}:{[hidden email]', 

It may be possible to provide basic HTTP authentication by adding your username and password to the URL. There is already a similar ticket open unfortunately without much progress. [1]

Please let me know if this works for  you otherwise we can try to find a different solution.

Best,

Fabian

[1] https://issues.apache.org/jira/browse/FLINK-22763

On 2. Jun 2021, at 10:58, tao xiao <[hidden email]> wrote:

Hi team,

Confluent schema registry supports HTTP basic authentication[1] but I don't find the corresponding configs in Flink documentation[2]. Is this achievable in Flink avro-confluent?

[1]https://docs.confluent.io/platform/current/confluent-security-plugins/schema-registry/install.html#authentication-mechanisms

[2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/avro-confluent/#format-options

--

[tao xiao]

Hi Fabian,

Unfortunately this will not work in our environment where we implement our own io.confluent.kafka.schemaregistry.client.security.bearerauth.BearerAuthCredentialProvider which does the login and supplies the JWT to authorization HTTP header. The only way it will work is to pass the schema registry config BEARER_AUTH_CREDENTIALS_SOURCE [1] to table format factory

[1] https://github.com/confluentinc/schema-registry/blob/master/schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java#L85

On Wed, Jun 2, 2021 at 5:27 PM Fabian Paul <[hidden email]> wrote:

Hi Tao,

Thanks for reaching out. Have you tried the following

 'value.avro-confluent.schema-registry.url' = 'https://{SR_API_KEY}:{[hidden email]', 

It may be possible to provide basic HTTP authentication by adding your username and password to the URL. There is already a similar ticket open unfortunately without much progress. [1]

Please let me know if this works for  you otherwise we can try to find a different solution.

Best,

Fabian

[1] https://issues.apache.org/jira/browse/FLINK-22763

On 2. Jun 2021, at 10:58, tao xiao <[hidden email]> wrote:

Hi team,

Confluent schema registry supports HTTP basic authentication[1] but I don't find the corresponding configs in Flink documentation[2]. Is this achievable in Flink avro-confluent?

[1]https://docs.confluent.io/platform/current/confluent-security-plugins/schema-registry/install.html#authentication-mechanisms

[2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/avro-confluent/#format-options

--

Regards,

Tao

--

[Fabian Paul]

Hi Tao,

I was browsing the code a bit and I think this is currently not support but it seems to be not too 

difficult to implement. You would need to allow a map of configurations and finally pass it to [1]

Can you create a ticket in our JIRA?

Would you be willing to contribute this feature?

Best,

Fabian

[1] https://github.com/apache/flink/blob/1db4e560d1b46fac27a18bce9556fec646f063d9/flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/CachedSchemaCoderProvider.java#L54

On 2. Jun 2021, at 13:57, tao xiao <[hidden email]> wrote:

Hi Fabian,

Unfortunately this will not work in our environment where we implement our own io.confluent.kafka.schemaregistry.client.security.bearerauth.BearerAuthCredentialProvider which does the login and supplies the JWT to authorization HTTP header. The only way it will work is to pass the schema registry config BEARER_AUTH_CREDENTIALS_SOURCE [1] to table format factory

[1] https://github.com/confluentinc/schema-registry/blob/master/schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java#L85

On Wed, Jun 2, 2021 at 5:27 PM Fabian Paul <[hidden email]> wrote:

Hi Tao,

Thanks for reaching out. Have you tried the following

 'value.avro-confluent.schema-registry.url' = 'https://{SR_API_KEY}:{[hidden email]', 

It may be possible to provide basic HTTP authentication by adding your username and password to the URL. There is already a similar ticket open unfortunately without much progress. [1]

Please let me know if this works for  you otherwise we can try to find a different solution.

Best,

Fabian

[1] https://issues.apache.org/jira/browse/FLINK-22763

On 2. Jun 2021, at 10:58, tao xiao <[hidden email]> wrote:

Hi team,

Confluent schema registry supports HTTP basic authentication[1] but I don't find the corresponding configs in Flink documentation[2]. Is this achievable in Flink avro-confluent?

[1]https://docs.confluent.io/platform/current/confluent-security-plugins/schema-registry/install.html#authentication-mechanisms

[2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/avro-confluent/#format-options

--

Regards,

Tao

--

[tao xiao]

JIRA created https://issues.apache.org/jira/browse/FLINK-22858 but I cannot assign it to myself. Can you pls assign it to me?

On Wed, Jun 2, 2021 at 11:00 PM Fabian Paul <[hidden email]> wrote:

Hi Tao,

I was browsing the code a bit and I think this is currently not support but it seems to be not too 

difficult to implement. You would need to allow a map of configurations and finally pass it to [1]

Can you create a ticket in our JIRA?

Would you be willing to contribute this feature?

Best,

Fabian

[1] https://github.com/apache/flink/blob/1db4e560d1b46fac27a18bce9556fec646f063d9/flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/CachedSchemaCoderProvider.java#L54

On 2. Jun 2021, at 13:57, tao xiao <[hidden email]> wrote:

Hi Fabian,

Unfortunately this will not work in our environment where we implement our own io.confluent.kafka.schemaregistry.client.security.bearerauth.BearerAuthCredentialProvider which does the login and supplies the JWT to authorization HTTP header. The only way it will work is to pass the schema registry config BEARER_AUTH_CREDENTIALS_SOURCE [1] to table format factory

[1] https://github.com/confluentinc/schema-registry/blob/master/schema-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaSchemaSerDeConfig.java#L85

On Wed, Jun 2, 2021 at 5:27 PM Fabian Paul <[hidden email]> wrote:

Hi Tao,

Thanks for reaching out. Have you tried the following

 'value.avro-confluent.schema-registry.url' = 'https://{SR_API_KEY}:{[hidden email]', 

It may be possible to provide basic HTTP authentication by adding your username and password to the URL. There is already a similar ticket open unfortunately without much progress. [1]

Please let me know if this works for  you otherwise we can try to find a different solution.

Best,

Fabian

[1] https://issues.apache.org/jira/browse/FLINK-22763

On 2. Jun 2021, at 10:58, tao xiao <[hidden email]> wrote:

Hi team,

Confluent schema registry supports HTTP basic authentication[1] but I don't find the corresponding configs in Flink documentation[2]. Is this achievable in Flink avro-confluent?

[1]https://docs.confluent.io/platform/current/confluent-security-plugins/schema-registry/install.html#authentication-mechanisms

[2]https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/table/formats/avro-confluent/#format-options

--

Regards,

Tao

--

Regards,

Tao

--