Conflicts between the JDBC and postgresql-cdc SQL connectors

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Conflicts between the JDBC and postgresql-cdc SQL connectors

Sebastián Magrí
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Dawid Wysakowicz-2

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Sebastián Magrí
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Jark Wu-3
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Sebastián Magrí
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí

pom.xml (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Jark Wu-3
Hi Sebastián,

Could you try to add combine.children="append" attribute to the transformers configuration?
You can also see the full shade plugin configuration here [1].

Best,
Jark


On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <[hidden email]> wrote:
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Sebastián Magrí
Applied that parameter and that seems to get me some progress here.

I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file.

However, now on runtime the application fails to find the debezium source function class coming down to this error:

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.

The class is indeed in jar, though.

Any thougths?

On Thu, 28 Jan 2021 at 09:57, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

Could you try to add combine.children="append" attribute to the transformers configuration?
You can also see the full shade plugin configuration here [1].

Best,
Jark


On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <[hidden email]> wrote:
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Sebastián Magrí
The root of the previous error seemed to be the flink version the connector was compiled for. I've tried compiling my own postgresql-cdc connector, but still have some issues with dependencies.

On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <[hidden email]> wrote:
Applied that parameter and that seems to get me some progress here.

I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file.

However, now on runtime the application fails to find the debezium source function class coming down to this error:

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.

The class is indeed in jar, though.

Any thougths?

On Thu, 28 Jan 2021 at 09:57, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

Could you try to add combine.children="append" attribute to the transformers configuration?
You can also see the full shade plugin configuration here [1].

Best,
Jark


On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <[hidden email]> wrote:
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí
Reply | Threaded
Open this post in threaded view
|

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

rmetzger0
I don't know what your dependency issue is (post it here if you want help!), but I generally recommend using mvn dependency:tree to debug version clashes (and then pin or exclude versions)

On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí <[hidden email]> wrote:
The root of the previous error seemed to be the flink version the connector was compiled for. I've tried compiling my own postgresql-cdc connector, but still have some issues with dependencies.

On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <[hidden email]> wrote:
Applied that parameter and that seems to get me some progress here.

I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file.

However, now on runtime the application fails to find the debezium source function class coming down to this error:

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.

The class is indeed in jar, though.

Any thougths?

On Thu, 28 Jan 2021 at 09:57, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

Could you try to add combine.children="append" attribute to the transformers configuration?
You can also see the full shade plugin configuration here [1].

Best,
Jark


On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <[hidden email]> wrote:
Hi Jark!

Please find the full pom file attached.

Best Regards,

On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
Hi Sebastián,

I think Dawid is right. 

Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains  

com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory                     

Best,
Jark                                                                                                                                     

On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
Thanks a lot for looking into it Dawid,

In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see

org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory

Even after applying the ServicesResourceTransformer.


On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here?

However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.

If not, you could try applying the ServicesResourceTransformer[1]

Best,

Dawid

[1] https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

On 26/01/2021 12:29, Sebastián Magrí wrote:
Hi!

I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here:


In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success.

The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency.

Any guidance or hints I could follow would be really appreciated.

--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí


--
Sebastián Ramírez Magrí