Hi! I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here: In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success. The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency. Any guidance or hints I could follow would be really appreciated. -- Sebastián Ramírez Magrí |
Hi, Unfortunately I am not familiar with the packaging of flink-connector-postgres-cdc. Maybe @Jark could help here? However, I think the problem that you cannot find the connector is caused because of lack of entry in the resulting Manifest file. If there are overlapping classes maven does not exclude whole dependencies, but rather picks the overlapping class from one of the two. Could you check if you see entries for all tables in src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. If not, you could try applying the ServicesResourceTransformer[1] Best, Dawid On 26/01/2021 12:29, Sebastián Magrí
wrote:
|
Thanks a lot for looking into it Dawid, In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory Even after applying the ServicesResourceTransformer. On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <[hidden email]> wrote:
-- Sebastián Ramírez Magrí
|
Hi Sebastián, I think Dawid is right. Could you share the pom file? I also tried to package flink-connector-postgres-cdc with ServicesResourceTransformer, and the Factory file contains com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory Best, Jark On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <[hidden email]> wrote:
|
Hi Jark! Please find the full pom file attached. Best Regards, On Thu, 28 Jan 2021 at 03:21, Jark Wu <[hidden email]> wrote:
-- Sebastián Ramírez Magrí
|
Hi Sebastián, Could you try to add combine.children="append" attribute to the transformers configuration? You can also see the full shade plugin configuration here [1]. Best, Jark On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <[hidden email]> wrote:
|
Applied that parameter and that seems to get me some progress here. I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file. However, now on runtime the application fails to find the debezium source function class coming down to this error: Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction ClassLoader info: URL ClassLoader: Class not resolvable through given classloader. The class is indeed in jar, though. Any thougths? On Thu, 28 Jan 2021 at 09:57, Jark Wu <[hidden email]> wrote:
-- Sebastián Ramírez Magrí
|
The root of the previous error seemed to be the flink version the connector was compiled for. I've tried compiling my own postgresql-cdc connector, but still have some issues with dependencies. On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <[hidden email]> wrote:
-- Sebastián Ramírez Magrí
|
I don't know what your dependency issue is (post it here if you want help!), but I generally recommend using mvn dependency:tree to debug version clashes (and then pin or exclude versions) On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |