(DEPRECATED) Apache Flink User Mailing List archive.

Conversion to relational algebra failed to preserve datatypes

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
Yuan,Youjun
Reply | Threaded
Open this post in threaded view
♦
♦
|

Conversion to relational algebra failed to preserve datatypes

Yuan,Youjun

Hi,

 

I am getting the following error while submitting job to a cluster, which seems failed to compare 2 RelDateTypes, though they seems identical (from the error message), and everything is OK if I run it locally.

I guess calcite failed to compare the first field named ts, of type TIMESTAMP(3), because:

  • If I don’t select ts, then everything goes fine
  • If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then everything is fine
  • If I switch to EventTime, the issue also goes away. Currently it’s ProcessTime

 

I am using Flink 1.4, and submitting job to a standalone cluster.

 

Below are the error:

----------------------------------------------------------------------------------------------------------------------------------------------------------------

Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused an error:       

at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93) 

at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)      

at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76) 

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69) ... 9 more

Caused by: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:

validated type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

converted type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

...

at org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451) 

at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)       

at org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

thanks in advance,

youjun
Timo Walther
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Conversion to relational algebra failed to preserve datatypes

Timo Walther
Hi,

could you maybe post the query that caused the exception? I guess the exception is related to a time attribute [1] for the optimizer time attributes and timestamps make no difference however they have a slightly different data type that might have caused the error. I think is a bug that should be fixed, once we have more context.

Regards,
Timo


[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming.html#time-attributes


Am 14.09.18 um 10:49 schrieb Yuan,Youjun:

Hi,

 

I am getting the following error while submitting job to a cluster, which seems failed to compare 2 RelDateTypes, though they seems identical (from the error message), and everything is OK if I run it locally.

I guess calcite failed to compare the first field named ts, of type TIMESTAMP(3), because:

  • If I don’t select ts, then everything goes fine
  • If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then everything is fine
  • If I switch to EventTime, the issue also goes away. Currently it’s ProcessTime

 

I am using Flink 1.4, and submitting job to a standalone cluster.

 

Below are the error:

----------------------------------------------------------------------------------------------------------------------------------------------------------------

Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused an error:       

at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93) 

at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)      

at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76) 

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69) ... 9 more

Caused by: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:

validated type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

converted type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

...

at org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451) 

at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)       

at org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

thanks in advance,

youjun
Yuan,Youjun
Reply | Threaded
Open this post in threaded view
♦
♦
|

答复: Conversion to relational algebra failed to preserve datatypes

Yuan,Youjun

Hi Timo,

 

I believe it has something to do with the number of field in the source. I can reproduce the issue with very simple SQL, like:

INSERT INTO IVC_ALARM_PUBLIC_OUT SELECT rowtime as ts FROM IVC_ALARM_PUBLIC_IN

Where rowtime actually represent proctime (I registered it as rowtime.proctime).

 

If I remove a few fields in the source, then everything goes fine.

 

Thanks

Youjun

 

发件人: Timo Walther <[hidden email]>
发送时间: Friday, September 14, 2018 9:17 PM
收件人: [hidden email]
主题: Re: Conversion to relational algebra failed to preserve datatypes

 

Hi,

could you maybe post the query that caused the exception? I guess the exception is related to a time attribute [1] for the optimizer time attributes and timestamps make no difference however they have a slightly different data type that might have caused the error. I think is a bug that should be fixed, once we have more context.

Regards,
Timo


[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming.html#time-attributes


Am 14.09.18 um 10:49 schrieb Yuan,Youjun:

Hi,

 

I am getting the following error while submitting job to a cluster, which seems failed to compare 2 RelDateTypes, though they seems identical (from the error message), and everything is OK if I run it locally.

I guess calcite failed to compare the first field named ts, of type TIMESTAMP(3), because:

  • If I don’t select ts, then everything goes fine
  • If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then everything is fine
  • If I switch to EventTime, the issue also goes away. Currently it’s ProcessTime

 

I am using Flink 1.4, and submitting job to a standalone cluster.

 

Below are the error:

----------------------------------------------------------------------------------------------------------------------------------------------------------------

Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused an error:       

at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93) 

at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)      

at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76) 

at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69) ... 9 more

Caused by: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:

validated type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

converted type:

RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL

...

at org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451) 

at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)       

at org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)

----------------------------------------------------------------------------------------------------------------------------------------------------------------

 

thanks in advance,

youjun

 
Free forum by Nabble Edit this page