(DEPRECATED) Apache Flink User Mailing List archive.

How can I set charset for flink sql？

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

How can I set charset for flink sql？

程骥
My sql like this(contain a Chinese word)

Get exception when I submit the job to cluster.

Is there anyone tell me how to deal with it,thanks!
Timo Walther
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How can I set charset for flink sql？

Timo Walther
Hi,

currently Flink does not support this charset in a LIKE expression. This is due to a limitation in the Apache Calcite library. Maybe you can open an issue there.

The easiest solution for this is to implement your own scalar function, that does a `string.contains("XXXX")`.

Here you can find some explanation how to implement a scalar function: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/udfs.html#scalar-functions

I hope that helps.

Regards,
Timo

Am 25.07.17 um 12:18 schrieb 程骥:
My sql like this(contain a Chinese word)

Get exception when I submit the job to cluster.

Is there anyone tell me how to deal with it,thanks!
Nico Kruber
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How can I set charset for flink sql？

Nico Kruber
In reply to this post by 程骥
Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.

On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!


signature.asc (201 bytes) Download Attachment
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

回复： How can I set charset for flink sql？

程骥
OK，thanks for remind me.

My sql like this(contain a Chinese word):

SELECT
'HIGH' AS LEVEL,
'Firewall uplink bandwidth exception:greater than 10000' AS content,
`system.process.username`,
`system.process.memory.rss.bytes`
FROM
test
WHERE
`system.process.username` LIKE '%高危%'
AND 
`system.process.memory.rss.bytes` > 10000

Get exception when I submit the job to cluster.

Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode '%高危%' in character set 'ISO-8859-1'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.util.NlsString.<init>(NlsString.java:81) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertLiteral(SqlNodeToRexConverterImpl.java:117) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4408) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3787) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4321) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertExpressionList(StandardConvertletTable.java:968) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
... 50 common frames omitted

Is there anyone tell me how to deal with it,thanks!

------------------ 原始邮件 ------------------
发件人: "Nico Kruber";<[hidden email]>;
发送时间: 2017年7月25日(星期二) 晚上11:48
收件人: "user"<[hidden email]>;
抄送: "程骥"<[hidden email]>;
主题: Re: How can I set charset for flink sql？

Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.

On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!
Ted Yu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How can I set charset for flink sql？

Ted Yu
Logged CALCITE-1903 for this bug.

FYI

On Tue, Jul 25, 2017 at 6:39 PM, 程骥 <[hidden email]> wrote:
OK，thanks for remind me.

My sql like this(contain a Chinese word):

SELECT
'HIGH' AS LEVEL,
'Firewall uplink bandwidth exception:greater than 10000' AS content,
`system.process.username`,
`system.process.memory.rss.bytes`
FROM
test
WHERE
`system.process.username` LIKE '%高危%'
AND 
`system.process.memory.rss.bytes` > 10000

Get exception when I submit the job to cluster.

Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode '%高危%' in character set 'ISO-8859-1'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.util.NlsString.<init>(NlsString.java:81) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertLiteral(SqlNodeToRexConverterImpl.java:117) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4408) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3787) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4321) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertExpressionList(StandardConvertletTable.java:968) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
... 50 common frames omitted

Is there anyone tell me how to deal with it,thanks!

------------------ 原始邮件 ------------------
发件人: "Nico Kruber";<[hidden email]>;
发送时间: 2017年7月25日(星期二) 晚上11:48
收件人: "user"<[hidden email]>;
抄送: "程骥"<[hidden email]>;
主题: Re: How can I set charset for flink sql？

Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.

On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How can I set charset for flink sql？

Fabian Hueske-2
As Timo proposed, I would implement a Scalar user-defined function which returns a boolean and use that instead of LIKE.

Have a look here [1].


2017-07-26 3:47 GMT+02:00 Ted Yu <[hidden email]>:
Logged CALCITE-1903 for this bug.

FYI

On Tue, Jul 25, 2017 at 6:39 PM, 程骥 <[hidden email]> wrote:
OK，thanks for remind me.

My sql like this(contain a Chinese word):

SELECT
'HIGH' AS LEVEL,
'Firewall uplink bandwidth exception:greater than 10000' AS content,
`system.process.username`,
`system.process.memory.rss.bytes`
FROM
test
WHERE
`system.process.username` LIKE '%高危%'
AND 
`system.process.memory.rss.bytes` > 10000

Get exception when I submit the job to cluster.

Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode '%高危%' in character set 'ISO-8859-1'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.util.NlsString.<init>(NlsString.java:81) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertLiteral(SqlNodeToRexConverterImpl.java:117) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4408) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3787) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4321) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertExpressionList(StandardConvertletTable.java:968) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
... 50 common frames omitted

Is there anyone tell me how to deal with it,thanks!

------------------ 原始邮件 ------------------
发件人: "Nico Kruber";<[hidden email]>;
发送时间: 2017年7月25日(星期二) 晚上11:48
收件人: "user"<[hidden email]>;
抄送: "程骥"<[hidden email]>;
主题: Re: How can I set charset for flink sql？

Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.

On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!
Free forum by Nabble Edit this page