Safer handling of Scala immutable collections

[Rex Fenley]

Hello,

I've been playing with UDFs using the Scala API and have repeatedly run into issues such as this:

```

flink-taskmanager_1    | java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$ cannot be cast to [J

```

Is there something that can be done on Flink's end, either to catch these errors in type checking or to cast them in a sane manner during runtime?

Thanks!

--

[Rex Fenley]

While trying to use a mutable.Set and later .asJava I receive the following

flink-jobmanager_1     | Caused by: java.lang.ClassCastException: scala.collection.convert.Wrappers$MutableSetWrapper cannot be cast to [J
flink-jobmanager_1     | at org.apache.flink.table.data.util.DataFormatConverters$PrimitiveLongArrayConverter.toInternalImpl(DataFormatConverters.java:1014) ~[flink-table-blink_2.12-1.11.1.jar:1.11.1]

...

On Wed, Oct 14, 2020 at 2:54 PM Rex Fenley <[hidden email]> wrote:

Hello,

I've been playing with UDFs using the Scala API and have repeatedly run into issues such as this:

```

flink-taskmanager_1    | java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$ cannot be cast to [J

```

Is there something that can be done on Flink's end, either to catch these errors in type checking or to cast them in a sane manner during runtime?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Jeff Zhang]

Could you share your code to reproduce it ?

Rex Fenley <[hidden email]> 于2020年10月15日周四 上午5:54写道：

Hello,

I've been playing with UDFs using the Scala API and have repeatedly run into issues such as this:

```

flink-taskmanager_1    | java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$ cannot be cast to [J

```

Is there something that can be done on Flink's end, either to catch these errors in type checking or to cast them in a sane manner during runtime?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Rex Fenley]

I believe I found the issue:

new RowTypeInfo(Types.PRIMITIVE_ARRAY(Types.LONG()))

vs

new RowTypeInfo(createTypeInformation[Array[Long]])

I didn't quite understand at the time whose type information I was meant to supply, now I do.

However, I think my question still stands. Is there a way for this information to be more directly encoded by scala api's types so this kind of confusion never happens?

E.g. If my UDF is returning an Array of Longs then for what reason should I ever need to then specify that directly from within getResultType?

On Wed, Oct 14, 2020 at 4:06 PM Jeff Zhang <[hidden email]> wrote:

Could you share your code to reproduce it ?

Rex Fenley <[hidden email]> 于2020年10月15日周四 上午5:54写道：

Hello,

I've been playing with UDFs using the Scala API and have repeatedly run into issues such as this:

```

flink-taskmanager_1    | java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$ cannot be cast to [J

```

Is there something that can be done on Flink's end, either to catch these errors in type checking or to cast them in a sane manner during runtime?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Best Regards

Jeff Zhang

--

[Rex Fenley]

(Or small correction; a Row with a column of Array of Longs, but still)

On Wed, Oct 14, 2020 at 4:46 PM Rex Fenley <[hidden email]> wrote:

I believe I found the issue:

new RowTypeInfo(Types.PRIMITIVE_ARRAY(Types.LONG()))

vs

new RowTypeInfo(createTypeInformation[Array[Long]])

I didn't quite understand at the time whose type information I was meant to supply, now I do.

However, I think my question still stands. Is there a way for this information to be more directly encoded by scala api's types so this kind of confusion never happens?

E.g. If my UDF is returning an Array of Longs then for what reason should I ever need to then specify that directly from within getResultType?

On Wed, Oct 14, 2020 at 4:06 PM Jeff Zhang <[hidden email]> wrote:

Could you share your code to reproduce it ?

Rex Fenley <[hidden email]> 于2020年10月15日周四 上午5:54写道：

Hello,

I've been playing with UDFs using the Scala API and have repeatedly run into issues such as this:

```

flink-taskmanager_1    | java.lang.ClassCastException: scala.collection.immutable.Set$EmptySet$ cannot be cast to [J

```

Is there something that can be done on Flink's end, either to catch these errors in type checking or to cast them in a sane manner during runtime?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Best Regards

Jeff Zhang

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--