DataStreamUtils and Scala

[Cory Monty]

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory

[Márton Balassi]

Hey Cory,

Sorry, I did not mean to break your code. One solution that I could suggest is to do it the way we have it for the batch api, namely having a scala version for DataStreamUtils too. It might be placed under flink-contrib for the time being.

Would that solution fit your needs?

On Mon, Feb 1, 2016 at 6:40 PM, Cory Monty <[hidden email]> wrote:

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory

[Stephan Ewen]

I would actually re-add the "getJavaStream()" method. There are probably other cases when people need it, and it does not hurt to expose it.

Also, it allows you to combine programs written partly against the Java and Scala API (if you would ever want to do that).

Stephan

On Mon, Feb 1, 2016 at 6:53 PM, Márton Balassi <[hidden email]> wrote:

Hey Cory,

Sorry, I did not mean to break your code. One solution that I could suggest is to do it the way we have it for the batch api, namely having a scala version for DataStreamUtils too. It might be placed under flink-contrib for the time being.

Would that solution fit your needs?

On Mon, Feb 1, 2016 at 6:40 PM, Cory Monty <[hidden email]> wrote:

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory

[Cory Monty]

Thanks for the quick response!

Either solution works for us. 

On Mon, Feb 1, 2016 at 12:07 PM, Stephan Ewen <[hidden email]> wrote:

I would actually re-add the "getJavaStream()" method. There are probably other cases when people need it, and it does not hurt to expose it.

Also, it allows you to combine programs written partly against the Java and Scala API (if you would ever want to do that).

Stephan

On Mon, Feb 1, 2016 at 6:53 PM, Márton Balassi <[hidden email]> wrote:

Hey Cory,

Sorry, I did not mean to break your code. One solution that I could suggest is to do it the way we have it for the batch api, namely having a scala version for DataStreamUtils too. It might be placed under flink-contrib for the time being.

Would that solution fit your needs?

On Mon, Feb 1, 2016 at 6:40 PM, Cory Monty <[hidden email]> wrote:

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory

[Márton Balassi]

I'll do the fix(es) tomorrow morning.

Best,

Marton

On Mon, Feb 1, 2016, 20:05 Cory Monty <[hidden email]> wrote:

Thanks for the quick response!

Either solution works for us. 

On Mon, Feb 1, 2016 at 12:07 PM, Stephan Ewen <[hidden email]> wrote:

I would actually re-add the "getJavaStream()" method. There are probably other cases when people need it, and it does not hurt to expose it.

Also, it allows you to combine programs written partly against the Java and Scala API (if you would ever want to do that).

Stephan

On Mon, Feb 1, 2016 at 6:53 PM, Márton Balassi <[hidden email]> wrote:

Hey Cory,

Sorry, I did not mean to break your code. One solution that I could suggest is to do it the way we have it for the batch api, namely having a scala version for DataStreamUtils too. It might be placed under flink-contrib for the time being.

Would that solution fit your needs?

On Mon, Feb 1, 2016 at 6:40 PM, Cory Monty <[hidden email]> wrote:

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory

[Márton Balassi]

Opened the PR. [1] Will merge the re-add  "getJavaStream()" method commit as soon as travis passes if no objections, the second approach can be discussed on github.

[1] https://github.com/apache/flink/pull/1574

Best,

Marton

On Mon, Feb 1, 2016 at 10:56 PM, Márton Balassi <[hidden email]> wrote:

I'll do the fix(es) tomorrow morning.

Best,

Marton

On Mon, Feb 1, 2016, 20:05 Cory Monty <[hidden email]> wrote:

Thanks for the quick response!

Either solution works for us. 

On Mon, Feb 1, 2016 at 12:07 PM, Stephan Ewen <[hidden email]> wrote:

I would actually re-add the "getJavaStream()" method. There are probably other cases when people need it, and it does not hurt to expose it.

Also, it allows you to combine programs written partly against the Java and Scala API (if you would ever want to do that).

Stephan

On Mon, Feb 1, 2016 at 6:53 PM, Márton Balassi <[hidden email]> wrote:

Hey Cory,

Sorry, I did not mean to break your code. One solution that I could suggest is to do it the way we have it for the batch api, namely having a scala version for DataStreamUtils too. It might be placed under flink-contrib for the time being.

Would that solution fit your needs?

On Mon, Feb 1, 2016 at 6:40 PM, Cory Monty <[hidden email]> wrote:

Hey there,

We were using DataStreamUtils.collect in Scala for automated testing, which only works because of `DataStream.getJavaStream` accessor in the Scala version of `DataStream`. However, a recent commit (https://github.com/apache/flink/commit/086acf681f01f2da530c04289e0682c56f98a378) removed `getJavaStream` and replaced the method with a private version that we can no longer access from our tests.

Do you have an alternative version of DataStreamUtils that does not require a Java DataStream?

Cheers,

Cory