Unexpected behavior with Scala App trait.

[Andrea Sella]

Hi,

I was implementing TF-IDF example of flink-training when I faced a problem with NPE during the deploy of my Job.

Source code: https://github.com/alkagin/flink-tfidf-example

I used 0.10.1 version and started in local mode.

During the deploy of TFIDFNPE Job, which it extends App, Flink throws NullPointerException on both flatMap functions.
If I include the tokenize function into the closures of flatMap functions, the Job works fine; see example TFIDFApp.

To avoid this unexpected behavior I don't have use Scala App trait, see TFIDF, but why?


Thanks,
Andrea

[Chiwan Park-2]

Hi Andrea,

I’m not expert of Scala but It seems about closure cleaning problem. Scala App trait extends DelayedInit trait to initialize object. But Flink serialization stack doesn’t handle this special initialization. (It is just my opinion, not verified.)

To run TFIDFNPE safely, you need to just change tokenize and uniqueWords to method like following:

```
def tokenize = …
def uniqueWords = ...
```

With this change, I tested that TFIDFNPE works safely in Flink 0.10.1 cluster.

About TFIDF object, you should avoid overriding main method if the object is derived by App trait. It is also related DelaytedInit mechanism [1].

[1]: https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31

> On Jan 19, 2016, at 12:08 AM, Andrea Sella <[hidden email]> wrote:
>
> Hi,
>
> I was implementing TF-IDF example of flink-training when I faced a problem with NPE during the deploy of my Job.
>
> Source code: https://github.com/alkagin/flink-tfidf-example
>
> I used 0.10.1 version and started in local mode.
> During the deploy of TFIDFNPE Job, which it extends App, Flink throws NullPointerException on both flatMap functions.
> If I include the tokenize function into the closures of flatMap functions, the Job works fine; see example TFIDFApp.
> To avoid this unexpected behavior I don't have use Scala App trait, see TFIDF, but why?
>
>
> Thanks,
> Andrea
>

Regards,
Chiwan Park

[Andrea Sella]

Hi Chiwan,

I’m not expert of Scala but It seems about closure cleaning problem. Scala App trait extends DelayedInit trait to initialize object. But Flink serialization stack doesn’t handle this special initialization. (It is just my opinion, not verified.)

I arrived at the same conclusion with the DelayedInit trait. 

To run TFIDFNPE safely, you need to just change tokenize and uniqueWords to method like following:

```
def tokenize = …
def uniqueWords = ...
```

With this change, I tested that TFIDFNPE works safely in Flink 0.10.1 cluster.

Yeah, it works and i knew it. My aim it was to use tricky (functional) Scala mechanisms to test how much the Scala APIs are robust and idomitic-friendly for Scala users.

About TFIDF object, you should avoid overriding main method if the object is derived by App trait. It is also related DelaytedInit mechanism [1].

Yeah, App trait is into a comment 'cause I am overriding the main method. TFIDF is equal to TFIDFNPE with App trait and not overriding main method.


Thanks,
Andrea

[1]: https://github.com/scala/scala/blob/2.10.x/src/library/scala/App.scala#L31

> On Jan 19, 2016, at 12:08 AM, Andrea Sella <[hidden email]> wrote:
>
> Hi,
>
> I was implementing TF-IDF example of flink-training when I faced a problem with NPE during the deploy of my Job.
>
> Source code: https://github.com/alkagin/flink-tfidf-example
>
> I used 0.10.1 version and started in local mode.
> During the deploy of TFIDFNPE Job, which it extends App, Flink throws NullPointerException on both flatMap functions.
> If I include the tokenize function into the closures of flatMap functions, the Job works fine; see example TFIDFApp.
> To avoid this unexpected behavior I don't have use Scala App trait, see TFIDF, but why?
>
>
> Thanks,
> Andrea
>

Regards,
Chiwan Park