Re: Logs meaning states

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Logs-meaning-states-tp1829p1830.html

Hi Juan!

This is an artifact of a workaround right now. The actual collect() logic happens in the flatMap() and the sink is a dummy that executes nothing. The flatMap writes the data to be collected to the "accumulator" that delivers it back.

Greetings,

Stephan

On Mon, Jun 29, 2015 at 2:30 PM, Juan Fumero <[hidden email]> wrote:

Hi,
  I am starting with Flink. I have tried to look for the documentation but I havent found it clear. 

I wonder the difference between these two states:

FlatMap RUNNING vs DataSink RUNNIG.

FlatMap is doing data any data transformation? Compilation? In which point is actually executing the function provided in the MapFunction? How could I know exactly the time for the kernel computation?

It seems is using one thread in this step, even though I specified 16 threads in the createLocalEnvironment.

CHAIN DataSource (at applyFunction(ApplyFunction.java:96) (org.apache.flink.api.java.io.CollectionInputFormat)) -> Map (Map at applyFunction(ApplyFunction.java:108)) -> FlatMap (collect())(1/1) switched to RUNNING

Here is running only one thread for almost 35 seconds.

The rest of the execution is very fast (less than one second for computing the square of an array of 500000 integer  elements)

Thanks
Juan

Here the full log.

06/29/2015 14:13:25 Job execution switched to status RUNNING.
06/29/2015 14:13:25 CHAIN DataSource (at applyFunction(ApplyFunction.java:96) (org.apache.flink.api.java.io.CollectionInputFormat)) -> Map (Map at applyFunction(ApplyFunction.java:108)) -> FlatMap (collect())(1/1) switched to SCHEDULED
06/29/2015 14:13:25 CHAIN DataSource (at applyFunction(ApplyFunction.java:96) (org.apache.flink.api.java.io.CollectionInputFormat)) -> Map (Map at applyFunction(ApplyFunction.java:108)) -> FlatMap (collect())(1/1) switched to DEPLOYING
06/29/2015 14:13:26 CHAIN DataSource (at applyFunction(ApplyFunction.java:96) (org.apache.flink.api.java.io.CollectionInputFormat)) -> Map (Map at applyFunction(ApplyFunction.java:108)) -> FlatMap (collect())(1/1) switched to RUNNING
06/29/2015 14:14:01 DataSink (collect() sink)(1/1) switched to SCHEDULED
06/29/2015 14:14:01 CHAIN DataSource (at applyFunction(ApplyFunction.java:96) (org.apache.flink.api.java.io.CollectionInputFormat)) -> Map (Map at applyFunction(ApplyFunction.java:108)) -> FlatMap (collect())(1/1) switched to FINISHED
06/29/2015 14:14:01 DataSink (collect() sink)(1/1) switched to DEPLOYING
06/29/2015 14:14:01 DataSink (collect() sink)(1/1) switched to RUNNING
06/29/2015 14:14:01 DataSink (collect() sink)(1/1) switched to FINISHED
06/29/2015 14:14:01 Job execution switched to status FINISHED.