Using collect and accessing accumulator results

[Tamara Mendt]

Hey!

I am currently running a job in which I wish to use collect to trigger my job execution, but I also need to have access to the final accumulator results. Up until now I have been accessing the accumulator results through the JobExecutionResult that the function execute() returns.

Not surprisingly, if I use collect() and execute() I get the following exception:

java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:921)
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:904)
    at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:50)

Is there some way in which I can access the accumulator results while using the collect method to trigger execution?

Cheers,

[Maximilian Michels]

Hi Tamara!

Yes, there is. Since count/collect/print trigger an execution of the ExecutionEnvironment, you can get the result afterwards using env.getLastExecutionResult().

Best,

Max

On Thu, Jun 18, 2015 at 3:57 PM, Tamara Mendt <[hidden email]> wrote:

Hey!

I am currently running a job in which I wish to use collect to trigger my job execution, but I also need to have access to the final accumulator results. Up until now I have been accessing the accumulator results through the JobExecutionResult that the function execute() returns.

Not surprisingly, if I use collect() and execute() I get the following exception:

java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:921)
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:904)
    at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:50)

Is there some way in which I can access the accumulator results while using the collect method to trigger execution?

Cheers,

[Ufuk Celebi]

Should we add this to the Javadoc of the eagerly executed operations?

On 18 Jun 2015, at 16:11, Maximilian Michels <[hidden email]> wrote:

> Hi Tamara!
>
> Yes, there is. Since count/collect/print trigger an execution of the ExecutionEnvironment, you can get the result afterwards using env.getLastExecutionResult().
>
> Best,
> Max
>
> On Thu, Jun 18, 2015 at 3:57 PM, Tamara Mendt <[hidden email]> wrote:
> Hey!
>
> I am currently running a job in which I wish to use collect to trigger my job execution, but I also need to have access to the final accumulator results. Up until now I have been accessing the accumulator results through the JobExecutionResult that the function execute() returns.
>
> Not surprisingly, if I use collect() and execute() I get the following exception:
>
> java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:921)
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:904)
>     at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:50)
>
>
> Is there some way in which I can access the accumulator results while using the collect method to trigger execution?
>
> Cheers,
>

[Tamara Mendt]

Great, thanks!

On Thu, Jun 18, 2015 at 4:16 PM, Ufuk Celebi <[hidden email]> wrote:

Should we add this to the Javadoc of the eagerly executed operations?

On 18 Jun 2015, at 16:11, Maximilian Michels <[hidden email]> wrote:

> Hi Tamara!
>
> Yes, there is. Since count/collect/print trigger an execution of the ExecutionEnvironment, you can get the result afterwards using env.getLastExecutionResult().
>
> Best,
> Max
>
> On Thu, Jun 18, 2015 at 3:57 PM, Tamara Mendt <[hidden email]> wrote:
> Hey!
>
> I am currently running a job in which I wish to use collect to trigger my job execution, but I also need to have access to the final accumulator results. Up until now I have been accessing the accumulator results through the JobExecutionResult that the function execute() returns.
>
> Not surprisingly, if I use collect() and execute() I get the following exception:
>
> java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:921)
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:904)
>     at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:50)
>
>
> Is there some way in which I can access the accumulator results while using the collect method to trigger execution?
>
> Cheers,
>

--

[Aljoscha Krettek]

@Ufuk, probably should. yes.

On Thu, 18 Jun 2015 at 16:18 Tamara Mendt <[hidden email]> wrote:

Great, thanks!

On Thu, Jun 18, 2015 at 4:16 PM, Ufuk Celebi <[hidden email]> wrote:

Should we add this to the Javadoc of the eagerly executed operations?

On 18 Jun 2015, at 16:11, Maximilian Michels <[hidden email]> wrote:

> Hi Tamara!
>
> Yes, there is. Since count/collect/print trigger an execution of the ExecutionEnvironment, you can get the result afterwards using env.getLastExecutionResult().
>
> Best,
> Max
>
> On Thu, Jun 18, 2015 at 3:57 PM, Tamara Mendt <[hidden email]> wrote:
> Hey!
>
> I am currently running a job in which I wish to use collect to trigger my job execution, but I also need to have access to the final accumulator results. Up until now I have been accessing the accumulator results through the JobExecutionResult that the function execute() returns.
>
> Not surprisingly, if I use collect() and execute() I get the following exception:
>
> java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:921)
>     at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:904)
>     at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:50)
>
>
> Is there some way in which I can access the accumulator results while using the collect method to trigger execution?
>
> Cheers,
>

--

Tamara Mendt