Using RemoteExecutionEnvironment from IDE

[Benjamin Burkhardt-2]

Hi all,

I’m currently debugging and enjoy it executing my task directly from the IDE.

The IDE starts a LocalExecutionEnvironment and immediately runs the task.

Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)

To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.

mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1

I know you can send the task to a running env like this:

final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);

But thats not what I’m looking for….

Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?

Thanks.

Benjamin

[Andrey Zagrebin-3]

Hi Benjamin,

what kind of problem with RemoteInputChannels are you debugging?

If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.

Best,

Andrey

[1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters

On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <[hidden email]> wrote:

Hi all,

I’m currently debugging and enjoy it executing my task directly from the IDE.

The IDE starts a LocalExecutionEnvironment and immediately runs the task.

Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)

To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.

mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1

I know you can send the task to a running env like this:

final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);

But thats not what I’m looking for….

Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?

Thanks.

Benjamin

[Benjamin Burkhardt-2]

Hi Andrey,

I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run.

Attaching to the IDE is pretty helpful, I already tried that before.

But I think that there is just no solution for this and I have rerun the environment manually.

Thanks.

Benjamin

Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <[hidden email]>:

Hi Benjamin,

what kind of problem with RemoteInputChannels are you debugging?

If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.

Best,

Andrey

[1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters

On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <[hidden email]> wrote:

Hi all,

I’m currently debugging and enjoy it executing my task directly from the IDE.

The IDE starts a LocalExecutionEnvironment and immediately runs the task.

Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)

To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.

mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1

I know you can send the task to a running env like this:

final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);

But thats not what I’m looking for….

Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?

Thanks.

Benjamin

[Till Rohrmann]

Hi Benjamin,

what do you mean with "start a RemoteExecutionEnvironment from the IDE from scratch"? Are you asking whether you can restart the cluster against which the RemoteExecutionEnvironment submits the job? This is not possible.

If you want to debug `RemoteInputChannels` locally, you could use the MiniClusterWithClientResource to start a MiniCluster with more than 1 TaskExecutor. That way it would instantiate the RemoteInputChannel in order to communicate between the different TaskExecutors. The resource will automatically set a special TestStreamEnvironment such that jobs will be submitted against the MiniCluster.

public class FoobarITCase {

@ClassRule

public static final MiniClusterWithClientResource MINI_CLUSTER = new MiniClusterWithClientResource(

new MiniClusterResourceConfiguration.Builder()

.setNumberTaskManagers(2)

.setNumberSlotsPerTaskManager(2)

.build());

@Test

public void testFoobar() {

MyApplication.main(new String[0]);

}

}

The class MiniClusterWithClientResource lives in the flink-test-utils package.

Cheers,

Till

On Fri, Feb 15, 2019 at 10:29 AM Benjamin Burkhardt <[hidden email]> wrote:

Hi Andrey,

I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run.

Attaching to the IDE is pretty helpful, I already tried that before.

But I think that there is just no solution for this and I have rerun the environment manually.

Thanks.

Benjamin

Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <[hidden email]>:

Hi Benjamin,

what kind of problem with RemoteInputChannels are you debugging?

If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.

Best,

Andrey

[1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters

On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <[hidden email]> wrote:

Hi all,

I’m currently debugging and enjoy it executing my task directly from the IDE.

The IDE starts a LocalExecutionEnvironment and immediately runs the task.

Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)

To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.

mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1

I know you can send the task to a running env like this:

final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);

But thats not what I’m looking for….

Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?

Thanks.

Benjamin

[Benjamin Burkhardt-2]

Hi Till,

thanks for the great explanation! This was exactly what I was looking for.

Have a great day!

Regards,

Benjamin

Am 15. Feb. 2019, 10:55 +0100 schrieb Till Rohrmann <[hidden email]>:

Hi Benjamin,

what do you mean with "start a RemoteExecutionEnvironment from the IDE from scratch"? Are you asking whether you can restart the cluster against which the RemoteExecutionEnvironment submits the job? This is not possible.

If you want to debug `RemoteInputChannels` locally, you could use the MiniClusterWithClientResource to start a MiniCluster with more than 1 TaskExecutor. That way it would instantiate the RemoteInputChannel in order to communicate between the different TaskExecutors. The resource will automatically set a special TestStreamEnvironment such that jobs will be submitted against the MiniCluster.

public class FoobarITCase {

@ClassRule

public static final MiniClusterWithClientResource MINI_CLUSTER = new MiniClusterWithClientResource(

new MiniClusterResourceConfiguration.Builder()

.setNumberTaskManagers(2)

.setNumberSlotsPerTaskManager(2)

.build());

@Test

public void testFoobar() {

MyApplication.main(new String[0]);

}

}

The class MiniClusterWithClientResource lives in the flink-test-utils package.

Cheers,

Till

On Fri, Feb 15, 2019 at 10:29 AM Benjamin Burkhardt <[hidden email]> wrote:

Hi Andrey,

I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run.

Attaching to the IDE is pretty helpful, I already tried that before.

But I think that there is just no solution for this and I have rerun the environment manually.

Thanks.

Benjamin

Am 13. Feb. 2019, 17:57 +0100 schrieb Andrey Zagrebin <[hidden email]>:

Hi Benjamin,

what kind of problem with RemoteInputChannels are you debugging?

If the problem is in a Task Manager, you can try enabling IDE debug in flink-conf.yaml [1] before starting the task manager and then attach IDE to the JVM process of the Task Manager.

Best,

Andrey

[1] https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters

On Wed, Feb 13, 2019 at 12:15 PM Benjamin Burkhardt <[hidden email]> wrote:

Hi all,

I’m currently debugging and enjoy it executing my task directly from the IDE.

The IDE starts a LocalExecutionEnvironment and immediately runs the task.

Now my errors seem to only appear when executing in the RemoteExecutionEnvironment. (it’s around the RemoteInputChannels et. al.)

To run it and test my modifications I always need to recompile it with maven which takes about 2.5 minutes.

mvn install -Dcheckstyle.skip -Drat.skip=true -DskipTests -T 1

I know you can send the task to a running env like this:

final ExecutionEnvironment env = ExecutionEnvironment.createRemoteEnvironment("localhost", 8081, „target/task.jar“);

But thats not what I’m looking for….

Is there a way to start a RemoteExecutionEnvironment from the IDE from scratch?

Thanks.

Benjamin