How to tell between a local mode run vs. remote mode run?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to tell between a local mode run vs. remote mode run?

Yik San Chan
Hi,


When executing jobs in mini cluster(e.g. when executing jobs in IDE) ... please remember to explicitly wait for the job execution to finish as these APIs are asynchronous.

I hope my program will be able to run in both local mode as well as in remote mode. Therefore I hope to do something like:

```python
result = ...
if local_mode:
  result.wait()
else:
  result
```

Is there a way to tell if the program is run under local mode vs. remote mode?

Best,
Yik San
Reply | Threaded
Open this post in threaded view
|

Re: How to tell between a local mode run vs. remote mode run?

Xingbo Huang
Hi Yik San,
You can check whether the execution environment used is `LocalStreamEnvironment` and you can get the class object corresponding to the corresponding java object through py4j in PyFlink. You can take a look at the example I wrote below, I hope it will help you
```
from pyflink.table import EnvironmentSettings, StreamTableEnvironment
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.java_gateway import get_gateway
from py4j.java_gateway import get_java_class


def test():
    env = StreamExecutionEnvironment.get_execution_environment()
    table_env = StreamTableEnvironment.create(
        env, environment_settings=EnvironmentSettings.new_instance()
        .in_streaming_mode().use_blink_planner().build())
    gateway = get_gateway()

    # get the execution environment class
    env_class = table_env._j_tenv.getPlanner().getExecEnv().getClass()

    # get the LocalStreamEnvironment class
    local_stream_environment_class = get_java_class(
        gateway.jvm.org.apache.flink.streaming.api.environment.LocalStreamEnvironment)
    print(env_class == local_stream_environment_class)


if __name__ == '__main__':
    test()

```

Yik San Chan <[hidden email]> 于2021年5月5日周三 下午12:04写道:
Hi,


When executing jobs in mini cluster(e.g. when executing jobs in IDE) ... please remember to explicitly wait for the job execution to finish as these APIs are asynchronous.

I hope my program will be able to run in both local mode as well as in remote mode. Therefore I hope to do something like:

```python
result = ...
if local_mode:
  result.wait()
else:
  result
```

Is there a way to tell if the program is run under local mode vs. remote mode?

Best,
Yik San
Reply | Threaded
Open this post in threaded view
|

Re: How to tell between a local mode run vs. remote mode run?

Yik San Chan
Hi Xingbo,

Thank you!

On Thu, May 6, 2021 at 10:01 AM Xingbo Huang <[hidden email]> wrote:
Hi Yik San,
You can check whether the execution environment used is `LocalStreamEnvironment` and you can get the class object corresponding to the corresponding java object through py4j in PyFlink. You can take a look at the example I wrote below, I hope it will help you
```
from pyflink.table import EnvironmentSettings, StreamTableEnvironment
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.java_gateway import get_gateway
from py4j.java_gateway import get_java_class


def test():
    env = StreamExecutionEnvironment.get_execution_environment()
    table_env = StreamTableEnvironment.create(
        env, environment_settings=EnvironmentSettings.new_instance()
        .in_streaming_mode().use_blink_planner().build())
    gateway = get_gateway()

    # get the execution environment class
    env_class = table_env._j_tenv.getPlanner().getExecEnv().getClass()

    # get the LocalStreamEnvironment class
    local_stream_environment_class = get_java_class(
        gateway.jvm.org.apache.flink.streaming.api.environment.LocalStreamEnvironment)
    print(env_class == local_stream_environment_class)


if __name__ == '__main__':
    test()

```

Yik San Chan <[hidden email]> 于2021年5月5日周三 下午12:04写道:
Hi,


When executing jobs in mini cluster(e.g. when executing jobs in IDE) ... please remember to explicitly wait for the job execution to finish as these APIs are asynchronous.

I hope my program will be able to run in both local mode as well as in remote mode. Therefore I hope to do something like:

```python
result = ...
if local_mode:
  result.wait()
else:
  result
```

Is there a way to tell if the program is run under local mode vs. remote mode?

Best,
Yik San