Flink multiple task managers setup

[saksham sapra]

Hi ,

I am unable to set two task managers in my local machine and neither any documentation provided for the same.

I want to run a parallel job in two task managers using flink.

kindly help me with the same, how can i set up in my local without using any zookeeper or something.

Thanks & Regards,

Saksham Sapra

[Yangze Guo]

Hi,

From my understanding, you want to set up a standalone cluster in your
local machine. If that is the case, you could simply edit the
$FLINK_DIST/conf/workers, in which each line represents a TM host. By
default, there is only one TM in localhost. In your case, you could
add a line 'localhost' to it. Then, execute the
$FLINK_DIST/bin/start-cluster.sh, you could see a standalone cluster
with two TM in your local machine.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html#configuring-flink

Best,
Yangze Guo

On Thu, Sep 17, 2020 at 3:16 PM saksham sapra <[hidden email]> wrote:

>
> Hi ,
>
> I am unable to set two task managers in my local machine and neither any documentation provided for the same.
>
> I want to run a parallel job in two task managers using flink.
> kindly help me with the same, how can i set up in my local without using any zookeeper or something.
>
>
> Thanks & Regards,
> Saksham Sapra
>
>

[Yangze Guo]

Hi,

> I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
Yes, for 1.9.3, you need to edit the 'slaves' file.

I think we need more information to figure out what happened.
- What is the output when you execute ./bin/start-cluster.sh, could
you see two "Starting taskexecutor daemon on host" lines?
- Could you see two flink-xxx-taskexecutor-xxx.log in $FLINK_DIST/log?
If so, could you share these two log files?

Best,
Yangze Guo

Best,
Yangze Guo


On Thu, Sep 17, 2020 at 4:06 PM saksham sapra <[hidden email]> wrote:

>
> Hi Yangze,
>
> Thanks for replying, but i still have some questions.
> I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
>
>
>
>
>
>
> and then again started flink , but i can see only one task manager
>
>
> Please find my config.yaml file attached.
>
>
> Thanks for helping.
>
> Thanks & Regards,
> Saksham Sapra
>
> On Thu, Sep 17, 2020 at 12:57 PM Yangze Guo <[hidden email]> wrote:
>>
>> Hi,
>>
>> From my understanding, you want to set up a standalone cluster in your
>> local machine. If that is the case, you could simply edit the
>> $FLINK_DIST/conf/workers, in which each line represents a TM host. By
>> default, there is only one TM in localhost. In your case, you could
>> add a line 'localhost' to it. Then, execute the
>> $FLINK_DIST/bin/start-cluster.sh, you could see a standalone cluster
>> with two TM in your local machine.
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html#configuring-flink
>>
>> Best,
>> Yangze Guo
>>
>> On Thu, Sep 17, 2020 at 3:16 PM saksham sapra <[hidden email]> wrote:
>> >
>> > Hi ,
>> >
>> > I am unable to set two task managers in my local machine and neither any documentation provided for the same.
>> >
>> > I want to run a parallel job in two task managers using flink.
>> > kindly help me with the same, how can i set up in my local without using any zookeeper or something.
>> >
>> >
>> > Thanks & Regards,
>> > Saksham Sapra
>> >
>> >

[Yangze Guo]

Hi,

It seems you run it in Windows. In that case, only start-cluster.bat
could be used. However, this script could only start one TM[1] no
matter how you configure the slaves/workers.

[1] https://github.com/apache/flink/blob/release-1.9/flink-dist/src/main/flink-bin/bin/start-cluster.bat

Best,
Yangze Guo

On Thu, Sep 17, 2020 at 4:53 PM saksham sapra <[hidden email]> wrote:

>
> HI Yangze,
>
> I tried to run start-cluster.sh and i can see in host , when flink tries to run second task manager or executor, pop up or host gets closed.
> Please find attached logs for two command : start-cluster.sh and start-cluster.bat.
>
> Thanks & Regards,
> Saksham
>
> On Thu, Sep 17, 2020 at 2:00 PM Yangze Guo <[hidden email]> wrote:
>>
>> Hi,
>>
>> > I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
>> Yes, for 1.9.3, you need to edit the 'slaves' file.
>>
>> I think we need more information to figure out what happened.
>> - What is the output when you execute ./bin/start-cluster.sh, could
>> you see two "Starting taskexecutor daemon on host" lines?
>> - Could you see two flink-xxx-taskexecutor-xxx.log in $FLINK_DIST/log?
>> If so, could you share these two log files?
>>
>> Best,
>> Yangze Guo
>>
>> Best,
>> Yangze Guo
>>
>>
>> On Thu, Sep 17, 2020 at 4:06 PM saksham sapra <[hidden email]> wrote:
>> >
>> > Hi Yangze,
>> >
>> > Thanks for replying, but i still have some questions.
>> > I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
>> >
>> >
>> >
>> >
>> >
>> >
>> > and then again started flink , but i can see only one task manager
>> >
>> >
>> > Please find my config.yaml file attached.
>> >
>> >
>> > Thanks for helping.
>> >
>> > Thanks & Regards,
>> > Saksham Sapra
>> >
>> > On Thu, Sep 17, 2020 at 12:57 PM Yangze Guo <[hidden email]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> From my understanding, you want to set up a standalone cluster in your
>> >> local machine. If that is the case, you could simply edit the
>> >> $FLINK_DIST/conf/workers, in which each line represents a TM host. By
>> >> default, there is only one TM in localhost. In your case, you could
>> >> add a line 'localhost' to it. Then, execute the
>> >> $FLINK_DIST/bin/start-cluster.sh, you could see a standalone cluster
>> >> with two TM in your local machine.
>> >>
>> >> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html#configuring-flink
>> >>
>> >> Best,
>> >> Yangze Guo
>> >>
>> >> On Thu, Sep 17, 2020 at 3:16 PM saksham sapra <[hidden email]> wrote:
>> >> >
>> >> > Hi ,
>> >> >
>> >> > I am unable to set two task managers in my local machine and neither any documentation provided for the same.
>> >> >
>> >> > I want to run a parallel job in two task managers using flink.
>> >> > kindly help me with the same, how can i set up in my local without using any zookeeper or something.
>> >> >
>> >> >
>> >> > Thanks & Regards,
>> >> > Saksham Sapra
>> >> >
>> >> >

[Yangze Guo]

Sorry that the community decided to not maintain it anymore, you could
take a look at [1].

[1] https://lists.apache.org/thread.html/r7693d0c06ac5ced9a34597c662bcf37b34ef8e799c32cc0edee373b2%40%3Cdev.flink.apache.org%3E

Best,
Yangze Guo

On Thu, Sep 17, 2020 at 5:21 PM saksham sapra <[hidden email]> wrote:

>
> Thanks Yangze, So should i raise a JIRA Ticket for the same on the flink community group?
>
> Thanks & Regards,
> Saksham
>
> On Thu, Sep 17, 2020 at 2:38 PM Yangze Guo <[hidden email]> wrote:
>>
>> Hi,
>>
>> It seems you run it in Windows. In that case, only start-cluster.bat
>> could be used. However, this script could only start one TM[1] no
>> matter how you configure the slaves/workers.
>>
>> [1] https://github.com/apache/flink/blob/release-1.9/flink-dist/src/main/flink-bin/bin/start-cluster.bat
>>
>> Best,
>> Yangze Guo
>>
>> On Thu, Sep 17, 2020 at 4:53 PM saksham sapra <[hidden email]> wrote:
>> >
>> > HI Yangze,
>> >
>> > I tried to run start-cluster.sh and i can see in host , when flink tries to run second task manager or executor, pop up or host gets closed.
>> > Please find attached logs for two command : start-cluster.sh and start-cluster.bat.
>> >
>> > Thanks & Regards,
>> > Saksham
>> >
>> > On Thu, Sep 17, 2020 at 2:00 PM Yangze Guo <[hidden email]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> > I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
>> >> Yes, for 1.9.3, you need to edit the 'slaves' file.
>> >>
>> >> I think we need more information to figure out what happened.
>> >> - What is the output when you execute ./bin/start-cluster.sh, could
>> >> you see two "Starting taskexecutor daemon on host" lines?
>> >> - Could you see two flink-xxx-taskexecutor-xxx.log in $FLINK_DIST/log?
>> >> If so, could you share these two log files?
>> >>
>> >> Best,
>> >> Yangze Guo
>> >>
>> >> Best,
>> >> Yangze Guo
>> >>
>> >>
>> >> On Thu, Sep 17, 2020 at 4:06 PM saksham sapra <[hidden email]> wrote:
>> >> >
>> >> > Hi Yangze,
>> >> >
>> >> > Thanks for replying, but i still have some questions.
>> >> > I wasnt having "workers" file in conf/workers so i created one, but i have "slaves" file in  conf/workers, so i edited both two localhost like screenshot given below :
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > and then again started flink , but i can see only one task manager
>> >> >
>> >> >
>> >> > Please find my config.yaml file attached.
>> >> >
>> >> >
>> >> > Thanks for helping.
>> >> >
>> >> > Thanks & Regards,
>> >> > Saksham Sapra
>> >> >
>> >> > On Thu, Sep 17, 2020 at 12:57 PM Yangze Guo <[hidden email]> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> From my understanding, you want to set up a standalone cluster in your
>> >> >> local machine. If that is the case, you could simply edit the
>> >> >> $FLINK_DIST/conf/workers, in which each line represents a TM host. By
>> >> >> default, there is only one TM in localhost. In your case, you could
>> >> >> add a line 'localhost' to it. Then, execute the
>> >> >> $FLINK_DIST/bin/start-cluster.sh, you could see a standalone cluster
>> >> >> with two TM in your local machine.
>> >> >>
>> >> >> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html#configuring-flink
>> >> >>
>> >> >> Best,
>> >> >> Yangze Guo
>> >> >>
>> >> >> On Thu, Sep 17, 2020 at 3:16 PM saksham sapra <[hidden email]> wrote:
>> >> >> >
>> >> >> > Hi ,
>> >> >> >
>> >> >> > I am unable to set two task managers in my local machine and neither any documentation provided for the same.
>> >> >> >
>> >> >> > I want to run a parallel job in two task managers using flink.
>> >> >> > kindly help me with the same, how can i set up in my local without using any zookeeper or something.
>> >> >> >
>> >> >> >
>> >> >> > Thanks & Regards,
>> >> >> > Saksham Sapra
>> >> >> >
>> >> >> >

[saksham sapra]

HI,

 i installed cygdrive and tried to run start-cluster.sh where zookeeper is up and running and defined one job manager and one task manager,

but getting this issue. 

$ start-cluster.sh start
Starting HA cluster with 1 masters.
-zTheFIND: Invalid switch
 system cannot find the file specified.
[ERROR] Flink distribution jar not found in /cygdrive/d/Apacheflink/dist/apache-flink-1.9.3/deps/lib.
File not found - D:\Apacheflink\dist\apache-flink-1.9.3\deps/log/flink--standalo
nesession-7-PLRENT-5LC73H2*
Starting standalonesession daemon on host PLRENT-5LC73H2.
-zThe system cannot FIND: Invalid switch
find the file specified.
[ERROR] Flink distribution jar not found in /cygdrive/d/Apacheflink/dist/apache-flink-1.9.3/deps/lib.
File not found - D:\Apacheflink\dist\apache-flink-1.9.3\deps/log/flink--taskexec
utor-7-PLRENT-5LC73H2*
Starting taskexecutor daemon on host PLRENT-5LC73H2.

[Yangze Guo]

Hi,

As the error message said, it could not find the flink-dist.jar in
"/cygdrive/d/Apacheflink/dist/apache-flink-1.9.3/deps/lib". Where is
your flink distribution and do you change the directory structure of
it?

Best,
Yangze Guo

On Mon, Sep 21, 2020 at 5:31 PM saksham sapra <[hidden email]> wrote:

>
> HI,
>
>  i installed cygdrive and tried to run start-cluster.sh where zookeeper is up and running and defined one job manager and one task manager,
> but getting this issue.
>
> $ start-cluster.sh start
> Starting HA cluster with 1 masters.
> -zTheFIND: Invalid switch
>  system cannot find the file specified.
> [ERROR] Flink distribution jar not found in /cygdrive/d/Apacheflink/dist/apache-flink-1.9.3/deps/lib.
> File not found - D:\Apacheflink\dist\apache-flink-1.9.3\deps/log/flink--standalo
> nesession-7-PLRENT-5LC73H2*
> Starting standalonesession daemon on host PLRENT-5LC73H2.
> -zThe system cannot FIND: Invalid switch
> find the file specified.
> [ERROR] Flink distribution jar not found in /cygdrive/d/Apacheflink/dist/apache-flink-1.9.3/deps/lib.
> File not found - D:\Apacheflink\dist\apache-flink-1.9.3\deps/log/flink--taskexec
> utor-7-PLRENT-5LC73H2*
> Starting taskexecutor daemon on host PLRENT-5LC73H2.
>>>
>>>