How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode

[范超]

Hi, Dev and Users
I’ve 3 machines each one is 8 cores and 16GB memory.
Following it’s my Resource Manager screenshot the cluster have 36GB total.
I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
I tried set the �Cp �Ctm �Cjm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
My question is how to set the cli parameter to start all of my three machine (all task manager start on 3 machines)

Thanks a lot
[cid:image001.png@01D67546.62291B70]


Chao fan

[Yangze Guo]

Hi,

Do you start the NodeManager in all the three machines? If so, could
you check all the NMs correctly connect to the ResourceManager?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:

>
> Hi, Dev and Users
> I’ve 3 machines each one is 8 cores and 16GB memory.
> Following it’s my Resource Manager screenshot the cluster have 36GB total.
> I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> My question is how to set the cli parameter to start all of my three machine (all task manager start on 3 machines)
>
> Thanks a lot
> [cid:image001.png@01D67546.62291B70]
>
>
> Chao fan
>

[范超]

Thanks Yangze

All 3 machines NodeManager is started.

I just don't know why not three machines each running a Flink TaskManager and how to achieve this

-----邮件原件-----
发件人: Yangze Guo [mailto:[hidden email]]
发送时间: 2020年8月18日 星期二 10:10
收件人: 范超 <[hidden email]>
抄送: user ([hidden email]) <[hidden email]>
主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode

Hi,

Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:

>
> Hi, Dev and Users
> I’ve 3 machines each one is 8 cores and 16GB memory.
> Following it’s my Resource Manager screenshot the cluster have 36GB total.
> I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> My question is how to set the cli parameter to start all of my three
> machine (all task manager start on 3 machines)
>
> Thanks a lot
> [cid:image001.png@01D67546.62291B70]
>
>
> Chao fan
>

[Yangze Guo]

Hi,

I think that is only related to the Yarn scheduling strategy. AFAIK,
Flink could not control it. You could check the RM log to figure out
why it did not schedule the containers to all the three machines. BTW,
if you have specific requirements to start with all the three
machines, how about deploying a standalone cluster instead?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
>
> All 3 machines NodeManager is started.
>
> I just don't know why not three machines each running a Flink TaskManager and how to achieve this
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 10:10
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode
>
> Hi,
>
> Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> >
> > Hi, Dev and Users
> > I’ve 3 machines each one is 8 cores and 16GB memory.
> > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > My question is how to set the cli parameter to start all of my three
> > machine (all task manager start on 3 machines)
> >
> > Thanks a lot
> > [cid:image001.png@01D67546.62291B70]
> >
> >
> > Chao fan
> >

[范超]

Thanks Yangze
The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me.
If Flink can not control how many tm to start , could anyone providing me some best practice for deploying on yarn please? I read the [1] and still don't very clear

[1] https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-general-guidelines

-----邮件原件-----
发件人: Yangze Guo [mailto:[hidden email]]
发送时间: 2020年8月18日 星期二 10:50
收件人: 范超 <[hidden email]>
抄送: user ([hidden email]) <[hidden email]>
主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode

Hi,

I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
>
> All 3 machines NodeManager is started.
>
> I just don't know why not three machines each running a Flink
> TaskManager and how to achieve this
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 10:10
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> using Per-Job Mode
>
> Hi,
>
> Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> >
> > Hi, Dev and Users
> > I’ve 3 machines each one is 8 cores and 16GB memory.
> > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > My question is how to set the cli parameter to start all of my three
> > machine (all task manager start on 3 machines)
> >
> > Thanks a lot
> > [cid:image001.png@01D67546.62291B70]
> >
> >
> > Chao fan
> >

[Yangze Guo]

Hi,

Flink can control how many TM to start, but where to start the TMs
depends on Yarn.

Do you meet any problem when deploying on Yarn or running Flink job?
Why do you need to start the TMs on all the three machines?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 11:25 AM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
> The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me.
> If Flink can not control how many tm to start , could anyone providing me some best practice for deploying on yarn please? I read the [1] and still don't very clear
>
> [1] https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-general-guidelines
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 10:50
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode
>
> Hi,
>
> I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:
> >
> > Thanks Yangze
> >
> > All 3 machines NodeManager is started.
> >
> > I just don't know why not three machines each running a Flink
> > TaskManager and how to achieve this
> >
> > -----邮件原件-----
> > 发件人: Yangze Guo [mailto:[hidden email]]
> > 发送时间: 2020年8月18日 星期二 10:10
> > 收件人: 范超 <[hidden email]>
> > 抄送: user ([hidden email]) <[hidden email]>
> > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > using Per-Job Mode
> >
> > Hi,
> >
> > Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> > >
> > > Hi, Dev and Users
> > > I’ve 3 machines each one is 8 cores and 16GB memory.
> > > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > > My question is how to set the cli parameter to start all of my three
> > > machine (all task manager start on 3 machines)
> > >
> > > Thanks a lot
> > > [cid:image001.png@01D67546.62291B70]
> > >
> > >
> > > Chao fan
> > >

[范超]

Thanks Yangze

1. Do you meet any problem when deploying on Yarn or running Flink job?
My job works well

2. Why do you need to start the TMs on all the three machines?
From cluster perspective, I wonder if the process pressure can be balance to 3 machines.

3. Flink can control how many TM to start, but where to start the TMs depends on Yarn.
Yes, the job where to start the TM is depend on Yarn.
Could you please tell me parameter controls how many TM to start, the yn parameter is delete from 1.10 as the 1.9 doc sample list[1] below

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html

Run example program using a per-job YARN cluster with 2 TaskManagers:

./bin/flink run -m yarn-cluster -yn 2 \
                       ./examples/batch/WordCount.jar \
                       --input hdfs:///user/hamlet.txt --output hdfs:///user/wordcount_out

-----邮件原件-----
发件人: Yangze Guo [mailto:[hidden email]]
发送时间: 2020年8月18日 星期二 11:31
收件人: 范超 <[hidden email]>
抄送: user ([hidden email]) <[hidden email]>
主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode

Hi,

Flink can control how many TM to start, but where to start the TMs depends on Yarn.

Do you meet any problem when deploying on Yarn or running Flink job?
Why do you need to start the TMs on all the three machines?

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 11:25 AM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
> The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me.
> If Flink can not control how many tm to start , could anyone providing
> me some best practice for deploying on yarn please? I read the [1] and
> still don't very clear
>
> [1]
> https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-g
> eneral-guidelines
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 10:50
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> using Per-Job Mode
>
> Hi,
>
> I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:
> >
> > Thanks Yangze
> >
> > All 3 machines NodeManager is started.
> >
> > I just don't know why not three machines each running a Flink
> > TaskManager and how to achieve this
> >
> > -----邮件原件-----
> > 发件人: Yangze Guo [mailto:[hidden email]]
> > 发送时间: 2020年8月18日 星期二 10:10
> > 收件人: 范超 <[hidden email]>
> > 抄送: user ([hidden email]) <[hidden email]>
> > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > using Per-Job Mode
> >
> > Hi,
> >
> > Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> > >
> > > Hi, Dev and Users
> > > I’ve 3 machines each one is 8 cores and 16GB memory.
> > > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > > My question is how to set the cli parameter to start all of my
> > > three machine (all task manager start on 3 machines)
> > >
> > > Thanks a lot
> > > [cid:image001.png@01D67546.62291B70]
> > >
> > >
> > > Chao fan
> > >

[Yangze Guo]

The number of TM mainly depends on the parallelism and job graph.
Flink now allows you to set the maximum slots number
(slotmanager-number-of-slots-max[1]). There is also a plan to support
setting the minimum number of slots[2].

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#slotmanager-number-of-slots-max
[2] https://issues.apache.org/jira/browse/FLINK-15959

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 12:21 PM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
>
> 1. Do you meet any problem when deploying on Yarn or running Flink job?
> My job works well
>
> 2. Why do you need to start the TMs on all the three machines?
> From cluster perspective, I wonder if the process pressure can be balance to 3 machines.
>
> 3. Flink can control how many TM to start, but where to start the TMs depends on Yarn.
> Yes, the job where to start the TM is depend on Yarn.
> Could you please tell me parameter controls how many TM to start, the yn parameter is delete from 1.10 as the 1.9 doc sample list[1] below
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html
>
> Run example program using a per-job YARN cluster with 2 TaskManagers:
>
> ./bin/flink run -m yarn-cluster -yn 2 \
>                        ./examples/batch/WordCount.jar \
>                        --input hdfs:///user/hamlet.txt --output hdfs:///user/wordcount_out
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 11:31
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode
>
> Hi,
>
> Flink can control how many TM to start, but where to start the TMs depends on Yarn.
>
> Do you meet any problem when deploying on Yarn or running Flink job?
> Why do you need to start the TMs on all the three machines?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 11:25 AM 范超 <[hidden email]> wrote:
> >
> > Thanks Yangze
> > The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me.
> > If Flink can not control how many tm to start , could anyone providing
> > me some best practice for deploying on yarn please? I read the [1] and
> > still don't very clear
> >
> > [1]
> > https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-g
> > eneral-guidelines
> >
> > -----邮件原件-----
> > 发件人: Yangze Guo [mailto:[hidden email]]
> > 发送时间: 2020年8月18日 星期二 10:50
> > 收件人: 范超 <[hidden email]>
> > 抄送: user ([hidden email]) <[hidden email]>
> > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > using Per-Job Mode
> >
> > Hi,
> >
> > I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead?
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:
> > >
> > > Thanks Yangze
> > >
> > > All 3 machines NodeManager is started.
> > >
> > > I just don't know why not three machines each running a Flink
> > > TaskManager and how to achieve this
> > >
> > > -----邮件原件-----
> > > 发件人: Yangze Guo [mailto:[hidden email]]
> > > 发送时间: 2020年8月18日 星期二 10:10
> > > 收件人: 范超 <[hidden email]>
> > > 抄送: user ([hidden email]) <[hidden email]>
> > > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > > using Per-Job Mode
> > >
> > > Hi,
> > >
> > > Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> > > >
> > > > Hi, Dev and Users
> > > > I’ve 3 machines each one is 8 cores and 16GB memory.
> > > > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > > > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > > > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > > > My question is how to set the cli parameter to start all of my
> > > > three machine (all task manager start on 3 machines)
> > > >
> > > > Thanks a lot
> > > > [cid:image001.png@01D67546.62291B70]
> > > >
> > > >
> > > > Chao fan
> > > >

[范超]

Thanks Yangze for providing these links I'll try it !

-----邮件原件-----
发件人: Yangze Guo [mailto:[hidden email]]
发送时间: 2020年8月18日 星期二 12:57
收件人: 范超 <[hidden email]>
抄送: user ([hidden email]) <[hidden email]>
主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode

The number of TM mainly depends on the parallelism and job graph.
Flink now allows you to set the maximum slots number (slotmanager-number-of-slots-max[1]). There is also a plan to support setting the minimum number of slots[2].

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#slotmanager-number-of-slots-max
[2] https://issues.apache.org/jira/browse/FLINK-15959

Best,
Yangze Guo

On Tue, Aug 18, 2020 at 12:21 PM 范超 <[hidden email]> wrote:

>
> Thanks Yangze
>
> 1. Do you meet any problem when deploying on Yarn or running Flink job?
> My job works well
>
> 2. Why do you need to start the TMs on all the three machines?
> From cluster perspective, I wonder if the process pressure can be balance to 3 machines.
>
> 3. Flink can control how many TM to start, but where to start the TMs depends on Yarn.
> Yes, the job where to start the TM is depend on Yarn.
> Could you please tell me parameter controls how many TM to start, the
> yn parameter is delete from 1.10 as the 1.9 doc sample list[1] below
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.ht
> ml
>
> Run example program using a per-job YARN cluster with 2 TaskManagers:
>
> ./bin/flink run -m yarn-cluster -yn 2 \
>                        ./examples/batch/WordCount.jar \
>                        --input hdfs:///user/hamlet.txt --output
> hdfs:///user/wordcount_out
>
> -----邮件原件-----
> 发件人: Yangze Guo [mailto:[hidden email]]
> 发送时间: 2020年8月18日 星期二 11:31
> 收件人: 范超 <[hidden email]>
> 抄送: user ([hidden email]) <[hidden email]>
> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> using Per-Job Mode
>
> Hi,
>
> Flink can control how many TM to start, but where to start the TMs depends on Yarn.
>
> Do you meet any problem when deploying on Yarn or running Flink job?
> Why do you need to start the TMs on all the three machines?
>
> Best,
> Yangze Guo
>
> On Tue, Aug 18, 2020 at 11:25 AM 范超 <[hidden email]> wrote:
> >
> > Thanks Yangze
> > The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me.
> > If Flink can not control how many tm to start , could anyone
> > providing me some best practice for deploying on yarn please? I read
> > the [1] and still don't very clear
> >
> > [1]
> > https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster
> > -g
> > eneral-guidelines
> >
> > -----邮件原件-----
> > 发件人: Yangze Guo [mailto:[hidden email]]
> > 发送时间: 2020年8月18日 星期二 10:50
> > 收件人: 范超 <[hidden email]>
> > 抄送: user ([hidden email]) <[hidden email]>
> > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > using Per-Job Mode
> >
> > Hi,
> >
> > I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead?
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Aug 18, 2020 at 10:24 AM 范超 <[hidden email]> wrote:
> > >
> > > Thanks Yangze
> > >
> > > All 3 machines NodeManager is started.
> > >
> > > I just don't know why not three machines each running a Flink
> > > TaskManager and how to achieve this
> > >
> > > -----邮件原件-----
> > > 发件人: Yangze Guo [mailto:[hidden email]]
> > > 发送时间: 2020年8月18日 星期二 10:10
> > > 收件人: 范超 <[hidden email]>
> > > 抄送: user ([hidden email]) <[hidden email]>
> > > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster
> > > using Per-Job Mode
> > >
> > > Hi,
> > >
> > > Do you start the NodeManager in all the three machines? If so, could you check all the NMs correctly connect to the ResourceManager?
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Aug 18, 2020 at 10:01 AM 范超 <[hidden email]> wrote:
> > > >
> > > > Hi, Dev and Users
> > > > I’ve 3 machines each one is 8 cores and 16GB memory.
> > > > Following it’s my Resource Manager screenshot the cluster have 36GB total.
> > > > I specify the paralism to 3 or even up to 12,  But the task manager is always running on two nodes not all three machine, the third node does not start the task manager.
> > > > I tried set the –p –tm –jm parameters, but it always the same, only different is more container on the two maching but not all three machine start the task manager.
> > > > My question is how to set the cli parameter to start all of my
> > > > three machine (all task manager start on 3 machines)
> > > >
> > > > Thanks a lot
> > > > [cid:image001.png@01D67546.62291B70]
> > > >
> > > >
> > > > Chao fan
> > > >