Google Compute Engine Cluster

[Giacomo Licari]

Hi guys,

I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied (publickey).

Someone had the same problem?

Thanks for your help.

[Maximilian Michels]

Hi Giacomo,

Have you tried setting up Flink on GCE using bdutil? It is very easy: http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html

If you don't want to use bdutil:

I'm assuming you want to start Flink using the "start_cluster.sh" script which uses ssh to start the task managers on the workers. By default, instances on GCE are not authenticated via ssh with each other. Using the gcloud tool, you can configure an ssh key on all hosts (gcloud compute config-ssh) which lets you ssh into all instances. To enable public key ssh authentication between your GCE nodes, I'd advise to create a new key on your machine for just this purpose and then distribute the key (public and private) to the master. Then add the public key to all worker nodes's "authorized_keys" file from your machine.

Best,

Max

On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <[hidden email]> wrote:

Hi guys,

I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied (publickey).

Someone had the same problem?

Thanks for your help.

[Giacomo Licari]

Thanks a lot Max,

now I can set up ssh keys.

I started my cluster correctly with ./start-cluster.sh, do you use an example program to test if everything works fine?

Thanks again,

Giacomo

On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <[hidden email]> wrote:

Hi Giacomo,

Have you tried setting up Flink on GCE using bdutil? It is very easy: http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html

If you don't want to use bdutil:

I'm assuming you want to start Flink using the "start_cluster.sh" script which uses ssh to start the task managers on the workers. By default, instances on GCE are not authenticated via ssh with each other. Using the gcloud tool, you can configure an ssh key on all hosts (gcloud compute config-ssh) which lets you ssh into all instances. To enable public key ssh authentication between your GCE nodes, I'd advise to create a new key on your machine for just this purpose and then distribute the key (public and private) to the master. Then add the public key to all worker nodes's "authorized_keys" file from your machine.

Best,

Max

On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <[hidden email]> wrote:

Hi guys,

I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied (publickey).

Someone had the same problem?

Thanks for your help.

[Maximilian Michels]

Sure, just ran an included example.

./bin/flink run examples/*WordCount.jar

By default, it will use a really small example DataSet but it should be enough to verify the cluster setup.

You're welcome.

Best,

Max

On Wed, Apr 15, 2015 at 12:24 PM, Giacomo Licari <[hidden email]> wrote:

Thanks a lot Max,

now I can set up ssh keys.

I started my cluster correctly with ./start-cluster.sh, do you use an example program to test if everything works fine?

Thanks again,

Giacomo

On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <[hidden email]> wrote:

Hi Giacomo,

Have you tried setting up Flink on GCE using bdutil? It is very easy: http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html

If you don't want to use bdutil:

I'm assuming you want to start Flink using the "start_cluster.sh" script which uses ssh to start the task managers on the workers. By default, instances on GCE are not authenticated via ssh with each other. Using the gcloud tool, you can configure an ssh key on all hosts (gcloud compute config-ssh) which lets you ssh into all instances. To enable public key ssh authentication between your GCE nodes, I'd advise to create a new key on your machine for just this purpose and then distribute the key (public and private) to the master. Then add the public key to all worker nodes's "authorized_keys" file from your machine.

Best,

Max

On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <[hidden email]> wrote:

Hi guys,

I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied (publickey).

Someone had the same problem?

Thanks for your help.

[Giacomo Licari]

Thank you Max,

everything works well.

On Wed, Apr 15, 2015 at 1:04 PM, Maximilian Michels <[hidden email]> wrote:

Sure, just ran an included example.

./bin/flink run examples/*WordCount.jar

By default, it will use a really small example DataSet but it should be enough to verify the cluster setup.

You're welcome.

Best,

Max

On Wed, Apr 15, 2015 at 12:24 PM, Giacomo Licari <[hidden email]> wrote:

Thanks a lot Max,

now I can set up ssh keys.

I started my cluster correctly with ./start-cluster.sh, do you use an example program to test if everything works fine?

Thanks again,

Giacomo

On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <[hidden email]> wrote:

Hi Giacomo,

Have you tried setting up Flink on GCE using bdutil? It is very easy: http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html

If you don't want to use bdutil:

I'm assuming you want to start Flink using the "start_cluster.sh" script which uses ssh to start the task managers on the workers. By default, instances on GCE are not authenticated via ssh with each other. Using the gcloud tool, you can configure an ssh key on all hosts (gcloud compute config-ssh) which lets you ssh into all instances. To enable public key ssh authentication between your GCE nodes, I'd advise to create a new key on your machine for just this purpose and then distribute the key (public and private) to the master. Then add the public key to all worker nodes's "authorized_keys" file from your machine.

Best,

Max

On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <[hidden email]> wrote:

Hi guys,

I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied (publickey).

Someone had the same problem?

Thanks for your help.