Flink cluster dev environment in Docker

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink cluster dev environment in Docker

Emmanuel
FYI 

Posted my dev cluster deployment in Docker here:


Still need to work on aggregating the logs but I hope it can get people started easy.

Cheers
Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

Flavio Pompermaier
Great addition!

On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
FYI 

Posted my dev cluster deployment in Docker here:


Still need to work on aggregating the logs but I hope it can get people started easy.

Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

rmetzger0
Hey Emmanuel,

thank you for this great contribution. I'm going to test the docker deployment soon.

I would actually like to include the files into the Flink source repository.
Either into the "flink-contrib" module, or into the "tools" directory.
Whats the take of the other committers on this? 

On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier <[hidden email]> wrote:
Great addition!

On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
FYI 

Posted my dev cluster deployment in Docker here:


Still need to work on aggregating the logs but I hope it can get people started easy.

Cheers


Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

Henry Saputra
+1 for the idea.

I cross post this to dev@ list for FYI

- Henry

On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:

> Hey Emmanuel,
>
> thank you for this great contribution. I'm going to test the docker
> deployment soon.
>
> I would actually like to include the files into the Flink source repository.
> Either into the "flink-contrib" module, or into the "tools" directory.
> Whats the take of the other committers on this?
>
> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier <[hidden email]>
> wrote:
>>
>> Great addition!
>>
>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
>>>
>>> FYI
>>>
>>> Posted my dev cluster deployment in Docker here:
>>>
>>> https://github.com/streamnsight/docker-flink
>>>
>>> Still need to work on aggregating the logs but I hope it can get people
>>> started easy.
>>>
>>> Cheers
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

rmetzger0
I just tried the image/scripts: Amazing work! Very well documented, also for docker beginners like me.

@Emmanuel: Would you like to open a pull request to add the files to the "flink-contib" directory?
If you're too busy right now, I can also do it for you (only if you agree of course ;)) If you would like to keep the files in your repository, we can also link to it.

On Fri, Mar 20, 2015 at 6:05 AM, Henry Saputra <[hidden email]> wrote:
+1 for the idea.

I cross post this to dev@ list for FYI

- Henry

On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:
> Hey Emmanuel,
>
> thank you for this great contribution. I'm going to test the docker
> deployment soon.
>
> I would actually like to include the files into the Flink source repository.
> Either into the "flink-contrib" module, or into the "tools" directory.
> Whats the take of the other committers on this?
>
> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier <[hidden email]>
> wrote:
>>
>> Great addition!
>>
>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
>>>
>>> FYI
>>>
>>> Posted my dev cluster deployment in Docker here:
>>>
>>> https://github.com/streamnsight/docker-flink
>>>
>>> Still need to work on aggregating the logs but I hope it can get people
>>> started easy.
>>>
>>> Cheers
>
>

Reply | Threaded
Open this post in threaded view
|

RE: Flink cluster dev environment in Docker

Emmanuel
I'm new to Docker too, but I find it to be a great tool to setup a dev environment that's reproducible.

Not sure how to send a pull request to the Flink repo, so if you want to do that, go ahead, or get me the link/how to do it.

BTW, I would add info on how to use the web API, but I haven't see a doc for it.  the Web client is cool but not super practicle to debug I find, because I don't want to spend my time uploading jar on a web client, I'd rather send a one liner with Curl... Which is why I've mostly been using scp/ssh for this. 
I can hack the Webclient but if there was a doc for the endpoints and format, it would be easier.

Cheers


From: [hidden email]
Date: Tue, 24 Mar 2015 09:00:23 +0100
Subject: Re: Flink cluster dev environment in Docker
To: [hidden email]
CC: [hidden email]

I just tried the image/scripts: Amazing work! Very well documented, also for docker beginners like me.

@Emmanuel: Would you like to open a pull request to add the files to the "flink-contib" directory?
If you're too busy right now, I can also do it for you (only if you agree of course ;)) If you would like to keep the files in your repository, we can also link to it.

On Fri, Mar 20, 2015 at 6:05 AM, Henry Saputra <[hidden email]> wrote:
+1 for the idea.

I cross post this to dev@ list for FYI

- Henry

On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:
> Hey Emmanuel,
>
> thank you for this great contribution. I'm going to test the docker
> deployment soon.
>
> I would actually like to include the files into the Flink source repository.
> Either into the "flink-contrib" module, or into the "tools" directory.
> Whats the take of the other committers on this?
>
> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier <[hidden email]>
> wrote:
>>
>> Great addition!
>>
>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
>>>
>>> FYI
>>>
>>> Posted my dev cluster deployment in Docker here:
>>>
>>> https://github.com/streamnsight/docker-flink
>>>
>>> Still need to work on aggregating the logs but I hope it can get people
>>> started easy.
>>>
>>> Cheers
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

Henry Saputra
HI Emmanuel,

We are using Github mirror [1] to accept patches and Pull Requests (PR).

I have ticket item to update the how to contribute page about sending
pull requests for Flink but it is the same as sending PR to other
Github repos [2].

Like Robert said before, we could do it for you but if you would try
it it would be great =)

Thanks,

Henry


[1] https://github.com/apache/flink
[2] https://help.github.com/articles/using-pull-requests

On Tue, Mar 24, 2015 at 5:33 PM, Emmanuel <[hidden email]> wrote:

> I'm new to Docker too, but I find it to be a great tool to setup a dev
> environment that's reproducible.
>
> Not sure how to send a pull request to the Flink repo, so if you want to do
> that, go ahead, or get me the link/how to do it.
>
> BTW, I would add info on how to use the web API, but I haven't see a doc for
> it.  the Web client is cool but not super practicle to debug I find, because
> I don't want to spend my time uploading jar on a web client, I'd rather send
> a one liner with Curl... Which is why I've mostly been using scp/ssh for
> this.
> I can hack the Webclient but if there was a doc for the endpoints and
> format, it would be easier.
>
> Cheers
>
> ________________________________
> From: [hidden email]
> Date: Tue, 24 Mar 2015 09:00:23 +0100
> Subject: Re: Flink cluster dev environment in Docker
> To: [hidden email]
> CC: [hidden email]
>
>
> I just tried the image/scripts: Amazing work! Very well documented, also for
> docker beginners like me.
>
> @Emmanuel: Would you like to open a pull request to add the files to the
> "flink-contib" directory?
> If you're too busy right now, I can also do it for you (only if you agree of
> course ;)) If you would like to keep the files in your repository, we can
> also link to it.
>
> On Fri, Mar 20, 2015 at 6:05 AM, Henry Saputra <[hidden email]>
> wrote:
>
> +1 for the idea.
>
> I cross post this to dev@ list for FYI
>
> - Henry
>
> On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:
>> Hey Emmanuel,
>>
>> thank you for this great contribution. I'm going to test the docker
>> deployment soon.
>>
>> I would actually like to include the files into the Flink source
>> repository.
>> Either into the "flink-contrib" module, or into the "tools" directory.
>> Whats the take of the other committers on this?
>>
>> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier
>> <[hidden email]>
>> wrote:
>>>
>>> Great addition!
>>>
>>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
>>>>
>>>> FYI
>>>>
>>>> Posted my dev cluster deployment in Docker here:
>>>>
>>>> https://github.com/streamnsight/docker-flink
>>>>
>>>> Still need to work on aggregating the logs but I hope it can get people
>>>> started easy.
>>>>
>>>> Cheers
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Flink cluster dev environment in Docker

Emmanuel
done...


> Date: Tue, 24 Mar 2015 18:01:56 -0700

> Subject: Re: Flink cluster dev environment in Docker
> From: [hidden email]
> To: [hidden email]
>
> HI Emmanuel,
>
> We are using Github mirror [1] to accept patches and Pull Requests (PR).
>
> I have ticket item to update the how to contribute page about sending
> pull requests for Flink but it is the same as sending PR to other
> Github repos [2].
>
> Like Robert said before, we could do it for you but if you would try
> it it would be great =)
>
> Thanks,
>
> Henry
>
>
> [1] https://github.com/apache/flink
> [2] https://help.github.com/articles/using-pull-requests
>
> On Tue, Mar 24, 2015 at 5:33 PM, Emmanuel <[hidden email]> wrote:
> > I'm new to Docker too, but I find it to be a great tool to setup a dev
> > environment that's reproducible.
> >
> > Not sure how to send a pull request to the Flink repo, so if you want to do
> > that, go ahead, or get me the link/how to do it.
> >
> > BTW, I would add info on how to use the web API, but I haven't see a doc for
> > it. the Web client is cool but not super practicle to debug I find, because
> > I don't want to spend my time uploading jar on a web client, I'd rather send
> > a one liner with Curl... Which is why I've mostly been using scp/ssh for
> > this.
> > I can hack the Webclient but if there was a doc for the endpoints and
> > format, it would be easier.
> >
> > Cheers
> >
> > ________________________________
> > From: [hidden email]
> > Date: Tue, 24 Mar 2015 09:00:23 +0100
> > Subject: Re: Flink cluster dev environment in Docker
> > To: [hidden email]
> > CC: [hidden email]
> >
> >
> > I just tried the image/scripts: Amazing work! Very well documented, also for
> > docker beginners like me.
> >
> > @Emmanuel: Would you like to open a pull request to add the files to the
> > "flink-contib" directory?
> > If you're too busy right now, I can also do it for you (only if you agree of
> > course ;)) If you would like to keep the files in your repository, we can
> > also link to it.
> >
> > On Fri, Mar 20, 2015 at 6:05 AM, Henry Saputra <[hidden email]>
> > wrote:
> >
> > +1 for the idea.
> >
> > I cross post this to dev@ list for FYI
> >
> > - Henry
> >
> > On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:
> >> Hey Emmanuel,
> >>
> >> thank you for this great contribution. I'm going to test the docker
> >> deployment soon.
> >>
> >> I would actually like to include the files into the Flink source
> >> repository.
> >> Either into the "flink-contrib" module, or into the "tools" directory.
> >> Whats the take of the other committers on this?
> >>
> >> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier
> >> <[hidden email]>
> >> wrote:
> >>>
> >>> Great addition!
> >>>
> >>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
> >>>>
> >>>> FYI
> >>>>
> >>>> Posted my dev cluster deployment in Docker here:
> >>>>
> >>>> https://github.com/streamnsight/docker-flink
> >>>>
> >>>> Still need to work on aggregating the logs but I hope it can get people
> >>>> started easy.
> >>>>
> >>>> Cheers
> >>
> >>
> >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Flink cluster dev environment in Docker

rmetzger0
Great. Thank you for opening a pull request. I hope to find time to merge it later today.


On Wed, Mar 25, 2015 at 5:08 AM, Emmanuel <[hidden email]> wrote:
done...


> Date: Tue, 24 Mar 2015 18:01:56 -0700
> Subject: Re: Flink cluster dev environment in Docker
> From: [hidden email]

> To: [hidden email]

>
> HI Emmanuel,
>
> We are using Github mirror [1] to accept patches and Pull Requests (PR).
>
> I have ticket item to update the how to contribute page about sending
> pull requests for Flink but it is the same as sending PR to other
> Github repos [2].
>
> Like Robert said before, we could do it for you but if you would try
> it it would be great =)
>
> Thanks,
>
> Henry
>
>
> [1] https://github.com/apache/flink
> [2] https://help.github.com/articles/using-pull-requests
>
> On Tue, Mar 24, 2015 at 5:33 PM, Emmanuel <[hidden email]> wrote:
> > I'm new to Docker too, but I find it to be a great tool to setup a dev
> > environment that's reproducible.
> >
> > Not sure how to send a pull request to the Flink repo, so if you want to do
> > that, go ahead, or get me the link/how to do it.
> >
> > BTW, I would add info on how to use the web API, but I haven't see a doc for
> > it. the Web client is cool but not super practicle to debug I find, because
> > I don't want to spend my time uploading jar on a web client, I'd rather send
> > a one liner with Curl... Which is why I've mostly been using scp/ssh for
> > this.
> > I can hack the Webclient but if there was a doc for the endpoints and
> > format, it would be easier.
> >
> > Cheers
> >
> > ________________________________
> > From: [hidden email]
> > Date: Tue, 24 Mar 2015 09:00:23 +0100
> > Subject: Re: Flink cluster dev environment in Docker
> > To: [hidden email]
> > CC: [hidden email]
> >
> >
> > I just tried the image/scripts: Amazing work! Very well documented, also for
> > docker beginners like me.
> >
> > @Emmanuel: Would you like to open a pull request to add the files to the
> > "flink-contib" directory?
> > If you're too busy right now, I can also do it for you (only if you agree of
> > course ;)) If you would like to keep the files in your repository, we can
> > also link to it.
> >
> > On Fri, Mar 20, 2015 at 6:05 AM, Henry Saputra <[hidden email]>
> > wrote:
> >
> > +1 for the idea.
> >
> > I cross post this to dev@ list for FYI
> >
> > - Henry
> >
> > On Tue, Mar 17, 2015 at 2:54 AM, Robert Metzger <[hidden email]> wrote:
> >> Hey Emmanuel,
> >>
> >> thank you for this great contribution. I'm going to test the docker
> >> deployment soon.
> >>
> >> I would actually like to include the files into the Flink source
> >> repository.
> >> Either into the "flink-contrib" module, or into the "tools" directory.
> >> Whats the take of the other committers on this?
> >>
> >> On Tue, Mar 17, 2015 at 10:01 AM, Flavio Pompermaier
> >> <[hidden email]>
> >> wrote:
> >>>
> >>> Great addition!
> >>>
> >>> On Tue, Mar 17, 2015 at 5:11 AM, Emmanuel <[hidden email]> wrote:
> >>>>
> >>>> FYI
> >>>>
> >>>> Posted my dev cluster deployment in Docker here:
> >>>>
> >>>> https://github.com/streamnsight/docker-flink
> >>>>
> >>>> Still need to work on aggregating the logs but I hope it can get people
> >>>> started easy.
> >>>>
> >>>> Cheers
> >>
> >>
> >
> >