(DEPRECATED) Apache Flink User Mailing List archive.

Apache Beam and Flink

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
Ashutosh Kumar-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Apache Beam and Flink

Ashutosh Kumar-2
How does apache beam fits with  flink ? Is it an alternative for flink or complementary to it ?

Thanks
Ashutosh
Slim Baltagi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Apache Beam and Flink

Slim Baltagi
Hi Ashutosh

Apache Beam provides a Unified API for batch and streaming.
It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service). 
It is not an alternative to Flink because it is an API and you still need an execution engine.
It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API. 
It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.  
Many of Flink committers are also Apache Beam committers!
The following blogs describe why Apache Beam:

A few recent resources about Apache Beam published this month: May 2016 

I hope this helps.

Thanks

Slim Baltagi

On May 26, 2016, at 2:20 AM, Ashutosh Kumar <[hidden email]> wrote:

How does apache beam fits with  flink ? Is it an alternative for flink or complementary to it ?

Thanks
Ashutosh

Ashutosh Kumar-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Apache Beam and Flink

Ashutosh Kumar-2
Thanks . So if we use Beam API with flink engine then we can get inter action between batch and stream ? As i know currently in flink Dataset and DStream can not talk . Is this correct ?
 Thanks
Ashutosh
 

On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <[hidden email]> wrote:
Hi Ashutosh

Apache Beam provides a Unified API for batch and streaming.
It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service). 
It is not an alternative to Flink because it is an API and you still need an execution engine.
It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API. 
It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.  
Many of Flink committers are also Apache Beam committers!
The following blogs describe why Apache Beam:

A few recent resources about Apache Beam published this month: May 2016 

I hope this helps.

Thanks

Slim Baltagi

On May 26, 2016, at 2:20 AM, Ashutosh Kumar <[hidden email]> wrote:

How does apache beam fits with  flink ? Is it an alternative for flink or complementary to it ?

Thanks
Ashutosh


Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Apache Beam and Flink

Fabian Hueske-2
No, that is not supported yet.
Beam provides a common API but the Flink runner translates programs against batch sources into the DataSet API programs and Beam programs against streaming source into DataStream programs.
It is not possible to mix both.

2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <[hidden email]>:
Thanks . So if we use Beam API with flink engine then we can get inter action between batch and stream ? As i know currently in flink Dataset and DStream can not talk . Is this correct ?
 Thanks
Ashutosh
 

On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <[hidden email]> wrote:
Hi Ashutosh

Apache Beam provides a Unified API for batch and streaming.
It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service). 
It is not an alternative to Flink because it is an API and you still need an execution engine.
It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API. 
It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.  
Many of Flink committers are also Apache Beam committers!
The following blogs describe why Apache Beam:

A few recent resources about Apache Beam published this month: May 2016 

I hope this helps.

Thanks

Slim Baltagi

On May 26, 2016, at 2:20 AM, Ashutosh Kumar <[hidden email]> wrote:

How does apache beam fits with  flink ? Is it an alternative for flink or complementary to it ?

Thanks
Ashutosh



Slim Baltagi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Apache Beam and Flink

Slim Baltagi
Hi Ashutosh

There is a related open JIRA: Enable DataSet and DataStream Joins https://issues.apache.org/jira/browse/FLINK-2320

Slim 


On May 26, 2016, at 3:05 AM, Fabian Hueske <[hidden email]> wrote:

No, that is not supported yet.
Beam provides a common API but the Flink runner translates programs against batch sources into the DataSet API programs and Beam programs against streaming source into DataStream programs.
It is not possible to mix both.

2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <[hidden email]>:
Thanks . So if we use Beam API with flink engine then we can get inter action between batch and stream ? As i know currently in flink Dataset and DStream can not talk . Is this correct ?
 Thanks
Ashutosh
 

On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <[hidden email]> wrote:
Hi Ashutosh

Apache Beam provides a Unified API for batch and streaming.
It also supports multiple ‘runners’: local, Apache Spark, Apache Flink and Google Cloud Data Flow (commercial service). 
It is not an alternative to Flink because it is an API and you still need an execution engine.
It can be used as an alternative API to using the two Flink APIs : DataSet API and DataStream API. 
It can be complementary to Flink in the way that you use Beam as API and Flink as the execution engine.  
Many of Flink committers are also Apache Beam committers!
The following blogs describe why Apache Beam:

A few recent resources about Apache Beam published this month: May 2016 

I hope this helps.

Thanks

Slim Baltagi

On May 26, 2016, at 2:20 AM, Ashutosh Kumar <[hidden email]> wrote:

How does apache beam fits with  flink ? Is it an alternative for flink or complementary to it ?

Thanks
Ashutosh




Maximilian Michels
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Apache Beam and Flink

Maximilian Michels
Small addition: The Flink Runner translates into the DataSet or
DataStream API depending on the "streaming" flag of the
PipelineOptions. The default mode is batch. Ultimately, this flag we
be removed and replaced with an automated decision depending on the
sources used.

On Thu, May 26, 2016 at 10:20 AM, Slim Baltagi <[hidden email]> wrote:

> Hi Ashutosh
>
> There is a related open JIRA: Enable DataSet and DataStream Joins
> https://issues.apache.org/jira/browse/FLINK-2320
>
> Slim
>
>
> On May 26, 2016, at 3:05 AM, Fabian Hueske <[hidden email]> wrote:
>
> No, that is not supported yet.
> Beam provides a common API but the Flink runner translates programs against
> batch sources into the DataSet API programs and Beam programs against
> streaming source into DataStream programs.
> It is not possible to mix both.
>
> 2016-05-26 10:00 GMT+02:00 Ashutosh Kumar <[hidden email]>:
>>
>> Thanks . So if we use Beam API with flink engine then we can get inter
>> action between batch and stream ? As i know currently in flink Dataset and
>> DStream can not talk . Is this correct ?
>>  Thanks
>> Ashutosh
>>
>>
>> On Thu, May 26, 2016 at 1:09 PM, Slim Baltagi <[hidden email]> wrote:
>>>
>>> Hi Ashutosh
>>>
>>> Apache Beam provides a Unified API for batch and streaming.
>>> It also supports multiple ‘runners’: local, Apache Spark, Apache Flink
>>> and Google Cloud Data Flow (commercial service).
>>> It is not an alternative to Flink because it is an API and you still need
>>> an execution engine.
>>> It can be used as an alternative API to using the two Flink APIs :
>>> DataSet API and DataStream API.
>>> It can be complementary to Flink in the way that you use Beam as API and
>>> Flink as the execution engine.
>>> Many of Flink committers are also Apache Beam committers!
>>> The following blogs describe why Apache Beam:
>>>
>>>  from Flink perspective: http://data-artisans.com/why-apache-beam/
>>>  from Google perspective.
>>> https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective
>>>
>>>
>>> A few recent resources about Apache Beam published this month: May 2016
>>>
>>> Running Apache Beam (screencast)
>>> https://www.youtube.com/watch?v=dwxUbzbwtyI
>>> Introduction to Apache Beam ( presentation)
>>> https://skillsmatter.com/skillscasts/8036-apache-flink-may-meetup
>>> Introduction to Apache Beam ( blog)
>>> http://www.talend.com/blog/2016/05/02/introduction-to-apache-beam
>>>
>>>
>>> I hope this helps.
>>>
>>> Thanks
>>>
>>> Slim Baltagi
>>>
>>> On May 26, 2016, at 2:20 AM, Ashutosh Kumar <[hidden email]>
>>> wrote:
>>>
>>> How does apache beam fits with  flink ? Is it an alternative for flink or
>>> complementary to it ?
>>>
>>> Thanks
>>> Ashutosh
>>>
>>>
>>
>
>
Free forum by Nabble Edit this page