Java library for Flink-Kudu integration

[ruben.casado.tejedor]

Hi all,

 

I apologize for sending the email to both accounts, but not sure where this topic fits better.

 

In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library allows reading/writing from/to Kudu tablets using DataSet API and also writing to Kudu using DataStream API.

 

You can find the code and documentation (including some examples) in https://github.com/rubencasado/Flink-Kudu

 

Any comment/suggestion/contribution is very welcomed J

 

We will try to publish this contribution to the Apache Bahir project.

 

Best

 

----------------------------------------

Rubén Casado Tejedor, PhD

> accenture digital

Big Data Manager

' + 34 629 009 429

* [hidden email]

---

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[Fabian Hueske-2]

Hi Ruben,

thanks for sharing this!

A Flink Kudu connector is a great contribution and Bahir seems to be the right place for it.

Thanks, Fabian

2017-03-27 15:35 GMT+02:00 <[hidden email]>:

Hi all,

I apologize for sending the email to both accounts, but not sure where this topic fits better.

In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library allows reading/writing from/to Kudu tablets using DataSet API and also writing to Kudu using DataStream API.

You can find the code and documentation (including some examples) in https://github.com/rubencasado/Flink-Kudu

Any comment/suggestion/contribution is very welcomed ☺

We will try to publish this contribution to the Apache Bahir project.

Best

----------------------------------------
Rubén Casado Tejedor, PhD
> accenture digital
Big Data Manager
' <a href="tel:%2B%2034%20629%20009%20429" value="[phone]">+ 34 629 009 429
• [hidden email]<mailto:[hidden email]>

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[ruben.casado.tejedor]

Hi all,

 

I apologize for sending the email to both user and dev accounts, but not sure where this topic fits better.

 

In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library allows reading/writing from/to Kudu tablets using DataSet API and also writing to Kudu using DataStream API.

 

You can find the code and documentation (including some examples) in https://github.com/rubencasado/Flink-Kudu

 

Any comment/suggestion/contribution is very welcomed J

 

We will try to publish this contribution to the Apache Bahir project.

 

Best

 

----------------------------------------

Rubén Casado Tejedor, PhD

> accenture digital

Big Data Manager

' + 34 629 009 429

* [hidden email]

---

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com

[Fabian Hueske-2]

No, we do not want to move all connectors to Apache Bahir or replace the connectors by Bahir.

The Flink community aims to maintain the most important connectors within Flink.
Maintaining all connectors would be a huge effort. So, we decided to move some of the less frequently used connectors to Bahir.

Best, Fabian

2017-03-28 8:31 GMT+02:00 shijinkui <[hidden email]>:

Hi, Fabian
Do we have plan to replace Flink connectors with bahir-flink[1]?

[1] https://github.com/apache/Bahir-flink

在 2017/3/28 上午12:15， "Fabian Hueske" <[hidden email]> 写入:

>Hi Ruben,
>
>thanks for sharing this!
>A Flink Kudu connector is a great contribution and Bahir seems to be the
>right place for it.
>
>Thanks, Fabian
>
>
>2017-03-27 15:35 GMT+02:00 <[hidden email]>:
>
>> Hi all,
>>
>> I apologize for sending the email to both accounts, but not sure where
>> this topic fits better.
>>
>> In my team, we have been working in some PoCs and PoVs about new data
>> architectures. As part of this work, we have implemented a library to
>> connect Kudu and Flink. The library allows reading/writing from/to Kudu
>> tablets using DataSet API and also writing to Kudu using DataStream API.
>>
>> You can find the code and documentation (including some examples) in
>> https://github.com/rubencasado/Flink-Kudu
>>
>> Any comment/suggestion/contribution is very welcomed ☺
>>
>> We will try to publish this contribution to the Apache Bahir project.
>>
>> Best
>>
>> ----------------------------------------
>> Rubén Casado Tejedor, PhD
>> > accenture digital
>> Big Data Manager
>> ' <a href="tel:%2B%2034%20629%20009%20429" value="[phone]">+ 34 629 009 429
>> • [hidden email]<mailto:[hidden email].
>> [hidden email]>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you
>>have
>> received it in error, please notify the sender immediately and delete
>>the
>> original. Any other use of the e-mail by you is prohibited. Where
>>allowed
>> by local law, electronic communications with Accenture and its
>>affiliates,
>> including e-mail and instant messaging (including content), may be
>>scanned
>> by our systems for the purposes of information security and assessment
>>of
>> internal compliance with Accenture policy.
>> ____________________________________________________________
>> __________________________
>>
>> www.accenture.com
>>

[Stephan Ewen]

We are currently looking into how we can keep the size of the code base under control (because it is growing super large).

Part is moving libraries into a dedicated subrepository (there is a separate mailing list thread on that) and some connectors to Bahir.

Connectors can move between Flink and Bahir, for example it makes sense to move heavily worked on connectors to the Flink code base.

On Tue, Mar 28, 2017 at 10:01 AM, Fabian Hueske <[hidden email]> wrote:

No, we do not want to move all connectors to Apache Bahir or replace the connectors by Bahir.

The Flink community aims to maintain the most important connectors within Flink.
Maintaining all connectors would be a huge effort. So, we decided to move some of the less frequently used connectors to Bahir.

Best, Fabian

2017-03-28 8:31 GMT+02:00 shijinkui <[hidden email]>:

Hi, Fabian
Do we have plan to replace Flink connectors with bahir-flink[1]?

[1] https://github.com/apache/Bahir-flink

在 2017/3/28 上午12:15， "Fabian Hueske" <[hidden email]> 写入:

>Hi Ruben,
>
>thanks for sharing this!
>A Flink Kudu connector is a great contribution and Bahir seems to be the
>right place for it.
>
>Thanks, Fabian
>
>
>2017-03-27 15:35 GMT+02:00 <[hidden email]>:
>
>> Hi all,
>>
>> I apologize for sending the email to both accounts, but not sure where
>> this topic fits better.
>>
>> In my team, we have been working in some PoCs and PoVs about new data
>> architectures. As part of this work, we have implemented a library to
>> connect Kudu and Flink. The library allows reading/writing from/to Kudu
>> tablets using DataSet API and also writing to Kudu using DataStream API.
>>
>> You can find the code and documentation (including some examples) in
>> https://github.com/rubencasado/Flink-Kudu
>>
>> Any comment/suggestion/contribution is very welcomed ☺
>>
>> We will try to publish this contribution to the Apache Bahir project.
>>
>> Best
>>
>> ----------------------------------------
>> Rubén Casado Tejedor, PhD
>> > accenture digital
>> Big Data Manager
>> ' <a href="tel:%2B%2034%20629%20009%20429" value="[phone]" target="_blank">+ 34 629 009 429
>> • [hidden email]<mailto:[hidden email].
>> [hidden email]>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you
>>have
>> received it in error, please notify the sender immediately and delete
>>the
>> original. Any other use of the e-mail by you is prohibited. Where
>>allowed
>> by local law, electronic communications with Accenture and its
>>affiliates,
>> including e-mail and instant messaging (including content), may be
>>scanned
>> by our systems for the purposes of information security and assessment
>>of
>> internal compliance with Accenture policy.
>> ____________________________________________________________
>> __________________________
>>
>> www.accenture.com
>>