Flink to BigTable

[Pierre Oberholzer]

Dear Community,

I would like to use BigTable as a sink for a Flink job:

1) Is there a connector out-of-the-box ?

2) Can I use Datastream API ?

3) How can I optimally pass a sparse object (99% sparsity), i.e. ensure no key/value are created in BigTable for nulls ?

I have searched the documentation for the above topics but couldn't answer those questions.

Many thanks for your support !

--

[Niels Basjes]

Hi,

I haven't tried it myself yet but there is a Flink connector for HBase and I remember someone telling me that Google has made a library available which is effectively the HBase client which talks to BigTable in the backend.

Like I said: I haven't tried this yet myself.

Niels Basjes

Op zo 24 jan. 2021 om 19:29 schreef Pierre Oberholzer <[hidden email]>:

Dear Community,

I would like to use BigTable as a sink for a Flink job:

1) Is there a connector out-of-the-box ?

2) Can I use Datastream API ?

3) How can I optimally pass a sparse object (99% sparsity), i.e. ensure no key/value are created in BigTable for nulls ?

I have searched the documentation for the above topics but couldn't answer those questions.

Many thanks for your support !

--

Pierre

--

[Pierre Oberholzer]

Dear Community,

More precise feedback on the points raised will be highly appreciated !

@Niels: thanks for this first statement 

Thanks a lot !

Le dim. 24 janv. 2021 à 19:37, Niels Basjes <[hidden email]> a écrit :

Hi,

I haven't tried it myself yet but there is a Flink connector for HBase and I remember someone telling me that Google has made a library available which is effectively the HBase client which talks to BigTable in the backend.

Like I said: I haven't tried this yet myself.

Niels Basjes

Op zo 24 jan. 2021 om 19:29 schreef Pierre Oberholzer <[hidden email]>:

Dear Community,

I would like to use BigTable as a sink for a Flink job:

1) Is there a connector out-of-the-box ?

2) Can I use Datastream API ?

3) How can I optimally pass a sparse object (99% sparsity), i.e. ensure no key/value are created in BigTable for nulls ?

I have searched the documentation for the above topics but couldn't answer those questions.

Many thanks for your support !

--

Pierre

--

Best regards / Met vriendelijke groeten,

Niels Basjes

--