Regarding external metastore like HIVE

[Shivam Sharma]

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

[Chesnay Schepler]

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

[Ashwin Sinha]

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Chuanlei Ni]

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

[Shivam Sharma]

Hi,

Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog.

Currently, Flink provides an InMemoryExternalCatalog for demo and testing purposes. However, the ExternalCatalog interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks

On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <[hidden email]> wrote:

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

[Fabian Hueske-2]

Hi,

The docs explain that the ExternalCatalog interface can be used to implement a catalog for HCatalog or Metastore.

However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself.

I think there would be quite a few people interested in such a connector.

Best, Fabian

2018-07-03 10:24 GMT+02:00 Shivam Sharma <[hidden email]>:

Hi,

Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog.

Currently, Flink provides an InMemoryExternalCatalog for demo and testing purposes. However, the ExternalCatalog interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks

On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <[hidden email]> wrote:

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

[Timo Walther]

Hi,

you can follow the progress here: https://issues.apache.org/jira/browse/FLINK-9171

Regards,
Timo


Am 03.07.18 um 10:32 schrieb Fabian Hueske:

Hi,

The docs explain that the ExternalCatalog interface can be used to implement a catalog for HCatalog or Metastore.

However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself.

I think there would be quite a few people interested in such a connector.

Best, Fabian

2018-07-03 10:24 GMT+02:00 Shivam Sharma <[hidden email]>:

Hi,

Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog.

Currently, Flink provides an InMemoryExternalCatalog for demo and testing purposes. However, the ExternalCatalog interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks

On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <[hidden email]> wrote:

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

[Shivam Sharma]

Thanks Timo, Fabian

I will follow this.

Best

On Tue, Jul 3, 2018 at 3:01 PM Timo Walther <[hidden email]> wrote:

Hi,

you can follow the progress here: https://issues.apache.org/jira/browse/FLINK-9171

Regards,
Timo


Am 03.07.18 um 10:32 schrieb Fabian Hueske:

Hi,

The docs explain that the ExternalCatalog interface can be used to implement a catalog for HCatalog or Metastore.

However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself.

I think there would be quite a few people interested in such a connector.

Best, Fabian

2018-07-03 10:24 GMT+02:00 Shivam Sharma <[hidden email]>:

Hi,

Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog.

Currently, Flink provides an InMemoryExternalCatalog for demo and testing purposes. However, the ExternalCatalog interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks

On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <[hidden email]> wrote:

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

--

[Rong Rong]

+1 on this feature, there have been a lot of pains for us trying to connect to external catalog / metastore as well.

@shivam can you comment on the tickets regarding the specific use case and the type of external catalogs you are interested ?

Thanks,

Rong

On Tue, Jul 3, 2018 at 3:16 AM Shivam Sharma <[hidden email]> wrote:

Thanks Timo, Fabian

I will follow this.

Best

On Tue, Jul 3, 2018 at 3:01 PM Timo Walther <[hidden email]> wrote:

Hi,

you can follow the progress here: https://issues.apache.org/jira/browse/FLINK-9171

Regards,
Timo


Am 03.07.18 um 10:32 schrieb Fabian Hueske:

Hi,

The docs explain that the ExternalCatalog interface can be used to implement a catalog for HCatalog or Metastore.

However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself.

I think there would be quite a few people interested in such a connector.

Best, Fabian

2018-07-03 10:24 GMT+02:00 Shivam Sharma <[hidden email]>:

Hi,

Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog.

Currently, Flink provides an InMemoryExternalCatalog for demo and testing purposes. However, the ExternalCatalog interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.

Thanks

On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <[hidden email]> wrote:

@ashwin

I cannot find the documention about `metastore`

Could you give the reference?

Ashwin Sinha <[hidden email]> 于2018年7月3日周二 下午4:08写道：

By metastore we mean catalog where table information is stored.

On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <[hidden email]> wrote:

What do you mean with "Metastore"? Are you referring to state backends?

On 02.07.2018 18:41, Shivam Sharma wrote:

Hi,

I have read Flink documentation that Flink supports Metastore which is currently InMemory. Is Flink community thinking to implement external Metastore like Hive? 

Thanks

--

::DISCLAIMER::

----------------------------------------------------------------------------------------------------------------------------------------------------

This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.

--

--