Using RocksDB as lookup source in Flink

[srikanth flink]

Hi there,

Can someone help me implement Flink source Kafka to Flink Sink RocksDB, while I could use UDF for lookup RocksDB in SQL queries?

Context: I get a list of IPaddresses in a stream which I wish to store in RocksDB. Therefore the other stream perform a lookup to match the IPaddress.

Thanks

Srikanth

[OpenInx]

Hi

The Kafka table source & sink connector has been implemented (at least flink1.9 support this), but the RocksDB connector

not support yet, you may need to implement it by yourself.  Here[1] we have a brief wiki to show  what interfaces we need to implement,

but seems it's not detailed enough personally. I think the existed kafka connector code will be helpful for implementing your RocksDB

sink [2].

[1]. https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html#define-a-tablesink

[2]. https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java

On Mon, Nov 4, 2019 at 6:05 PM srikanth flink <[hidden email]> wrote:

Hi there,

Can someone help me implement Flink source Kafka to Flink Sink RocksDB, while I could use UDF for lookup RocksDB in SQL queries?

Context: I get a list of IPaddresses in a stream which I wish to store in RocksDB. Therefore the other stream perform a lookup to match the IPaddress.

Thanks

Srikanth

[Yun Tang]

Hi Srikanth

 

As RocksDB is a single node DB which just like InfluxDB, I recommend you could refer to an implementation of InfluxDB sink. [1]

 

[1] https://github.com/apache/bahir-flink/tree/master/flink-connector-influxdb

 

Best

Yun Tang

 

 

From: OpenInx <[hidden email]>
Date: Monday, November 4, 2019 at 6:28 PM
To: srikanth flink <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: Using RocksDB as lookup source in Flink

 

Hi

 

The Kafka table source & sink connector has been implemented (at least flink1.9 support this), but the RocksDB connector

not support yet, you may need to implement it by yourself.  Here[1] we have a brief wiki to show  what interfaces we need to implement,

but seems it's not detailed enough personally. I think the existed kafka connector code will be helpful for implementing your RocksDB

sink [2].

 

[1]. https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html#define-a-tablesink

[2]. https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaTableSink.java

 

On Mon, Nov 4, 2019 at 6:05 PM srikanth flink <[hidden email]> wrote:

Hi there,

 

Can someone help me implement Flink source Kafka to Flink Sink RocksDB, while I could use UDF for lookup RocksDB in SQL queries?

 

Context: I get a list of IPaddresses in a stream which I wish to store in RocksDB. Therefore the other stream perform a lookup to match the IPaddress.

 

 

Thanks

Srikanth