(DEPRECATED) Apache Flink User Mailing List archive.

Re: Join of DataStream and DataSet

Posted by Hequn Cheng on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Join-of-DataStream-and-DataSet-tp27211p27279.html

Hi Reminia,

Currently, we can't join a DataStream with a DataSet in Flink. However, the DataSet is actually a kind of bounded stream. From the point of this view, you can use a streaming job to achieve your goal. Flink Table API & SQL support different kinds of join[1]. You can take a closer look at them. Probably a regular join[2] is ok for you.

Finally, I think you raised a very good point. It would be better if Flink can support such kind of join more direct and efficient.

Best, Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/tableApi.html#joins

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#regular-joins

On Thu, Apr 11, 2019 at 5:16 PM Reminia Scarlet <[hidden email]> wrote:

Spark streaming supports direct join from stream DataFrame and batch DataFrame , and it's
easy to implement an enrich pipeline that joins a stream and a dimension table.

I checked the doc of flink, seems that this feature is a jira ticket which haven't been resolved yet.

So how can I implement such a pipeline easily in Flink?