Joining on multiple row values produced by TableFunction

[Samuel Doyle]

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

[Fabian Hueske-2]

Hi,

it might be a problem, that your TableFunction is called like an attribute (both are called fields).

If that's not the case, can you provide more information?

- Is there an error message?

  - Yes: what's the error message

  - No: what's the output and what would you expect?

- Is this for batch or streaming?

Thanks, Fabian

2017-05-09 0:27 GMT+02:00 Samuel Doyle <[hidden email]>:

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

[Samuel Doyle]

I was able to get it work, albeit probably not the best solution, by calling the table function twice and assigning different column aliases

On Tue, May 9, 2017, 12:43 AM Fabian Hueske <[hidden email]> wrote:

Hi,

it might be a problem, that your TableFunction is called like an attribute (both are called fields).

If that's not the case, can you provide more information?

- Is there an error message?

  - Yes: what's the error message

  - No: what's the output and what would you expect?

- Is this for batch or streaming?

Thanks, Fabian

2017-05-09 0:27 GMT+02:00 Samuel Doyle <[hidden email]>:

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

--

[Samuel Doyle]

Is it possible to define windows in such a way that I can use straight sql like with the table environment vs the sql 'like' syntax you use when using the table api?

My project is based off Java.

Thanks

On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <[hidden email]> wrote:

I was able to get it work, albeit probably not the best solution, by calling the table function twice and assigning different column aliases

On Tue, May 9, 2017, 12:43 AM Fabian Hueske <[hidden email]> wrote:

Hi,

it might be a problem, that your TableFunction is called like an attribute (both are called fields).

If that's not the case, can you provide more information?

- Is there an error message?

  - Yes: what's the error message

  - No: what's the output and what would you expect?

- Is this for batch or streaming?

Thanks, Fabian

2017-05-09 0:27 GMT+02:00 Samuel Doyle <[hidden email]>:

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

--

--

[Fabian Hueske-2]

Flink 1.2 does not support windows in SQL.
The upcoming Flink 1.3, which is currently in testing phase and should be released in a few weeks, allows you define several types of windows in SQL.

There are three types of group windows [1] (note that the syntax for the time indicators changed a bit, check the test cases for example [2]) and OVER windows as know from standard SQL (limitations are all aggregations in a SELECT stmt have to operate on the same window, order by only on time, and no FOLLOWING rows/range). Examples for OVER windows are in the tests as well [3].

Hope this helps,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table_api.html#group-windows
[2] https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/WindowAggregateTest.scala
[3] https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/OverWindowITCase.scala

2017-05-09 21:59 GMT+02:00 Samuel Doyle <[hidden email]>:

Is it possible to define windows in such a way that I can use straight sql like with the table environment vs the sql 'like' syntax you use when using the table api?

My project is based off Java.

Thanks

On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <[hidden email]> wrote:

I was able to get it work, albeit probably not the best solution, by calling the table function twice and assigning different column aliases

On Tue, May 9, 2017, 12:43 AM Fabian Hueske <[hidden email]> wrote:

Hi,

it might be a problem, that your TableFunction is called like an attribute (both are called fields).

If that's not the case, can you provide more information?

- Is there an error message?

  - Yes: what's the error message

  - No: what's the output and what would you expect?

- Is this for batch or streaming?

Thanks, Fabian

2017-05-09 0:27 GMT+02:00 Samuel Doyle <[hidden email]>:

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

--

--

[Samuel Doyle]

Great, thanks for the information! Looking forward to the new release.

S.D.

On Tue, May 9, 2017 at 1:40 PM Fabian Hueske <[hidden email]> wrote:

Flink 1.2 does not support windows in SQL.
The upcoming Flink 1.3, which is currently in testing phase and should be released in a few weeks, allows you define several types of windows in SQL.

There are three types of group windows [1] (note that the syntax for the time indicators changed a bit, check the test cases for example [2]) and OVER windows as know from standard SQL (limitations are all aggregations in a SELECT stmt have to operate on the same window, order by only on time, and no FOLLOWING rows/range). Examples for OVER windows are in the tests as well [3].

Hope this helps,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table_api.html#group-windows
[2] https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/WindowAggregateTest.scala
[3] https://github.com/apache/flink/blob/release-1.3/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/api/scala/stream/sql/OverWindowITCase.scala

2017-05-09 21:59 GMT+02:00 Samuel Doyle <[hidden email]>:

Is it possible to define windows in such a way that I can use straight sql like with the table environment vs the sql 'like' syntax you use when using the table api?

My project is based off Java.

Thanks

On Tue, May 9, 2017 at 10:34 AM Samuel Doyle <[hidden email]> wrote:

I was able to get it work, albeit probably not the best solution, by calling the table function twice and assigning different column aliases

On Tue, May 9, 2017, 12:43 AM Fabian Hueske <[hidden email]> wrote:

Hi,

it might be a problem, that your TableFunction is called like an attribute (both are called fields).

If that's not the case, can you provide more information?

- Is there an error message?

  - Yes: what's the error message

  - No: what's the output and what would you expect?

- Is this for batch or streaming?

Thanks, Fabian

2017-05-09 0:27 GMT+02:00 Samuel Doyle <[hidden email]>:

I want to do something like the following

            .join("fields(fields) as (name, content)")

            .where("text = 'password for user' && name='text' && !content.like('%accepted%') && name='appname' && content.like('%hostd%')")

Fields collects 4 rows in this case which contain those values

This doesn't work with flink 1.2. Is there a way to accomplish this?

…

--

--

--

--