BlinkPlanner limitation related clarification

[RKandoji]

Hi Team,

I've been using Blink Planner and just came across this page https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.9.html#known-shortcomings-or-limitations-for-new-features and saw below limitation:

Due to a bug with how transformations are not being cleared on execution, TableEnvironment instances should not be reused across multiple SQL statements when using the Blink planner.

In my code I've created a StreamTableEnvironment (like shown below) and reusing this instance everywhere for registering data streams, registering tables and performing multiple SQL queries. So I'm a bit concerned if I need to change anything? Would above limitation affect StreamTableEnvironment as well?

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

Could someone please clarify and provide more details about the implications.

Thanks,

RKandoji

[Jingsong Li]

Hi RKandoji,

IMO, yes, you can not reuse table env, you should create a new tEnv after executing, 1.9.1 still has this problem.

Related issue is [1], fixed in 1.9.2 and 1.10.

[1] https://issues.apache.org/jira/browse/FLINK-13708

Best,

Jingsong Lee

On Fri, Jan 24, 2020 at 11:14 AM RKandoji <[hidden email]> wrote:

Hi Team,

I've been using Blink Planner and just came across this page https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.9.html#known-shortcomings-or-limitations-for-new-features and saw below limitation:

Due to a bug with how transformations are not being cleared on execution, TableEnvironment instances should not be reused across multiple SQL statements when using the Blink planner.

In my code I've created a StreamTableEnvironment (like shown below) and reusing this instance everywhere for registering data streams, registering tables and performing multiple SQL queries. So I'm a bit concerned if I need to change anything? Would above limitation affect StreamTableEnvironment as well?

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

Could someone please clarify and provide more details about the implications.

Thanks,

RKandoji

--

[RKandoji]

Hi Jingsong,

Thanks for the information. Not sure if I'm missing anything but I have been reusing table env and didn't see anything wrong, I'm worried if I'm missed to note anything?

My use case:

I created a class level StreamTableEnvironment and used it throughout my code for creating multiple tables and running multiple SQL queries.

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

bsTableEnv.registerDataStream(...)

Table latestTbl1 = bsTableEnv.sqlQuery(...)

bsTableEnv.registerDataStream(...)
Table latestTbl2 = bsTableEnv.sqlQuery(...)
and so on..

Could you please let me know if anything specific I need to look at? I would like to understand what was wrong before changing the code. 

Thanks,

RK

On Thu, Jan 23, 2020 at 11:48 PM Jingsong Li <[hidden email]> wrote:

Hi RKandoji,

IMO, yes, you can not reuse table env, you should create a new tEnv after executing, 1.9.1 still has this problem.

Related issue is [1], fixed in 1.9.2 and 1.10.

[1] https://issues.apache.org/jira/browse/FLINK-13708

Best,

Jingsong Lee

On Fri, Jan 24, 2020 at 11:14 AM RKandoji <[hidden email]> wrote:

Hi Team,

I've been using Blink Planner and just came across this page https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.9.html#known-shortcomings-or-limitations-for-new-features and saw below limitation:

Due to a bug with how transformations are not being cleared on execution, TableEnvironment instances should not be reused across multiple SQL statements when using the Blink planner.

In my code I've created a StreamTableEnvironment (like shown below) and reusing this instance everywhere for registering data streams, registering tables and performing multiple SQL queries. So I'm a bit concerned if I need to change anything? Would above limitation affect StreamTableEnvironment as well?

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

Could someone please clarify and provide more details about the implications.

Thanks,

RKandoji

--

Best, Jingsong Lee

[Jingsong Li]

Hi RKandoji,

You understand this bug wrong, your code will not go wrong.

The bug is:

TableEnv tEnv = TableEnv.create(...);

Table t1 = tEnv.sqlQuery(...);

tEnv.insertInto("sink1", t1);

tEnv.execute("job1");

Table t2 = tEnv.sqlQuery(...);

tEnv.insertInto("sink2", t2);

tEnv.execute("job2");

This will wrong, job2 will be run contains the operators of job1.

If your job just have one "execute", it is OK.

Best,

Jingsong Lee

On Mon, Jan 27, 2020 at 12:14 AM RKandoji <[hidden email]> wrote:

Hi Jingsong,

Thanks for the information. Not sure if I'm missing anything but I have been reusing table env and didn't see anything wrong, I'm worried if I'm missed to note anything?

My use case:

I created a class level StreamTableEnvironment and used it throughout my code for creating multiple tables and running multiple SQL queries.

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

bsTableEnv.registerDataStream(...)

Table latestTbl1 = bsTableEnv.sqlQuery(...)

bsTableEnv.registerDataStream(...)
Table latestTbl2 = bsTableEnv.sqlQuery(...)
and so on..

Could you please let me know if anything specific I need to look at? I would like to understand what was wrong before changing the code. 

Thanks,

RK

On Thu, Jan 23, 2020 at 11:48 PM Jingsong Li <[hidden email]> wrote:

Hi RKandoji,

IMO, yes, you can not reuse table env, you should create a new tEnv after executing, 1.9.1 still has this problem.

Related issue is [1], fixed in 1.9.2 and 1.10.

[1] https://issues.apache.org/jira/browse/FLINK-13708

Best,

Jingsong Lee

On Fri, Jan 24, 2020 at 11:14 AM RKandoji <[hidden email]> wrote:

Hi Team,

I've been using Blink Planner and just came across this page https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.9.html#known-shortcomings-or-limitations-for-new-features and saw below limitation:

Due to a bug with how transformations are not being cleared on execution, TableEnvironment instances should not be reused across multiple SQL statements when using the Blink planner.

In my code I've created a StreamTableEnvironment (like shown below) and reusing this instance everywhere for registering data streams, registering tables and performing multiple SQL queries. So I'm a bit concerned if I need to change anything? Would above limitation affect StreamTableEnvironment as well?

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

Could someone please clarify and provide more details about the implications.

Thanks,

RKandoji

--

Best, Jingsong Lee

--

[RKandoji]

Hi Jingsong,

Thanks for the clarification!

The limitation description is a bit confusing to me but it was clear after seeing the above example posted by you.

Regards,

RK.

On Mon, Jan 27, 2020 at 6:25 AM Jingsong Li <[hidden email]> wrote:

Hi RKandoji,

You understand this bug wrong, your code will not go wrong.

The bug is:

TableEnv tEnv = TableEnv.create(...);

Table t1 = tEnv.sqlQuery(...);

tEnv.insertInto("sink1", t1);

tEnv.execute("job1");

Table t2 = tEnv.sqlQuery(...);

tEnv.insertInto("sink2", t2);

tEnv.execute("job2");

This will wrong, job2 will be run contains the operators of job1.

If your job just have one "execute", it is OK.

Best,

Jingsong Lee

On Mon, Jan 27, 2020 at 12:14 AM RKandoji <[hidden email]> wrote:

Hi Jingsong,

Thanks for the information. Not sure if I'm missing anything but I have been reusing table env and didn't see anything wrong, I'm worried if I'm missed to note anything?

My use case:

I created a class level StreamTableEnvironment and used it throughout my code for creating multiple tables and running multiple SQL queries.

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

bsTableEnv.registerDataStream(...)

Table latestTbl1 = bsTableEnv.sqlQuery(...)

bsTableEnv.registerDataStream(...)
Table latestTbl2 = bsTableEnv.sqlQuery(...)
and so on..

Could you please let me know if anything specific I need to look at? I would like to understand what was wrong before changing the code. 

Thanks,

RK

On Thu, Jan 23, 2020 at 11:48 PM Jingsong Li <[hidden email]> wrote:

Hi RKandoji,

IMO, yes, you can not reuse table env, you should create a new tEnv after executing, 1.9.1 still has this problem.

Related issue is [1], fixed in 1.9.2 and 1.10.

[1] https://issues.apache.org/jira/browse/FLINK-13708

Best,

Jingsong Lee

On Fri, Jan 24, 2020 at 11:14 AM RKandoji <[hidden email]> wrote:

Hi Team,

I've been using Blink Planner and just came across this page https://ci.apache.org/projects/flink/flink-docs-stable/release-notes/flink-1.9.html#known-shortcomings-or-limitations-for-new-features and saw below limitation:

Due to a bug with how transformations are not being cleared on execution, TableEnvironment instances should not be reused across multiple SQL statements when using the Blink planner.

In my code I've created a StreamTableEnvironment (like shown below) and reusing this instance everywhere for registering data streams, registering tables and performing multiple SQL queries. So I'm a bit concerned if I need to change anything? Would above limitation affect StreamTableEnvironment as well?

private static StreamTableEnvironment bsTableEnv = StreamTableEnvironment.create(bsEnv, bsSettings);

Could someone please clarify and provide more details about the implications.

Thanks,

RKandoji

--

Best, Jingsong Lee

--

Best, Jingsong Lee