(DEPRECATED) Apache Flink User Mailing List archive.

No result shown when submitting the SQL in cli

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
tao xiao
Reply | Threaded
Open this post in threaded view
♦
♦
|

No result shown when submitting the SQL in cli

tao xiao
Hi team,

I wrote a simple SQL job to select data from Kafka. I can see results printing out in IDE but when I submit the job to a standalone cluster in CLI there is no result shown. I am sure the job is running well in the cluster with debug log suggesting that the kafka consumer is fetching data from Kafka. I enabled debug log in CLI and I don't see any obvious log. Here is the job code snippet

public static void main(String[] args) throws Exception {
  StreamTableEnvironment tableEnv = StreamTableEnvironment
      .create(StreamExecutionEnvironment.getExecutionEnvironment().setParallelism(1));

  String sqls = new String(Files.readAllBytes(Paths.get(args[0])));
  splitIgnoreQuota(sqls, ';').forEach(sql -> {
    TableResult tableResult = tableEnv.executeSql(sql);
    tableResult.print();
  });
}
It simply parses a sql file and execute the statements

Here is the SQL statements

CREATE TABLE t1 (
  `f1` STRING,
  `f2` STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'topic',
  'properties.group.id' = 'test1',
  'properties.max.partition.fetch.bytes' = '16384',
  'properties.enable.auto.commit' = 'false',
  'properties.bootstrap.servers' = 'kafka:9092',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

SELECT * FROM t1

Below is the result I got from IDE
| +I | b8f5 |           abcd |
| +I | b8f5 |           abcd |

And this is the result from CLI
bin/flink run  -m localhost:8081 -c kafka.sample.flink.SQLSample ~/workspaces/kafka-sample/target/kafka-sample-0.1.2-jar-with-dependencies.jar /sample.sql
+--------+
| result |
+--------+
|     OK |
+--------+
1 row in set
Job has been submitted with JobID ace45d2ff850675243e2663d3bf11701
+----+--------------------------------+--------------------------------+
| op |                           uuid |                            ots |
+----+--------------------------------+--------------------------------+


--
Regards,
Tao
tao xiao
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: No result shown when submitting the SQL in cli

tao xiao
Does anyone help with this question?

On Thu, May 6, 2021 at 3:26 PM tao xiao <[hidden email]> wrote:
Hi team,

I wrote a simple SQL job to select data from Kafka. I can see results printing out in IDE but when I submit the job to a standalone cluster in CLI there is no result shown. I am sure the job is running well in the cluster with debug log suggesting that the kafka consumer is fetching data from Kafka. I enabled debug log in CLI and I don't see any obvious log. Here is the job code snippet

public static void main(String[] args) throws Exception {
  StreamTableEnvironment tableEnv = StreamTableEnvironment
      .create(StreamExecutionEnvironment.getExecutionEnvironment().setParallelism(1));

  String sqls = new String(Files.readAllBytes(Paths.get(args[0])));
  splitIgnoreQuota(sqls, ';').forEach(sql -> {
    TableResult tableResult = tableEnv.executeSql(sql);
    tableResult.print();
  });
}
It simply parses a sql file and execute the statements

Here is the SQL statements

CREATE TABLE t1 (
  `f1` STRING,
  `f2` STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'topic',
  'properties.group.id' = 'test1',
  'properties.max.partition.fetch.bytes' = '16384',
  'properties.enable.auto.commit' = 'false',
  'properties.bootstrap.servers' = 'kafka:9092',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

SELECT * FROM t1

Below is the result I got from IDE
| +I | b8f5 |           abcd |
| +I | b8f5 |           abcd |

And this is the result from CLI
bin/flink run  -m localhost:8081 -c kafka.sample.flink.SQLSample ~/workspaces/kafka-sample/target/kafka-sample-0.1.2-jar-with-dependencies.jar /sample.sql
+--------+
| result |
+--------+
|     OK |
+--------+
1 row in set
Job has been submitted with JobID ace45d2ff850675243e2663d3bf11701
+----+--------------------------------+--------------------------------+
| op |                           uuid |                            ots |
+----+--------------------------------+--------------------------------+


--
Regards,
Tao


--
Regards,
Tao
Jeff Zhang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: No result shown when submitting the SQL in cli

Jeff Zhang
The result is printed in TM.
It is local mode in IDE, so TM runs in your local jvm that's why you see the result
While it is distributed mode (either yarn or standalone mode) when you are in sql-client, you should be able to see the result in TM logs.


tao xiao <[hidden email]> 于2021年5月11日周二 下午11:40写道：
Does anyone help with this question?

On Thu, May 6, 2021 at 3:26 PM tao xiao <[hidden email]> wrote:
Hi team,

I wrote a simple SQL job to select data from Kafka. I can see results printing out in IDE but when I submit the job to a standalone cluster in CLI there is no result shown. I am sure the job is running well in the cluster with debug log suggesting that the kafka consumer is fetching data from Kafka. I enabled debug log in CLI and I don't see any obvious log. Here is the job code snippet

public static void main(String[] args) throws Exception {
  StreamTableEnvironment tableEnv = StreamTableEnvironment
      .create(StreamExecutionEnvironment.getExecutionEnvironment().setParallelism(1));

  String sqls = new String(Files.readAllBytes(Paths.get(args[0])));
  splitIgnoreQuota(sqls, ';').forEach(sql -> {
    TableResult tableResult = tableEnv.executeSql(sql);
    tableResult.print();
  });
}
It simply parses a sql file and execute the statements

Here is the SQL statements

CREATE TABLE t1 (
  `f1` STRING,
  `f2` STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'topic',
  'properties.group.id' = 'test1',
  'properties.max.partition.fetch.bytes' = '16384',
  'properties.enable.auto.commit' = 'false',
  'properties.bootstrap.servers' = 'kafka:9092',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

SELECT * FROM t1

Below is the result I got from IDE
| +I | b8f5 |           abcd |
| +I | b8f5 |           abcd |

And this is the result from CLI
bin/flink run  -m localhost:8081 -c kafka.sample.flink.SQLSample ~/workspaces/kafka-sample/target/kafka-sample-0.1.2-jar-with-dependencies.jar /sample.sql
+--------+
| result |
+--------+
|     OK |
+--------+
1 row in set
Job has been submitted with JobID ace45d2ff850675243e2663d3bf11701
+----+--------------------------------+--------------------------------+
| op |                           uuid |                            ots |
+----+--------------------------------+--------------------------------+


--
Regards,
Tao


--
Regards,
Tao


--
Best Regards

Jeff Zhang
tao xiao
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: No result shown when submitting the SQL in cli

tao xiao
Thanks for the reply. Finally got it working. The output actually get printed out on client side not on TM side 

On Tue, May 11, 2021 at 11:47 PM Jeff Zhang <[hidden email]> wrote:
The result is printed in TM.
It is local mode in IDE, so TM runs in your local jvm that's why you see the result
While it is distributed mode (either yarn or standalone mode) when you are in sql-client, you should be able to see the result in TM logs.


tao xiao <[hidden email]> 于2021年5月11日周二 下午11:40写道：
Does anyone help with this question?

On Thu, May 6, 2021 at 3:26 PM tao xiao <[hidden email]> wrote:
Hi team,

I wrote a simple SQL job to select data from Kafka. I can see results printing out in IDE but when I submit the job to a standalone cluster in CLI there is no result shown. I am sure the job is running well in the cluster with debug log suggesting that the kafka consumer is fetching data from Kafka. I enabled debug log in CLI and I don't see any obvious log. Here is the job code snippet

public static void main(String[] args) throws Exception {
  StreamTableEnvironment tableEnv = StreamTableEnvironment
      .create(StreamExecutionEnvironment.getExecutionEnvironment().setParallelism(1));

  String sqls = new String(Files.readAllBytes(Paths.get(args[0])));
  splitIgnoreQuota(sqls, ';').forEach(sql -> {
    TableResult tableResult = tableEnv.executeSql(sql);
    tableResult.print();
  });
}
It simply parses a sql file and execute the statements

Here is the SQL statements

CREATE TABLE t1 (
  `f1` STRING,
  `f2` STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'topic',
  'properties.group.id' = 'test1',
  'properties.max.partition.fetch.bytes' = '16384',
  'properties.enable.auto.commit' = 'false',
  'properties.bootstrap.servers' = 'kafka:9092',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);

SELECT * FROM t1

Below is the result I got from IDE
| +I | b8f5 |           abcd |
| +I | b8f5 |           abcd |

And this is the result from CLI
bin/flink run  -m localhost:8081 -c kafka.sample.flink.SQLSample ~/workspaces/kafka-sample/target/kafka-sample-0.1.2-jar-with-dependencies.jar /sample.sql
+--------+
| result |
+--------+
|     OK |
+--------+
1 row in set
Job has been submitted with JobID ace45d2ff850675243e2663d3bf11701
+----+--------------------------------+--------------------------------+
| op |                           uuid |                            ots |
+----+--------------------------------+--------------------------------+


--
Regards,
Tao


--
Regards,
Tao


--
Best Regards

Jeff Zhang


--
Regards,
Tao
Free forum by Nabble Edit this page