Can't send data to another service in addSink

[Duy Truong]

Hi

In my flink program, I want to send data to a server via MQTT protocol, here is my code

https://gist.github.com/duytruong/d240958d2c4140b554b604cbef05edc7

The problem is it ran well on IntelliJ, but when submitted to cluster (bin/start-local.sh), it could connect but failed to send data. When I've tried to move connectToMqttServer() to addSink(), it worked (on cluster), so I have 2 questions:

1. Why it could connect but failed to send data, I guess the cause is addSink() is executed in different thread from the connectToMqttServer's thread but I can't find any document about it.

2. Why it ran well in IntelliJ but failed on cluster (I didn't move connectToMqttServer() to addSink() in this case).

Thanks,

--

[Chesnay Schepler]

Please read the Basic API concepts guide in the documentation, in particular https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/api_concepts.html#lazy-evaluation.

The short answer is that main() is called on the client, while the sink is executed on a taskmanager, i.e. in a different JVM. The sink must be thus self-contained, i.e. it must call connectToMqttServer().

On 18.09.2017 06:00, Duy Truong wrote:

Hi

In my flink program, I want to send data to a server via MQTT protocol, here is my code

https://gist.github.com/duytruong/d240958d2c4140b554b604cbef05edc7

The problem is it ran well on IntelliJ, but when submitted to cluster (bin/start-local.sh), it could connect but failed to send data. When I've tried to move connectToMqttServer() to addSink(), it worked (on cluster), so I have 2 questions:

1. Why it could connect but failed to send data, I guess the cause is addSink() is executed in different thread from the connectToMqttServer's thread but I can't find any document about it.

2. Why it ran well in IntelliJ but failed on cluster (I didn't move connectToMqttServer() to addSink() in this case).

Thanks,

--