(DEPRECATED) Apache Flink User Mailing List archive.

Zeppelin Integration

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
Trevor Grant
Reply | Threaded
Open this post in threaded view
♦
♦
|

Zeppelin Integration

Trevor Grant
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Till Rohrmann

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Trevor Grant
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Trevor Grant
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Till Rohrmann

Hi Trevor,

that’s actually my bad since I only tested my branch against a remote cluster. I fixed the problem (not properly starting the LocalFlinkMiniCluster) so that you can now use Zeppelin also in local mode. Just check out my branch again.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 10:00 PM, Trevor Grant <[hidden email]> wrote:
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
rmetzger0
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

rmetzger0
For those interested, Trevor wrote a blog post describing how to setup Spark, Flink and Zeppelin, both locally and on clusters: http://trevorgrant.org/2015/11/03/apache-casserole-a-delicious-big-data-recipe-for-the-whole-family/
Thanks Trevor for the great tutorial!

On Thu, Oct 22, 2015 at 4:23 PM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

that’s actually my bad since I only tested my branch against a remote cluster. I fixed the problem (not properly starting the LocalFlinkMiniCluster) so that you can now use Zeppelin also in local mode. Just check out my branch again.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 10:00 PM, Trevor Grant <[hidden email]> wrote:
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Till Rohrmann
Really cool tutorial Trevor :-)

On Wed, Nov 4, 2015 at 3:26 PM, Robert Metzger <[hidden email]> wrote:
For those interested, Trevor wrote a blog post describing how to setup Spark, Flink and Zeppelin, both locally and on clusters: http://trevorgrant.org/2015/11/03/apache-casserole-a-delicious-big-data-recipe-for-the-whole-family/
Thanks Trevor for the great tutorial!

On Thu, Oct 22, 2015 at 4:23 PM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

that’s actually my bad since I only tested my branch against a remote cluster. I fixed the problem (not properly starting the LocalFlinkMiniCluster) so that you can now use Zeppelin also in local mode. Just check out my branch again.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 10:00 PM, Trevor Grant <[hidden email]> wrote:
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Leonard Wolters
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Leonard Wolters

Indeed very nice! Thanks

On Nov 4, 2015 5:04 PM, "Till Rohrmann" <[hidden email]> wrote:
Really cool tutorial Trevor :-)

On Wed, Nov 4, 2015 at 3:26 PM, Robert Metzger <[hidden email]> wrote:
For those interested, Trevor wrote a blog post describing how to setup Spark, Flink and Zeppelin, both locally and on clusters: http://trevorgrant.org/2015/11/03/apache-casserole-a-delicious-big-data-recipe-for-the-whole-family/
Thanks Trevor for the great tutorial!

On Thu, Oct 22, 2015 at 4:23 PM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

that’s actually my bad since I only tested my branch against a remote cluster. I fixed the problem (not properly starting the LocalFlinkMiniCluster) so that you can now use Zeppelin also in local mode. Just check out my branch again.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 10:00 PM, Trevor Grant <[hidden email]> wrote:
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Vasiliki Kalavri
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Zeppelin Integration

Vasiliki Kalavri
Great tutorial! Thanks a lot ^^

On 4 November 2015 at 17:12, Leonard Wolters <[hidden email]> wrote:

Indeed very nice! Thanks

On Nov 4, 2015 5:04 PM, "Till Rohrmann" <[hidden email]> wrote:
Really cool tutorial Trevor :-)

On Wed, Nov 4, 2015 at 3:26 PM, Robert Metzger <[hidden email]> wrote:
For those interested, Trevor wrote a blog post describing how to setup Spark, Flink and Zeppelin, both locally and on clusters: http://trevorgrant.org/2015/11/03/apache-casserole-a-delicious-big-data-recipe-for-the-whole-family/
Thanks Trevor for the great tutorial!

On Thu, Oct 22, 2015 at 4:23 PM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

that’s actually my bad since I only tested my branch against a remote cluster. I fixed the problem (not properly starting the LocalFlinkMiniCluster) so that you can now use Zeppelin also in local mode. Just check out my branch again.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 10:00 PM, Trevor Grant <[hidden email]> wrote:
Hey Till,

I cloned your branch of Zeplin and while it will compile, it fails tests on timeout, which consequently was the same issue I was having when trying to use Zeppelin. 

Ideas?

-------------------------------------------------------------------------------
Test set: org.apache.zeppelin.flink.FlinkInterpreterTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 100.347 sec <<< FAILURE! - in org.apache.zeppelin.flink.FlinkInterpreterTest
org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.flink.runtime.minicluster.FlinkMiniCluster.getLeaderIndex(FlinkMiniCluster.scala:171)
at org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.getLeaderRPCPort(LocalFlinkMiniCluster.scala:132)
at org.apache.zeppelin.flink.FlinkInterpreter.getPort(FlinkInterpreter.java:136)
at org.apache.zeppelin.flink.FlinkInterpreter.open(FlinkInterpreter.java:98)
at org.apache.zeppelin.flink.FlinkInterpreterTest.setUp(FlinkInterpreterTest.java:42)

org.apache.zeppelin.flink.FlinkInterpreterTest  Time elapsed: 100.347 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.zeppelin.flink.FlinkInterpreter.close(FlinkInterpreter.java:221)
at org.apache.zeppelin.flink.FlinkInterpreterTest.tearDown(FlinkInterpreterTest.java:48)



Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil


On Wed, Oct 21, 2015 at 11:57 AM, Till Rohrmann <[hidden email]> wrote:

Hi Trevor,

in order to use Zeppelin with a different Flink version in local mode, meaning that Zeppelin starts a LocalFlinkMiniCluster when executing your jobs, you have to build Zeppelin and change the flink.version property in the zeppelin/flink/pom.xml file to the version you want to use.

If you want to let Zeppelin submit jobs to a remote cluster, you should build Zeppelin with the version of your cluster. That’s because internally Zeppelin will use this version to construct a JobGraph which is then submitted to the cluster. In order to configure the remote cluster, you have to go the Interpreter page and scroll down to the flink section. There you have to specify the address of your cluster under host and the port under port. This should then be used to submit jobs to the Flink cluster.

I hope this answers your question.

Btw: If you want to use Zeppelin with the latest Flink 0.10-SNAPSHOT version, you should checkout my branch https://github.com/tillrohrmann/incubator-zeppelin/tree/flink-0.10-SNAPSHOT where I’ve made the necessary changes.

Cheers,
Till

​

On Wed, Oct 21, 2015 at 5:00 PM, Trevor Grant <[hidden email]> wrote:
I'm setting up some Flink/Spark/Zeppelin at work.  Spark+Zeppelin seems to be relatively well supported and configurable but the Flink is not so much.

I want Zeppelin to run against my 0.10 build instead of the 0.6 build that ships with Zeppelin.  My best guess at the moment on how to accomplish this is to create a symbolic link from the /opt/zepplin/flink folder to /opt/flink-0.10, but this feels dirty and wrong.

Does anyone out there have any experience connecting Zeppelin to a non-prepackaged Flink build?

I feel like there is a great opporutnity for a HOWTO write up if non currently exists.

I'm asking on the Zeppelin user mailing list too as soon as I am added. 

Thanks for any help

tg


Trevor Grant
Data Scientist

"Fortunate is he, who is able to know the causes of things."  -Virgil
Free forum by Nabble Edit this page