(DEPRECATED) Apache Flink User Mailing List archive.

Is Flink easy to deploy ?

classic Classic list List threaded Threaded
♦
♦
 7 messages Options
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Is Flink easy to deploy ?

Esa Heikkinen

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Is Flink easy to deploy ?

Fabian Hueske-2

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa
Esa Heikkinen-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Is Flink easy to deploy ?

Esa Heikkinen-2


Yes i have looked. For example, if i want to compile and run StreamTableExample.scala from:

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala

I have taken all examples (and also latest Flink at same time) to my Linux from git.

Where directory should i be in for compiling and running StreamTableExample in command line ? flink-examples-table ?

What is the command for compiling ? mvn clean install -Pbuild-jar ?

What is the command running the StreamTableExample ? /<path to flink bin>/flink run -c org.apache.flink.table.examples.scala.StreamTableExample target/flink-examples-table_2.1-1.5-SNAPSHOT.jar ?

This does not work because of error: java.lang.NoClassDefFoundError:...

BR Esa


Fabian Hueske kirjoitti 23.2.2018 klo 15:07:

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa
Niclas Hedhman
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Is Flink easy to deploy ?

Niclas Hedhman
I think you simply missing a bunch of the Flink artifacts.

Flink is broken into dozens of pieces, and you need to select from a large set of artifacts what you need to depend on.

Typically, there is one Flink artifact per "extension".
I ended up with 
   flink-core
   flink-core
   flink-connector-cassandra_2.11
   flink-connector-kafka_2.11
   flink-queryable-state-runtime_2.11
   flink-streaming-java_2.11
   flink-streaming-scala_2.11

With transitive dependencies enabled, meaning whatever Flink depends on is also included

Cheers

On Sat, Feb 24, 2018 at 3:22 AM, Esa Heikkinen <[hidden email]> wrote:


Yes i have looked. For example, if i want to compile and run StreamTableExample.scala from:

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala

I have taken all examples (and also latest Flink at same time) to my Linux from git.

Where directory should i be in for compiling and running StreamTableExample in command line ? flink-examples-table ?

What is the command for compiling ? mvn clean install -Pbuild-jar ?

What is the command running the StreamTableExample ? /<path to flink bin>/flink run -c org.apache.flink.table.examples.scala.StreamTableExample target/flink-examples-table_2.1-1.5-SNAPSHOT.jar ?

This does not work because of error: java.lang.NoClassDefFoundError:...

BR Esa


Fabian Hueske kirjoitti 23.2.2018 klo 15:07:

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa






--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Esa Heikkinen-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Is Flink easy to deploy ?

Esa Heikkinen-2
In reply to this post by Esa Heikkinen-2

Hi

I have done like that, but no "checkstyle" settings. Checkstyle only affects how the code looks ?

But should it be possible to compile and run without IDE ?

Best,
Esa

xingcan kirjoitti 24.2.2018 klo 3:24:
Hi Esa,

maybe you can follow the instructions to import Flink to your IDE and run the examples in it. That’s a good beginning I think.

Best,
Xingcan

On 24 Feb 2018, at 3:22 AM, Esa Heikkinen <[hidden email]> wrote:


Yes i have looked. For example, if i want to compile and run StreamTableExample.scala from:

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala

I have taken all examples (and also latest Flink at same time) to my Linux from git.

Where directory should i be in for compiling and running StreamTableExample in command line ? flink-examples-table ?

What is the command for compiling ? mvn clean install -Pbuild-jar ?

What is the command running the StreamTableExample ? /<path to flink bin>/flink run -c org.apache.flink.table.examples.scala.StreamTableExample target/flink-examples-table_2.1-1.5-SNAPSHOT.jar ?

This does not work because of error: java.lang.NoClassDefFoundError:...

BR Esa


Fabian Hueske kirjoitti 23.2.2018 klo 15:07:

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Is Flink easy to deploy ?

Esa Heikkinen
In reply to this post by Esa Heikkinen-2

Hi

 

Whether these instructions of IDE are only for Java, but no or Scala ?

 

Best,

Esa

 

From: xingcan [mailto:[hidden email]]
Sent: Saturday, February 24, 2018 3:25 AM
To: Esa Heikkinen <[hidden email]>
Cc: [hidden email]
Subject: Re: Is Flink easy to deploy ?

 

Hi Esa,

 

maybe you can follow the instructions to import Flink to your IDE and run the examples in it. That’s a good beginning I think.

 

Best,

Xingcan



On 24 Feb 2018, at 3:22 AM, Esa Heikkinen <[hidden email]> wrote:

 

 

Yes i have looked. For example, if i want to compile and run StreamTableExample.scala from:

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala

I have taken all examples (and also latest Flink at same time) to my Linux from git.

Where directory should i be in for compiling and running StreamTableExample in command line ? flink-examples-table ?

What is the command for compiling ? mvn clean install -Pbuild-jar ?

What is the command running the StreamTableExample ? /<path to flink bin>/flink run -c org.apache.flink.table.examples.scala.StreamTableExample target/flink-examples-table_2.1-1.5-SNAPSHOT.jar ?

This does not work because of error: java.lang.NoClassDefFoundError:...

BR Esa

 

Fabian Hueske kirjoitti 23.2.2018 klo 15:07:

 

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa

 

 

 
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: Is Flink easy to deploy ?

Esa Heikkinen
In reply to this post by Niclas Hedhman

 

I’d want to write simple Scala code that: 1) reads data of csv-file (time series data, where one column is timestamp) 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results. (I would very much like to find a complete example of this.)

 

What artifacts and “imports” do I need ?

 

Whether there is a list of all artifacts ?

 

And what “imports” is included in one artifacts ?

 

And what classes and “functions” is included in one “import” ?

 

Is it problem to include (almost) all possible artifacts and “imports” to project and code ?

 

By the way, is it right way to edit directly XML-code in pom.xml to change artifacts ? pom.xml looks like a bit of cryptic and I am not sure I edit  right places in pom.xml because <dependency>’s ja <artifactId>’s are in many places.

 

Best,

Esa

 

From: Niclas Hedhman [mailto:[hidden email]]
Sent: Saturday, February 24, 2018 4:25 AM
To: [hidden email]
Cc: Esa Heikkinen <[hidden email]>
Subject: Re: Is Flink easy to deploy ?

 

I think you simply missing a bunch of the Flink artifacts.

 

Flink is broken into dozens of pieces, and you need to select from a large set of artifacts what you need to depend on.

 

Typically, there is one Flink artifact per "extension".

I ended up with 

   flink-core

   flink-core

   flink-connector-cassandra_2.11

   flink-connector-kafka_2.11

   flink-queryable-state-runtime_2.11

   flink-streaming-java_2.11

   flink-streaming-scala_2.11

 

With transitive dependencies enabled, meaning whatever Flink depends on is also included

 

Cheers

 

On Sat, Feb 24, 2018 at 3:22 AM, Esa Heikkinen <[hidden email]> wrote:

 

Yes i have looked. For example, if i want to compile and run StreamTableExample.scala from:

https://github.com/apache/flink/blob/master/flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala/StreamTableExample.scala

I have taken all examples (and also latest Flink at same time) to my Linux from git.

Where directory should i be in for compiling and running StreamTableExample in command line ? flink-examples-table ?

What is the command for compiling ? mvn clean install -Pbuild-jar ?

What is the command running the StreamTableExample ? /<path to flink bin>/flink run -c org.apache.flink.table.examples.scala.StreamTableExample target/flink-examples-table_2.1-1.5-SNAPSHOT.jar ?

This does not work because of error: java.lang.NoClassDefFoundError:...

BR Esa

 

Fabian Hueske kirjoitti 23.2.2018 klo 15:07:

 

2018-02-23 13:30 GMT+01:00 Esa Heikkinen <[hidden email]>:

I have lot of difficulties to deploy Flink. That is maybe because I am new with Flink and its (Java and Maven) development environment, but I would hear the opinions of others. I would like to use Scala.

 

There are many examples, but often there are missing “imports” and settings in pom.xml. It seem to be very hard to job to find correct ones. Maybe use of IDE (IntelliJ IDEA) is almost mandatory and it helps to find “imports”, but it does not find all of them.

 

Generally you have to do and study a lot of basic work before you get into the actual thing ?

 

If there is a ready example (with source code) that is enough close to what you want, it is much easier to deploy. But if not, it can be a surprisingly difficult and time-consuming task. Because the documentation seem to be partially incomplete, it is often necessary to “google” and query the mailing list.

 

Or have I misunderstand something or I can not use Flink correctly (yet) ?

 

Features of Flink are so good that I would want to learn to use it.

 

Best Regards

Esa

 

 




--

Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Free forum by Nabble Edit this page