(DEPRECATED) Apache Flink User Mailing List archive.

CEP for time series in csv-file

classic Classic list List threaded Threaded
♦
♦
 6 messages Options
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

CEP for time series in csv-file

Esa Heikkinen

Hi

 

I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.

 

I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results

 

Do you have any hints or examples how to do that ?

 

By the way, what kind of time stamp should be in csv-file ?

 

 
Kostas Kloudas
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: CEP for time series in csv-file

Kostas Kloudas
Hi Esa,

I think the best place to start is the documentation available at the flink website.

Some pointers are the following: 


Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update

Cheers,
Kostas

On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <[hidden email]> wrote:

Hi
 
I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.
 
I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results
 
Do you have any hints or examples how to do that ?
 
By the way, what kind of time stamp should be in csv-file ?
Timo Walther
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: CEP for time series in csv-file

Timo Walther
You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:

http://training.data-artisans.com/exercises/CEP.html

Regards,
Timo


Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:
Hi Esa,

I think the best place to start is the documentation available at the flink website.

Some pointers are the following: 


Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update

Cheers,
Kostas

On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <[hidden email]> wrote:

Hi
 
I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.
 
I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results
 
Do you have any hints or examples how to do that ?
 
By the way, what kind of time stamp should be in csv-file ?
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: CEP for time series in csv-file

Esa Heikkinen

Hi

 

Thanks for the hints, but I am still very interested about simple working example with combination: sbt-project, scala, csv-file reading and cep processing. I have did not exactly find something like that. It would help me a lot.

 

It takes lot of time to learn and test many possible code combinations.. Too many “moving” parts..

For example “huge” amount of different “imports” and where I can find how use them and so on ?

I did not find strict “reference” guide. For example for readCsvFile(). Or should I look it from code ?

 

By the way what is better to use maven or sbt ? It seems most of examples use maven, but I haven’t got maven to work properly (yet) ..

 

Best Regards

Esa

 

From: Timo Walther [mailto:[hidden email]]
Sent: Thursday, February 8, 2018 7:23 PM
To: [hidden email]
Subject: Re: CEP for time series in csv-file

 

You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:

http://training.data-artisans.com/exercises/CEP.html

Regards,
Timo


Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:

Hi Esa,

 

I think the best place to start is the documentation available at the flink website.

 

Some pointers are the following: 

 

 

Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update

 

Cheers,

Kostas



On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <[hidden email]> wrote:

 

Hi

 

I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.

 

I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results

 

Do you have any hints or examples how to do that ?

 

By the way, what kind of time stamp should be in csv-file ?

 

 
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: CEP for time series in csv-file

Fabian Hueske-2
Hi,

I'm not aware of any example project that ticks all your requirements. As you said, too many combinations...
Flink uses Maven. So, most examples provided by committers will be based on Maven.

I would recommend to start with the Scala Maven quickstart archetypes [1].
Next, I'd implement a source function that reads your data from a single file. I've given some pointers in a previous mail.
Once you can read the data, I'd add the CEP library.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/scala_api_quickstart.html#maven

2018-02-09 15:35 GMT+01:00 Esa Heikkinen <[hidden email]>:

Hi

 

Thanks for the hints, but I am still very interested about simple working example with combination: sbt-project, scala, csv-file reading and cep processing. I have did not exactly find something like that. It would help me a lot.

 

It takes lot of time to learn and test many possible code combinations.. Too many “moving” parts..

For example “huge” amount of different “imports” and where I can find how use them and so on ?

I did not find strict “reference” guide. For example for readCsvFile(). Or should I look it from code ?

 

By the way what is better to use maven or sbt ? It seems most of examples use maven, but I haven’t got maven to work properly (yet) ..

 

Best Regards

Esa

 

From: Timo Walther [mailto:[hidden email]]
Sent: Thursday, February 8, 2018 7:23 PM
To: [hidden email]
Subject: Re: CEP for time series in csv-file

 

You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:

http://training.data-artisans.com/exercises/CEP.html

Regards,
Timo


Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:

Hi Esa,

 

I think the best place to start is the documentation available at the flink website.

 

Some pointers are the following: 

 

 

Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update

 

Cheers,

Kostas



On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <[hidden email]> wrote:

 

Hi

 

I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.

 

I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results

 

Do you have any hints or examples how to do that ?

 

By the way, what kind of time stamp should be in csv-file ?

 

 
Esa Heikkinen
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: CEP for time series in csv-file

Esa Heikkinen

Hi

 

I have little problem with this example about Scala maven. I created new project by command: “mvn archetype:generate …” as stated in the instructions.

 

What is the correct build command ? “mvn package –Pbuild-jar” in same directory with pom.xml ?

For example, it does not generate WordCount.jar. I only found target/art-1.0-SNAPSHOT.jar.

 

What is the correct run command of flink for WordCount in this case ?

 

Can I run it without starting of Flink (start-local.sh) and browser-interface (in port 8081)?

 

What did I do wrong ?

 

 

From: Fabian Hueske [mailto:[hidden email]]
Sent: Saturday, February 10, 2018 1:07 PM
To: Esa Heikkinen <[hidden email]>
Cc: Timo Walther <[hidden email]>; [hidden email]
Subject: Re: CEP for time series in csv-file

 

Hi,

I'm not aware of any example project that ticks all your requirements. As you said, too many combinations...

Flink uses Maven. So, most examples provided by committers will be based on Maven.

I would recommend to start with the Scala Maven quickstart archetypes [1].

Next, I'd implement a source function that reads your data from a single file. I've given some pointers in a previous mail.

Once you can read the data, I'd add the CEP library.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/scala_api_quickstart.html#maven

 

2018-02-09 15:35 GMT+01:00 Esa Heikkinen <[hidden email]>:

Hi

 

Thanks for the hints, but I am still very interested about simple working example with combination: sbt-project, scala, csv-file reading and cep processing. I have did not exactly find something like that. It would help me a lot.

 

It takes lot of time to learn and test many possible code combinations.. Too many “moving” parts..

For example “huge” amount of different “imports” and where I can find how use them and so on ?

I did not find strict “reference” guide. For example for readCsvFile(). Or should I look it from code ?

 

By the way what is better to use maven or sbt ? It seems most of examples use maven, but I haven’t got maven to work properly (yet) ..

 

Best Regards

Esa

 

From: Timo Walther [mailto:[hidden email]]
Sent: Thursday, February 8, 2018 7:23 PM
To: [hidden email]
Subject: Re: CEP for time series in csv-file

 

You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:

http://training.data-artisans.com/exercises/CEP.html

Regards,
Timo


Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:

Hi Esa,

 

I think the best place to start is the documentation available at the flink website.

 

Some pointers are the following: 

 

 

Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update

 

Cheers,

Kostas

 

On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <[hidden email]> wrote:

 

Hi

 

I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.

 

I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP  5) writes results

 

Do you have any hints or examples how to do that ?

 

By the way, what kind of time stamp should be in csv-file ?

 

 

 
Free forum by Nabble Edit this page