Issue with using siddhi extension function with flink

[Dipanjan Mazumder]

Hi ,

   i am trying to integrate siddhi with flink while trying to use siddhi extension function on deploying the job in flink cluster it is not able to find those libraries at run time , so i had to explicitly put those libraries to the /opt/flink/lib folder for the jobmanager and taskmanager , fat jar of the flink job application has those libraries but it cannot identify those extension functions at runtime and putting them to the lib folder is not a feasible choice. Can you give some pointer on this problem.. thanks in advance ..

I have tried multiple ways to load the classes using class.forname etc.. but nothing works even if the fat jar for the flink job application has the siddhi extensions in it.i don’t want to add those libraries to the jobmanage and taskmanagers lib folder everytime.

Any help will be appreciated.

Regards

Dipanjan

[Till Rohrmann]

Hi Dipanjan,

Please double check whether the libraries are really contained in the job jar you are submitting because if the library is contained in this jar, then it should be on the classpath and you should be able to load it.

Cheers,

Till

On Thu, May 20, 2021 at 3:43 PM Dipanjan Mazumder <[hidden email]> wrote:

Hi ,

   i am trying to integrate siddhi with flink while trying to use siddhi extension function on deploying the job in flink cluster it is not able to find those libraries at run time , so i had to explicitly put those libraries to the /opt/flink/lib folder for the jobmanager and taskmanager , fat jar of the flink job application has those libraries but it cannot identify those extension functions at runtime and putting them to the lib folder is not a feasible choice. Can you give some pointer on this problem.. thanks in advance ..

I have tried multiple ways to load the classes using class.forname etc.. but nothing works even if the fat jar for the flink job application has the siddhi extensions in it.i don’t want to add those libraries to the jobmanage and taskmanagers lib folder everytime.

Any help will be appreciated.

Regards

Dipanjan

[Salva Alcántara]

Hi Dipanjan,

I agree with Till. If the extensions are are included in the jar for your
job, it should work. I was having the same doubts some weeks a go and can
confirm that as long as the jar includes those extensions, it works.

One thing I needed to do is to register the different extensions. For
example:

```
siddhiManager.setExtension("map:create", classOf[CreateFunctionExtension])
```

Regards,

Salva



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Dipanjan Mazumder]

Hi All,

    Found the solution , 

Problem: I was actually using an intermediate library to integrate siddhi with Flink (https://github.com/haoch/flink-siddhi) and i was creating a SiddhiCEP instance and then calling "define()" on that instance , while i was registering the extension on the created instance , but the define method was creating an internal SiddhiCEP instance and using that for processing. So i found that out by debugging the application. It is an implementation problem with the library itself.

Solution: i used from() method on the created SiddhiCEP instance instead of define() and it was using the CEP i created for doing the rest of processing and the registered extension were accounted similarly and so they were recognised at runtime.

Regards

Dipanjan

On Friday, May 21, 2021, 01:51:09 PM GMT+5:30, Salva Alcántara <[hidden email]> wrote:

Hi Dipanjan,

I agree with Till. If the extensions are are included in the jar for your
job, it should work. I was having the same doubts some weeks a go and can
confirm that as long as the jar includes those extensions, it works.

One thing I needed to do is to register the different extensions. For
example:

```
siddhiManager.setExtension("map:create", classOf[CreateFunctionExtension])

```

Regards,

Salva



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/