Connector for REST End Point

[Archit Mittal]

Hi

Can we write a custom connector listening to a REST Api end point  in Flink

Thanks

Archit

[Chesnay Schepler]

Hello,

Please give us a bit more information, I'm really not sure what you mean. :(

Regards,
Chesnay

On 06.04.2017 09:40, Archit Mittal wrote:
> Hi
>
> Can we write a custom connector listening to a REST Api end point  in
> Flink
>
> Thanks
> Archit

[Archit Mittal]

Hi Chesnay

I have a REST Api that listens to events that come through Kafka but i dont have access to that Kafka . Can I write a connector that consumes events through the REST end point and not directly through Kafka. 

I have a solution that i can dump the events coming to the api into another Kafka instance and consume through that but it will add another hop to the flow. 

Is There a better way to do this by writing a custom connector or something like that . 

Thanks

Archit

On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

Please give us a bit more information, I'm really not sure what you mean. :(

Regards,
Chesnay

On 06.04.2017 09:40, Archit Mittal wrote:

Hi

Can we write a custom connector listening to a REST Api end point  in Flink

Thanks
Archit

[Archit Mittal]

Hi

Any suggestions regarding this.

Thanks

Archit

On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <[hidden email]> wrote:

Hi Chesnay

I have a REST Api that listens to events that come through Kafka but i dont have access to that Kafka . Can I write a connector that consumes events through the REST end point and not directly through Kafka. 

I have a solution that i can dump the events coming to the api into another Kafka instance and consume through that but it will add another hop to the flow. 

Is There a better way to do this by writing a custom connector or something like that . 

Thanks

Archit

On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

Please give us a bit more information, I'm really not sure what you mean. :(

Regards,
Chesnay

On 06.04.2017 09:40, Archit Mittal wrote:

Hi

Can we write a custom connector listening to a REST Api end point  in Flink

Thanks
Archit

[Chesnay Schepler]

Hello,

In general it should be possible for you to write a source that directly works with your REST API.

The SourceFunction interface is simple enough that you can do pretty much anything, which makes
me curious as to what makes you wonder whether it is possible. Have you tried implementing your
own source function but hit a wall somewhere?

Bear in mind though, if it has to work with checkpointing than you're probably better off using another
kafka instance though.

Regards,
Chesnay

On 10.04.2017 09:50, Archit Mittal wrote:

Hi

Any suggestions regarding this.

Thanks

Archit

On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <[hidden email]> wrote:

Hi Chesnay

I have a REST Api that listens to events that come through Kafka but i dont have access to that Kafka . Can I write a connector that consumes events through the REST end point and not directly through Kafka. 

I have a solution that i can dump the events coming to the api into another Kafka instance and consume through that but it will add another hop to the flow. 

Is There a better way to do this by writing a custom connector or something like that . 

Thanks

Archit

On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

Please give us a bit more information, I'm really not sure what you mean. :(

Regards,
Chesnay

On 06.04.2017 09:40, Archit Mittal wrote:

Hi

Can we write a custom connector listening to a REST Api end point  in Flink

Thanks
Archit

[Archit Mittal]

Hi 

I wanted to know the pros and cons of both approaches and if there would be some better approaches for the same. Thanks for the info regarding the approaches. From your input as it has to be fault tolerant with checkpointing i think adding my own kafka in between would be better. 

Thanks

Archit

On Mon, Apr 10, 2017 at 5:42 PM, Chesnay Schepler <[hidden email]> wrote:

Hello,

In general it should be possible for you to write a source that directly works with your REST API.

The SourceFunction interface is simple enough that you can do pretty much anything, which makes
me curious as to what makes you wonder whether it is possible. Have you tried implementing your
own source function but hit a wall somewhere?

Bear in mind though, if it has to work with checkpointing than you're probably better off using another
kafka instance though.

Regards,
Chesnay

On 10.04.2017 09:50, Archit Mittal wrote:

Hi

Any suggestions regarding this.

Thanks

Archit

On Thu, Apr 6, 2017 at 2:52 PM, Archit Mittal <[hidden email]> wrote:

Hi Chesnay

I have a REST Api that listens to events that come through Kafka but i dont have access to that Kafka . Can I write a connector that consumes events through the REST end point and not directly through Kafka. 

I have a solution that i can dump the events coming to the api into another Kafka instance and consume through that but it will add another hop to the flow. 

Is There a better way to do this by writing a custom connector or something like that . 

Thanks

Archit

On Thu, Apr 6, 2017 at 1:15 PM, Chesnay Schepler <[hidden email][hidden email]> wrote:

Hello,

Please give us a bit more information, I'm really not sure what you mean. :(

Regards,
Chesnay

On 06.04.2017 09:40, Archit Mittal wrote:

Hi

Can we write a custom connector listening to a REST Api end point  in Flink

Thanks
Archit