How can I make HTTP requests from an Apache Flink program?

[jacopo.gobbi]

Hi all,

 

I was wondering if anybody has any recommendation over making HTTP requests from Flink to another service.

On the long term we are looking for a solution that is both performing and integrates well with our flink program.

Does it matter the library we use? Do we need a special connector to make HTTP calls?

One library we thought that could fit our necessities Akka akka HTTP client API due to the possibility to make async HTTP calls.

 

We are using Scala 2.12 and Flink 1.7.

 

Kind regards,

 

Jacopo Gobbi

Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named.  If you are not the named addressee you
should not disseminate, distribute or copy this e-mail.  Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.
       
E-mails are not encrypted and cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses.  The sender
therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of e-mail transmission.  
If verification is required please request a hard-copy version.  This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities
or related financial instruments.

UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.

For information on how UBS processes and keeps secure your personal
data, please visit our Privacy Notice at
https://www.ubs.com/global/en/legal/privacy.html

[miki haiat]

Can you share more which use case are you trying to implement ?

On Tue, Jan 15, 2019 at 2:02 PM <[hidden email]> wrote:

Hi all,

 

I was wondering if anybody has any recommendation over making HTTP requests from Flink to another service.

On the long term we are looking for a solution that is both performing and integrates well with our flink program.

Does it matter the library we use? Do we need a special connector to make HTTP calls?

One library we thought that could fit our necessities Akka akka HTTP client API due to the possibility to make async HTTP calls.

 

We are using Scala 2.12 and Flink 1.7.

 

Kind regards,

 

Jacopo Gobbi

[jacopo.gobbi]

Hi,

 

I have a flink program which needs to process many messages and part of this processing is to process the data using an external web service using http calls.

Example:

val myStream: DataStream[String]
myStream
  .map(new MyProcessingFunction)
  .map(new MyWebServiceHttpClient)
  .print

Any suggestion?

 

Jac

 

 

From: miki haiat [mailto:[hidden email]]
Sent: Dienstag, 15. Januar 2019 14:03
To: Gobbi, Jacopo-XT
Cc: user
Subject: [External] Re: How can I make HTTP requests from an Apache Flink program?

 

Can you share more which use case are you trying to implement ?

 

 

 

On Tue, Jan 15, 2019 at 2:02 PM <[hidden email]> wrote:

Hi all,

 

I was wondering if anybody has any recommendation over making HTTP requests from Flink to another service.

On the long term we are looking for a solution that is both performing and integrates well with our flink program.

Does it matter the library we use? Do we need a special connector to make HTTP calls?

One library we thought that could fit our necessities Akka akka HTTP client API due to the possibility to make async HTTP calls.

 

We are using Scala 2.12 and Flink 1.7.

 

Kind regards,

 

Jacopo Gobbi

Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named.  If you are not the named addressee you
should not disseminate, distribute or copy this e-mail.  Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.
       
E-mails are not encrypted and cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses.  The sender
therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of e-mail transmission.  
If verification is required please request a hard-copy version.  This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities
or related financial instruments.

UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.

For information on how UBS processes and keeps secure your personal
data, please visit our Privacy Notice at
https://www.ubs.com/global/en/legal/privacy.html

[Alexx]

Hi Jacopo,

Check this: https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/asyncio.html

Best,

Alex

On Tue, 15 Jan 2019 at 13:57, <[hidden email]> wrote:

Hi,

 

I have a flink program which needs to process many messages and part of this processing is to process the data using an external web service using http calls.

Example:

val myStream: DataStream[String]
myStream
  .map(new MyProcessingFunction)
  .map(new MyWebServiceHttpClient)
  .print

Any suggestion?

 

Jac

 

 

From: miki haiat [mailto:[hidden email]]
Sent: Dienstag, 15. Januar 2019 14:03
To: Gobbi, Jacopo-XT
Cc: user
Subject: [External] Re: How can I make HTTP requests from an Apache Flink program?

 

Can you share more which use case are you trying to implement ?

 

 

 

On Tue, Jan 15, 2019 at 2:02 PM <[hidden email]> wrote:

Hi all,

 

I was wondering if anybody has any recommendation over making HTTP requests from Flink to another service.

On the long term we are looking for a solution that is both performing and integrates well with our flink program.

Does it matter the library we use? Do we need a special connector to make HTTP calls?

One library we thought that could fit our necessities Akka akka HTTP client API due to the possibility to make async HTTP calls.

 

We are using Scala 2.12 and Flink 1.7.

 

Kind regards,

 

Jacopo Gobbi